Attachment 856063 Details for Bug 1058300 – engine.log

engine.log

file_1058300.txt (text/plain), 689.12 KB, created by Andrew Lau on 2014-01-27 13:30:50 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Andrew Lau

Created: 2014-01-27 13:30:50 UTC

Size: 689.12 KB

patch

obsolete

>
>
>Thread-969598::INFO::2014-01-28 00:13:04,495::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-969598::INFO::2014-01-28 00:13:04,496::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000628415', 'lastCheck': '9.3', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000459553', 'lastCheck': '3.5', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00105881', 'lastCheck': '3.4', 'valid': True}}
>
>Thread-12::DEBUG::2014-01-28 00:13:04,682::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:13:04,728::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-129::DEBUG::2014-01-28 00:13:05,153::iscsi::393::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-129::DEBUG::2014-01-28 00:13:05,154::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:13:05,215::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:13:05,365::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-129::DEBUG::2014-01-28 00:13:05,403::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc
>Thread-129::DEBUG::2014-01-28 00:13:05,406::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-129::DEBUG::2014-01-28 00:13:05,418::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=562f0160-7b80-42e2-b248-5754455c40fc:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=7e0cc075-eee7-4268-946e-33a79dd08748', 'REMOTE_PATH=172.16.1.5:/HOSTED-ENGINE', 'ROLE=Master', 'SDUUID=562f0160-7b80-42e2-b248-5754455c40fc', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=14825cb0b944c5c2ae9ed025c82e06730a2d91c4']
>Thread-129::DEBUG::2014-01-28 00:13:05,421::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-129::INFO::2014-01-28 00:13:05,421::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_imageNS already registered
>Thread-129::INFO::2014-01-28 00:13:05,421::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_volumeNS already registered
>Thread-129::DEBUG::2014-01-28 00:13:05,435::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:13:05,482::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n478 bytes (478 B) copied, 0.000527485 s, 906 kB/s\n'; <rc> = 0
>Thread-969602::DEBUG::2014-01-28 00:13:05,559::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call getCapabilities with () {}
>Thread-969602::DEBUG::2014-01-28 00:13:05,561::utils::555::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-969602::DEBUG::2014-01-28 00:13:05,603::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-366769::INFO::2014-01-28 00:13:05,624::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='a746f514-51eb-4926-80d5-545108438f01', volUUID='ba8685ec-668a-4780-88ba-0b114e1a7e7d', options=None)
>Thread-969602::DEBUG::2014-01-28 00:13:05,634::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-366769::INFO::2014-01-28 00:13:05,690::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2744213504', 'apparentsize': '53687091200'}
>Thread-969602::DEBUG::2014-01-28 00:13:05,705::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969602::DEBUG::2014-01-28 00:13:05,730::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.2' (cwd None)
>Thread-969602::DEBUG::2014-01-28 00:13:05,774::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969602::DEBUG::2014-01-28 00:13:05,775::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.2' (cwd None)
>Thread-969602::DEBUG::2014-01-28 00:13:05,815::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969602::DEBUG::2014-01-28 00:13:05,818::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.3' (cwd None)
>Thread-969602::DEBUG::2014-01-28 00:13:05,858::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969602::DEBUG::2014-01-28 00:13:05,859::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.3' (cwd None)
>Thread-969602::DEBUG::2014-01-28 00:13:05,900::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969602::DEBUG::2014-01-28 00:13:05,903::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.10' (cwd None)
>Thread-969602::DEBUG::2014-01-28 00:13:05,945::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969602::DEBUG::2014-01-28 00:13:05,946::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.10' (cwd None)
>Thread-213624::INFO::2014-01-28 00:13:05,996::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='91386b87-5fe0-4368-9e87-a28896024015', volUUID='b0db0737-8168-4502-a236-1c3162f98f6a', options=None)
>Thread-219052::INFO::2014-01-28 00:13:05,999::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='00d2f5e6-0263-4ac7-aa12-98ea66f02d00', volUUID='98d6fa79-aecc-429e-8624-4b6f706bdf4f', options=None)
>Thread-969602::DEBUG::2014-01-28 00:13:06,011::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-213624::INFO::2014-01-28 00:13:06,050::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '5274959872', 'apparentsize': '26843545600'}
>Thread-219052::INFO::2014-01-28 00:13:06,056::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2936643584', 'apparentsize': '26843545600'}
>Thread-969602::DEBUG::2014-01-28 00:13:06,080::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.11' (cwd None)
>Thread-969602::DEBUG::2014-01-28 00:13:06,122::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969602::DEBUG::2014-01-28 00:13:06,126::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.11' (cwd None)
>Thread-969602::DEBUG::2014-01-28 00:13:06,168::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969602::DEBUG::2014-01-28 00:13:06,182::caps::453::root::(_getKeyPackages) rpm package glusterfs-rdma not found
>Thread-969602::DEBUG::2014-01-28 00:13:06,183::caps::453::root::(_getKeyPackages) rpm package glusterfs-fuse not found
>Thread-969602::DEBUG::2014-01-28 00:13:06,185::caps::453::root::(_getKeyPackages) rpm package gluster-swift not found
>Thread-969602::DEBUG::2014-01-28 00:13:06,191::caps::453::root::(_getKeyPackages) rpm package gluster-swift-object not found
>Thread-969602::DEBUG::2014-01-28 00:13:06,192::caps::453::root::(_getKeyPackages) rpm package glusterfs not found
>Thread-969602::DEBUG::2014-01-28 00:13:06,192::caps::453::root::(_getKeyPackages) rpm package gluster-swift-plugin not found
>Thread-969602::DEBUG::2014-01-28 00:13:06,200::caps::453::root::(_getKeyPackages) rpm package gluster-swift-account not found
>Thread-969602::DEBUG::2014-01-28 00:13:06,201::caps::453::root::(_getKeyPackages) rpm package gluster-swift-proxy not found
>Thread-969602::DEBUG::2014-01-28 00:13:06,201::caps::453::root::(_getKeyPackages) rpm package gluster-swift-doc not found
>Thread-969602::DEBUG::2014-01-28 00:13:06,202::caps::453::root::(_getKeyPackages) rpm package glusterfs-server not found
>Thread-969602::DEBUG::2014-01-28 00:13:06,202::caps::453::root::(_getKeyPackages) rpm package gluster-swift-container not found
>Thread-969602::DEBUG::2014-01-28 00:13:06,202::caps::453::root::(_getKeyPackages) rpm package glusterfs-geo-replication not found
>Thread-969602::DEBUG::2014-01-28 00:13:06,209::BindingXMLRPC::976::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.3.1.el6.x86_64', 'buildtime': 1388745567.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '33.gitf029545.el6', 'buildtime': 1390094004L, 'version': '4.14.0'}, 'qemu-kvm': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.2', 'buildtime': 1387360004L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'mom': {'release': '20140117.gitfd877c5.el6', 'buildtime': 1389966957L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           L5320  @ 1.86GHz', 'hooks': {'before_vm_start': {'50_hostedengine': {'md5': '45dde62155b5412eafbfff5ef265acc2'}}}, 'cpuSockets': '2', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'gateway': '172.16.0.1', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.2', 'vnet0']}, 'os_net': {'iface': 'os_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.10']}, 'r2o_net': {'iface': 'r2o_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}, 'storage_network': {'iface': u'bond0.3', 'addr': '172.16.1.5', 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000', 'netmask': '255.255.255.255', 'bridged': False, 'interface': u'bond0.3', 'ipv6gateway': '::', 'gateway': ''}}, 'bridges': {'ovirtmgmt': {'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '172.16.0.1', 'ports': ['bond0.2', 'vnet0']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}, 'os_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.10']}, 'r2o_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}}, 'uuid': '44454C4C-3100-104D-8046-B9C04F583153', 'lastClientIface': 'ovirtmgmt', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'cb327caa-ca77-4efe-b7b8-63aba3cda84d', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:85', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth2', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:85', 'speed': 1000, 'mtu': '4000'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'e401a6a5-8773-4971-9213-b3c2bb52f16c', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:83', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:83', 'speed': 1000, 'mtu': '4000'}, 'eth0': {'netmask': '', 'addr': '', 'hwaddr': '00:e0:4c:80:13:02', 'cfg': {'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': '00:e0:4c:80:13:02', 'STP': 'no', 'DEVICE': 'eth0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'speed': 1000, 'mtu': '1500'}}, 'software_revision': '33', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,tm2,ssse3,cx16,xtpr,pdcm,dca,lahf_lm,dts,tpr_shadow,vnmi,flexpriority,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': ['eth1', 'eth2'], 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'MTU': '4000', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'BONDING_OPTS': 'miimon=100 mode=balance-rr', 'DEVICE': 'bond0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '32105', 'cpuSpeed': '1861.936', 'version_name': 'Snow Man', 'vlans': {'bond0.2': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'ovirtmgmt', 'VLAN': 'yes', 'MTU': '4000', 'BOOTPROTO': 'none', 'DEVICE': 'bond0.2', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 2, 'mtu': '4000'}, 'bond0.3': {'netmask': '255.255.255.255', 'iface': 'bond0', 'addr': '172.16.1.5', 'cfg': {'VLAN': 'yes', 'DEFROUTE': 'no', 'MTU': '4000', 'NM_CONTROLLED': 'no', 'DEVICE': 'bond0.3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 3, 'mtu': '4000'}, 'eth0.11': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'r2o_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.11', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 11, 'mtu': '1500'}, 'eth0.10': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'os_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.10', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 10, 'mtu': '1500'}}, 'cpuCores': '8', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '8', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '172.16.0.11'}}
>Thread-12::DEBUG::2014-01-28 00:13:06,745::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:13:06,911::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-250271::INFO::2014-01-28 00:13:07,055::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='51206945-475a-48b1-96e1-689ca96312a5', volUUID='1732793c-bf4f-4f8d-a5b9-4f41e99154ba', options=None)
>Thread-218532::INFO::2014-01-28 00:13:07,055::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='50cabd29-7058-4510-b61f-db21862d99df', volUUID='96d9eaab-9dff-45f6-8fe7-b4fd992ee23e', options=None)
>Thread-218532::ERROR::2014-01-28 00:13:07,107::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/50cabd29-7058-4510-b61f-db21862d99df/96d9eaab-9dff-45f6-8fe7-b4fd992ee23e'
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>OSError: [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/50cabd29-7058-4510-b61f-db21862d99df/96d9eaab-9dff-45f6-8fe7-b4fd992ee23e'
>Thread-250271::ERROR::2014-01-28 00:13:07,108::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/51206945-475a-48b1-96e1-689ca96312a5/1732793c-bf4f-4f8d-a5b9-4f41e99154ba'
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>OSError: [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/51206945-475a-48b1-96e1-689ca96312a5/1732793c-bf4f-4f8d-a5b9-4f41e99154ba'
>Thread-218532::ERROR::2014-01-28 00:13:07,332::vm::3781::vm.Vm::(updateDriveVolume) vmId=`b3cde33c-f4c5-4b42-8e9d-820b80abc2fb`::Unable to update the volume 96d9eaab-9dff-45f6-8fe7-b4fd992ee23e (domain: ff3a0c94-11dc-446d-b093-5f68ad81520d image: 50cabd29-7058-4510-b61f-db21862d99df) for the drive vda
>Thread-250271::ERROR::2014-01-28 00:13:07,334::vm::3781::vm.Vm::(updateDriveVolume) vmId=`48ad952b-4457-4ae1-bf85-93f869382298`::Unable to update the volume 1732793c-bf4f-4f8d-a5b9-4f41e99154ba (domain: ff3a0c94-11dc-446d-b093-5f68ad81520d image: 51206945-475a-48b1-96e1-689ca96312a5) for the drive vda
>libvirtEventLoop::INFO::2014-01-28 00:13:07,431::vm::4507::vm.Vm::(_onAbnormalStop) vmId=`546b9733-3453-4210-90fd-2b79cf08d53b`::abnormal vm stop device virtio-disk0 error eother
>libvirtEventLoop::DEBUG::2014-01-28 00:13:07,442::vm::5098::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`546b9733-3453-4210-90fd-2b79cf08d53b`::event Suspended detail 2 opaque None
>Thread-969604::DEBUG::2014-01-28 00:13:07,448::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call tasksList with () {}
>Thread-969604::ERROR::2014-01-28 00:13:07,530::BindingXMLRPC::985::vds::(wrapper) vdsm exception occured
>Traceback (most recent call last):
>  File "/usr/share/vdsm/BindingXMLRPC.py", line 972, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/gluster/api.py", line 54, in wrapper
>    rv = func(*args, **kwargs)
>  File "/usr/share/vdsm/gluster/api.py", line 306, in tasksList
>    status = self.svdsmProxy.glusterTasksList(taskIds)
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in glusterTasksList
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 740, in _callmethod
>    raise convert_to_error(kind, result)
>GlusterCmdExecFailedException: Command execution failed
>error: tasks is not a valid status option
>Usage: volume status [all | <VOLNAME> [nfs|shd|<BRICK>]] [detail|clients|mem|inode|fd|callpool]
>return code: 1
>libvirtEventLoop::INFO::2014-01-28 00:13:07,598::vm::4507::vm.Vm::(_onAbnormalStop) vmId=`546b9733-3453-4210-90fd-2b79cf08d53b`::abnormal vm stop device virtio-disk0 error eother
>libvirtEventLoop::INFO::2014-01-28 00:13:07,598::vm::4507::vm.Vm::(_onAbnormalStop) vmId=`546b9733-3453-4210-90fd-2b79cf08d53b`::abnormal vm stop device virtio-disk0 error eother
>libvirtEventLoop::INFO::2014-01-28 00:13:07,598::vm::4507::vm.Vm::(_onAbnormalStop) vmId=`546b9733-3453-4210-90fd-2b79cf08d53b`::abnormal vm stop device virtio-disk0 error eother
>libvirtEventLoop::INFO::2014-01-28 00:13:07,599::vm::4507::vm.Vm::(_onAbnormalStop) vmId=`546b9733-3453-4210-90fd-2b79cf08d53b`::abnormal vm stop device virtio-disk0 error eother
>libvirtEventLoop::INFO::2014-01-28 00:13:07,601::vm::4507::vm.Vm::(_onAbnormalStop) vmId=`546b9733-3453-4210-90fd-2b79cf08d53b`::abnormal vm stop device virtio-disk0 error eother
>libvirtEventLoop::INFO::2014-01-28 00:13:07,602::vm::4507::vm.Vm::(_onAbnormalStop) vmId=`546b9733-3453-4210-90fd-2b79cf08d53b`::abnormal vm stop device virtio-disk0 error eother
>libvirtEventLoop::INFO::2014-01-28 00:13:07,602::vm::4507::vm.Vm::(_onAbnormalStop) vmId=`546b9733-3453-4210-90fd-2b79cf08d53b`::abnormal vm stop device virtio-disk0 error eother
>libvirtEventLoop::INFO::2014-01-28 00:13:07,603::vm::4507::vm.Vm::(_onAbnormalStop) vmId=`546b9733-3453-4210-90fd-2b79cf08d53b`::abnormal vm stop device virtio-disk0 error eother
>libvirtEventLoop::INFO::2014-01-28 00:13:07,603::vm::4507::vm.Vm::(_onAbnormalStop) vmId=`546b9733-3453-4210-90fd-2b79cf08d53b`::abnormal vm stop device virtio-disk0 error eother
>libvirtEventLoop::INFO::2014-01-28 00:13:07,604::vm::4507::vm.Vm::(_onAbnormalStop) vmId=`546b9733-3453-4210-90fd-2b79cf08d53b`::abnormal vm stop device virtio-disk0 error eother
>libvirtEventLoop::INFO::2014-01-28 00:13:07,604::vm::4507::vm.Vm::(_onAbnormalStop) vmId=`546b9733-3453-4210-90fd-2b79cf08d53b`::abnormal vm stop device virtio-disk0 error eother
>libvirtEventLoop::INFO::2014-01-28 00:13:08,133::vm::4507::vm.Vm::(_onAbnormalStop) vmId=`48ad952b-4457-4ae1-bf85-93f869382298`::abnormal vm stop device virtio-disk0 error eother
>libvirtEventLoop::DEBUG::2014-01-28 00:13:08,134::vm::5098::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`48ad952b-4457-4ae1-bf85-93f869382298`::event Suspended detail 2 opaque None
>
>Thread-969606::DEBUG::2014-01-28 00:13:08,511::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {} flowID [3e8469]
>Thread-969605::DEBUG::2014-01-28 00:13:08,543::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call vmGetStats with ('2d425c4c-91b1-4564-b276-f27cc581e6a1',) {}
>Thread-969605::DEBUG::2014-01-28 00:13:08,544::BindingXMLRPC::976::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'root', 'memUsage': '53', 'acpiEnable': 'true', 'guestFQDN': u'engine.melb.example.net', 'pid': '18113', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'/', u'total': '25323499520', u'used': '2592124928', u'fs': u'ext4'}, {u'path': u'/boot', u'total': '507744256', u'used': '55643136', u'fs': u'ext4'}], 'network': {u'vnet0': {'macAddr': '00:16:3e:0a:b7:55', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmType': 'kvm', 'guestName': u'engine.melb.example.net', 'elapsedTime': '610773', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'mem_free': '1970940', u'swap_in': '0', u'pageflt': '145', u'mem_total': '3922728', u'mem_unused': '1437304'}, 'cpuSys': '1.24', 'appsList': [u'ovirt-guest-agent-1.0.8-1.el6', u'kernel-2.6.32-431.el6', u'kernel-2.6.32-431.3.1.el6'], 'guestOs': u'2.6.32-431.3.1.el6.x86_64', 'displayType': 'vnc', 'status': 'Up', 'hash': '2950633987041661405', 'vmId': '2d425c4c-91b1-4564-b276-f27cc581e6a1', 'displayIp': '0', 'displayPort': u'5900', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '26843545600', 'writeLatency': '3946838', 'imageID': 'daa7fa08-1354-4b19-a80f-8694ea63f914', 'flushLatency': '78638', 'readRate': '0.00', 'truesize': '3920576512', 'writeRate': '24859.40'}, 'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.96', 'cpuUser': '11.84', 'lastLogin': 1390828070.405062, 'guestIPs': u'172.16.0.10', 'netIfaces': [{u'hw': u'00:16:3e:0a:b7:55', u'name': u'eth0', u'inet': [u'172.16.0.10'], u'inet6': [u'fe80::216:3eff:fe0a:b755']}]}]}
>Thread-969606::DEBUG::2014-01-28 00:13:08,562::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'OFFLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>libvirtEventLoop::INFO::2014-01-28 00:13:08,770::vm::4507::vm.Vm::(_onAbnormalStop) vmId=`01d705e3-6b62-4796-b9e4-4de1c477401a`::abnormal vm stop device virtio-disk0 error eother
>libvirtEventLoop::DEBUG::2014-01-28 00:13:08,771::vm::5098::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`01d705e3-6b62-4796-b9e4-4de1c477401a`::event Suspended detail 2 opaque None
>
>
>libvirtEventLoop::INFO::2014-01-28 00:13:08,828::vm::4507::vm.Vm::(_onAbnormalStop) vmId=`01d705e3-6b62-4796-b9e4-4de1c477401a`::abnormal vm stop device virtio-disk0 error eother
>libvirtEventLoop::INFO::2014-01-28 00:13:08,829::vm::4507::vm.Vm::(_onAbnormalStop) vmId=`01d705e3-6b62-4796-b9e4-4de1c477401a`::abnormal vm stop device virtio-disk0 error eother
>libvirtEventLoop::INFO::2014-01-28 00:13:08,829::vm::4507::vm.Vm::(_onAbnormalStop) vmId=`01d705e3-6b62-4796-b9e4-4de1c477401a`::abnormal vm stop device virtio-disk0 error eother
>Thread-12::DEBUG::2014-01-28 00:13:08,939::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:13:08,981::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969608::DEBUG::2014-01-28 00:13:09,788::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call vmGetStats with ('48ad952b-4457-4ae1-bf85-93f869382298',) {}
>Thread-969608::DEBUG::2014-01-28 00:13:09,790::BindingXMLRPC::976::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '67', 'acpiEnable': 'true', 'guestFQDN': u'zcs1.melb.example.net', 'pid': '10566', 'lastLogout': 1390378456.3976641, 'session': 'Unknown', 'displaySecurePort': u'5912', 'timeOffset': 0L, 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'mem_free': '1350508', u'swap_in': '0', u'pageflt': '703', u'mem_total': '3922480', u'mem_unused': '547764'}, 'balloonInfo': {'balloon_max': '4194304', 'balloon_target': '4194304', 'balloon_cur': '4194304', 'balloon_min': '4194304'}, 'pauseCode': 'EOTHER', 'disksUsage': [{u'path': u'/', u'total': '52316237824', u'used': '6471032832', u'fs': u'ext4'}, {u'path': u'/boot', u'total': '253869056', u'used': '51469312', u'fs': u'ext4'}], 'network': {u'vnet9': {'macAddr': '00:1a:4a:19:42:e0', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet9'}, u'vnet6': {'macAddr': '00:1a:4a:19:42:20', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet6'}}, 'displayType': 'qxl', 'guestName': u'zcs1.melb.example.net', 'elapsedTime': '449933', 'vmType': 'kvm', 'cpuSys': '2.08', 'appsList': [u'ovirt-guest-agent-1.0.8-1.el6', u'kernel-2.6.32-431.el6', u'kernel-2.6.32-431.3.1.el6'], 'guestOs': u'2.6.32-431.3.1.el6.x86_64', 'status': 'Paused', 'hash': '-6329857579508662808', 'vmId': '48ad952b-4457-4ae1-bf85-93f869382298', 'displayIp': '0', 'displayPort': u'5911', 'lastUser': 'Unknown', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '958847', 'apparentsize': '9283829760', 'writeLatency': '4224083', 'imageID': '51206945-475a-48b1-96e1-689ca96312a5', 'flushLatency': '2544723', 'readRate': '67.61', 'truesize': '9283706880', 'writeRate': '38131.50'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.28', 'cpuUser': '3.99', 'lastLogin': 1390614907.1540401, 'guestIPs': u'10.0.1.25 192.168.168.147', 'netIfaces': [{u'hw': u'00:1a:4a:19:42:20', u'name': u'eth0', u'inet': [u'10.0.1.25'], u'inet6': [u'fe80::21a:4aff:fe19:4220']}, {u'hw': u'00:1a:4a:19:42:e0', u'name': u'eth1', u'inet': [u'192.168.168.147'], u'inet6': [u'fe80::21a:4aff:fe19:42e0']}]}]}
>Thread-969609::DEBUG::2014-01-28 00:13:09,800::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call vmGetStats with ('546b9733-3453-4210-90fd-2b79cf08d53b',) {}
>Thread-969609::DEBUG::2014-01-28 00:13:09,803::BindingXMLRPC::976::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '45', 'acpiEnable': 'true', 'guestFQDN': u'ipa2.melb.example.net', 'pid': '28558', 'lastLogout': 1390354308.0958631, 'session': 'Unknown', 'displaySecurePort': u'5906', 'timeOffset': 39600L, 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'mem_free': '1134660', u'swap_in': '0', u'pageflt': '809', u'mem_total': '1922356', u'mem_unused': '235704'}, 'balloonInfo': {'balloon_max': '2097152', 'balloon_target': '2097152', 'balloon_cur': '2097152', 'balloon_min': '2097152'}, 'pauseCode': 'EOTHER', 'disksUsage': [{u'path': u'/', u'total': '25893892096', u'used': '2016739328', u'fs': u'ext4'}, {u'path': u'/boot', u'total': '253869056', u'used': '51484672', u'fs': u'ext4'}], 'network': {u'vnet8': {'macAddr': '00:1a:4a:19:42:1c', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet8'}, u'vnet3': {'macAddr': '00:1a:4a:19:42:cf', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet3'}}, 'displayType': 'qxl', 'guestName': u'ipa2.melb.example.net', 'elapsedTime': '474081', 'vmType': 'kvm', 'cpuSys': '0.66', 'appsList': [u'ovirt-guest-agent-1.0.8-1.el6', u'kernel-2.6.32-431.el6', u'kernel-2.6.32-431.3.1.el6'], 'guestOs': u'2.6.32-431.3.1.el6.x86_64', 'status': 'Paused', 'hash': '-7316784786077174997', 'vmId': '546b9733-3453-4210-90fd-2b79cf08d53b', 'displayIp': '0', 'displayPort': u'5905', 'lastUser': 'Unknown', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '26843545600', 'writeLatency': '8072732', 'imageID': '00d2f5e6-0263-4ac7-aa12-98ea66f02d00', 'flushLatency': '639150', 'readRate': '0.00', 'truesize': '2936643584', 'writeRate': '16018.46'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.57', 'cpuUser': '2.16', 'lastLogin': 1390445505.8023541, 'guestIPs': u'10.0.1.11 192.168.168.149', 'netIfaces': [{u'hw': u'00:1a:4a:19:42:cf', u'name': u'eth0', u'inet': [u'10.0.1.11'], u'inet6': [u'fe80::21a:4aff:fe19:42cf']}, {u'hw': u'00:1a:4a:19:42:1c', u'name': u'eth1', u'inet': [u'192.168.168.149'], u'inet6': [u'fe80::21a:4aff:fe19:421c']}]}]}
>Thread-969610::DEBUG::2014-01-28 00:13:09,812::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call vmGetStats with ('01d705e3-6b62-4796-b9e4-4de1c477401a',) {}
>Thread-969610::DEBUG::2014-01-28 00:13:09,816::BindingXMLRPC::976::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '21', 'acpiEnable': 'true', 'guestFQDN': u'rabbitmq1.melb.example.net', 'pid': '14066', 'lastLogout': 1390447160.358865, 'session': 'Unknown', 'displaySecurePort': u'5914', 'timeOffset': 39600L, 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'mem_free': '1637376', u'swap_in': '0', u'pageflt': '397', u'mem_total': '1922356', u'mem_unused': '970932'}, 'balloonInfo': {'balloon_max': '2097152', 'balloon_target': '2097152', 'balloon_cur': '2097152', 'balloon_min': '2097152'}, 'pauseCode': 'EOTHER', 'disksUsage': [{u'path': u'/', u'total': '52316237824', u'used': '1605017600', u'fs': u'ext4'}, {u'path': u'/boot', u'total': '253869056', u'used': '51504128', u'fs': u'ext4'}], 'network': {u'vnet10': {'macAddr': '00:1a:4a:19:42:1b', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet10'}}, 'displayType': 'qxl', 'guestName': u'rabbitmq1.melb.example.net', 'elapsedTime': '381229', 'vmType': 'kvm', 'cpuSys': '1.66', 'appsList': [u'ovirt-guest-agent-1.0.8-1.el6', u'kernel-2.6.32-431.el6', u'kernel-2.6.32-431.3.1.el6'], 'guestOs': u'2.6.32-431.3.1.el6.x86_64', 'status': 'Paused', 'hash': '566972331207234100', 'vmId': '01d705e3-6b62-4796-b9e4-4de1c477401a', 'displayIp': '0', 'displayPort': u'5913', 'lastUser': 'Unknown', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '53687091200', 'writeLatency': '1229596', 'imageID': 'a746f514-51eb-4926-80d5-545108438f01', 'flushLatency': '2536338', 'readRate': '0.00', 'truesize': '2744213504', 'writeRate': '6924.80'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.87', 'cpuUser': '1.94', 'lastLogin': 1390466649.5071361, 'guestIPs': u'10.0.1.100', 'netIfaces': [{u'hw': u'00:1a:4a:19:42:1b', u'name': u'eth0', u'inet': [u'10.0.1.100'], u'inet6': [u'fe80::21a:4aff:fe19:421b']}]}]}
>Thread-969611::DEBUG::2014-01-28 00:13:10,879::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-969611::INFO::2014-01-28 00:13:10,880::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-969611::INFO::2014-01-28 00:13:10,881::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-969612::DEBUG::2014-01-28 00:13:10,887::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-969612::INFO::2014-01-28 00:13:10,888::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-969612::INFO::2014-01-28 00:13:10,896::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': 'ff3a0c94-11dc-446d-b093-5f68ad81520d', 'name': 'NextDC_M1', 'version': '3', 'domains': 'ff3a0c94-11dc-446d-b093-5f68ad81520d:Active,436c344c-bc57-441b-b311-c9595c6039e1:Active', 'pool_status': 'connected', 'isoprefix': '/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/images/11111111-1111-1111-1111-111111111111', 'type': 'GLUSTERFS', 'master_ver': 1, 'lver': 1}, 'dominfo': {'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'status': 'Active', 'diskfree': '917084438528', 'isoprefix': '', 'alerts': [], 'disktotal': '982907879424', 'version': 3}, '436c344c-bc57-441b-b311-c9595c6039e1': {'status': 'Active', 'diskfree': '21445476352', 'isoprefix': '/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '25323634688', 'version': 0}}}
>Thread-36::DEBUG::2014-01-28 00:13:10,983::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:13:11,020::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-36::DEBUG::2014-01-28 00:13:11,118::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n364 bytes (364 B) copied, 0.000345614 s, 1.1 MB/s\n'; <rc> = 0
>Thread-12::DEBUG::2014-01-28 00:13:11,164::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-35::ERROR::2014-01-28 00:13:11,184::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain ff3a0c94-11dc-446d-b093-5f68ad81520d monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
>    self.domain.selftest()
>  File "/usr/share/vdsm/storage/nfsSD.py", line 113, in selftest
>    fileSD.FileStorageDomain.selftest(self)
>  File "/usr/share/vdsm/storage/fileSD.py", line 552, in selftest
>    self.oop.os.statvfs(self.domaindir)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 199, in callCrabRPCFunction
>    raise err
>OSError: [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d'
>Thread-35::DEBUG::2014-01-28 00:13:11,190::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain ff3a0c94-11dc-446d-b093-5f68ad81520d changed its status to Invalid
>Thread-969613::DEBUG::2014-01-28 00:13:11,195::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
>Thread-969613::DEBUG::2014-01-28 00:13:11,198::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
>Thread-969613::DEBUG::2014-01-28 00:13:11,201::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
>libvirtEventLoop::INFO::2014-01-28 00:13:11,453::vm::4507::vm.Vm::(_onAbnormalStop) vmId=`bbbd4efa-022d-44d3-8274-a0506bda2d5e`::abnormal vm stop device virtio-disk0 error eother
>libvirtEventLoop::DEBUG::2014-01-28 00:13:11,453::vm::5098::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`bbbd4efa-022d-44d3-8274-a0506bda2d5e`::event Suspended detail 2 opaque None
>Thread-969615::DEBUG::2014-01-28 00:13:11,719::BindingXMLRPC::159::vds::(wrapper) client [127.0.0.1]
>Thread-969615::INFO::2014-01-28 00:13:11,720::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='7e0cc075-eee7-4268-946e-33a79dd08748', conList=[{'port': '', 'connection': '172.16.1.5:/HOSTED-ENGINE', 'iqn': '', 'portal': '', 'user': 'kvm', 'protocol_version': '3', 'password': '******', 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}], options=None)
>Thread-969615::DEBUG::2014-01-28 00:13:11,724::hsm::2273::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE
>Thread-969615::DEBUG::2014-01-28 00:13:13,051::hsm::2297::Storage.HSM::(__prefetchDomains) Found SD uuids: ('562f0160-7b80-42e2-b248-5754455c40fc',)
>Thread-969615::DEBUG::2014-01-28 00:13:13,052::hsm::2353::Storage.HSM::(connectStorageServer) knownSDs: {562f0160-7b80-42e2-b248-5754455c40fc: storage.nfsSD.findDomain, ff3a0c94-11dc-446d-b093-5f68ad81520d: storage.glusterSD.findDomain, 436c344c-bc57-441b-b311-c9595c6039e1: storage.nfsSD.findDomain}
>Thread-969615::INFO::2014-01-28 00:13:13,053::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}]}
>Thread-12::DEBUG::2014-01-28 00:13:13,213::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:13:13,261::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969616::DEBUG::2014-01-28 00:13:13,356::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.11]
>Thread-969616::INFO::2014-01-28 00:13:13,357::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-969616::INFO::2014-01-28 00:13:13,358::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000527485', 'lastCheck': '7.9', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000345614', 'lastCheck': '2.2', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '2.2', 'valid': False}}
>Thread-969617::DEBUG::2014-01-28 00:13:13,699::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {}
>Thread-969617::DEBUG::2014-01-28 00:13:13,760::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'OFFLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>libvirtEventLoop::INFO::2014-01-28 00:13:14,165::vm::4507::vm.Vm::(_onAbnormalStop) vmId=`e85be056-7715-485c-9cb8-69f9980ed3a3`::abnormal vm stop device virtio-disk0 error eother
>libvirtEventLoop::DEBUG::2014-01-28 00:13:14,166::vm::5098::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`e85be056-7715-485c-9cb8-69f9980ed3a3`::event Suspended detail 2 opaque None
>libvirtEventLoop::INFO::2014-01-28 00:13:14,167::vm::4507::vm.Vm::(_onAbnormalStop) vmId=`e85be056-7715-485c-9cb8-69f9980ed3a3`::abnormal vm stop device virtio-disk0 error eother
>libvirtEventLoop::INFO::2014-01-28 00:13:14,167::vm::4507::vm.Vm::(_onAbnormalStop) vmId=`e85be056-7715-485c-9cb8-69f9980ed3a3`::abnormal vm stop device virtio-disk0 error eother
>Thread-969618::INFO::2014-01-28 00:13:14,493::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-969618::INFO::2014-01-28 00:13:14,493::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000527485', 'lastCheck': '9.0', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000345614', 'lastCheck': '3.3', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.3', 'valid': False}}
>Thread-12::DEBUG::2014-01-28 00:13:15,279::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:13:15,323::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-129::DEBUG::2014-01-28 00:13:15,491::iscsi::393::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-129::DEBUG::2014-01-28 00:13:15,493::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:13:15,581::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-969619::DEBUG::2014-01-28 00:13:15,623::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call getCapabilities with () {}
>Thread-969619::DEBUG::2014-01-28 00:13:15,628::utils::555::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-969619::DEBUG::2014-01-28 00:13:15,672::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969619::DEBUG::2014-01-28 00:13:15,713::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:13:15,763::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-969619::DEBUG::2014-01-28 00:13:15,797::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-129::DEBUG::2014-01-28 00:13:15,875::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc
>Thread-129::DEBUG::2014-01-28 00:13:15,891::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-969619::DEBUG::2014-01-28 00:13:15,899::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.2' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:13:15,945::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=562f0160-7b80-42e2-b248-5754455c40fc:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=7e0cc075-eee7-4268-946e-33a79dd08748', 'REMOTE_PATH=172.16.1.5:/HOSTED-ENGINE', 'ROLE=Master', 'SDUUID=562f0160-7b80-42e2-b248-5754455c40fc', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=14825cb0b944c5c2ae9ed025c82e06730a2d91c4']
>Thread-969619::DEBUG::2014-01-28 00:13:15,956::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969619::DEBUG::2014-01-28 00:13:15,962::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.2' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:13:15,999::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-129::INFO::2014-01-28 00:13:16,000::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_imageNS already registered
>Thread-129::INFO::2014-01-28 00:13:16,000::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_volumeNS already registered
>Thread-969619::DEBUG::2014-01-28 00:13:16,009::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969619::DEBUG::2014-01-28 00:13:16,020::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.3' (cwd None)
>Thread-969619::DEBUG::2014-01-28 00:13:16,072::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969619::DEBUG::2014-01-28 00:13:16,078::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.3' (cwd None)
>Thread-969619::DEBUG::2014-01-28 00:13:16,132::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969619::DEBUG::2014-01-28 00:13:16,153::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.10' (cwd None)
>Thread-969619::DEBUG::2014-01-28 00:13:16,207::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969619::DEBUG::2014-01-28 00:13:16,214::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.10' (cwd None)
>Thread-969619::DEBUG::2014-01-28 00:13:16,272::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-129::DEBUG::2014-01-28 00:13:16,294::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-969623::INFO::2014-01-28 00:13:16,373::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-969619::DEBUG::2014-01-28 00:13:16,379::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.11' (cwd None)
>Thread-969623::INFO::2014-01-28 00:13:16,383::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000527485', 'lastCheck': '10.9', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000345614', 'lastCheck': '5.2', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.2', 'valid': False}}
>Thread-129::DEBUG::2014-01-28 00:13:16,439::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n478 bytes (478 B) copied, 0.000672607 s, 711 kB/s\n'; <rc> = 0
>Thread-969619::DEBUG::2014-01-28 00:13:16,451::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969619::DEBUG::2014-01-28 00:13:16,457::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.11' (cwd None)
>Thread-969619::DEBUG::2014-01-28 00:13:16,540::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969619::DEBUG::2014-01-28 00:13:16,600::caps::453::root::(_getKeyPackages) rpm package glusterfs-rdma not found
>Thread-969619::DEBUG::2014-01-28 00:13:16,604::caps::453::root::(_getKeyPackages) rpm package glusterfs-fuse not found
>Thread-969619::DEBUG::2014-01-28 00:13:16,608::caps::453::root::(_getKeyPackages) rpm package gluster-swift not found
>Thread-969619::DEBUG::2014-01-28 00:13:16,617::caps::453::root::(_getKeyPackages) rpm package gluster-swift-object not found
>Thread-969619::DEBUG::2014-01-28 00:13:16,620::caps::453::root::(_getKeyPackages) rpm package glusterfs not found
>Thread-969619::DEBUG::2014-01-28 00:13:16,622::caps::453::root::(_getKeyPackages) rpm package gluster-swift-plugin not found
>Thread-969619::DEBUG::2014-01-28 00:13:16,632::caps::453::root::(_getKeyPackages) rpm package gluster-swift-account not found
>Thread-969619::DEBUG::2014-01-28 00:13:16,635::caps::453::root::(_getKeyPackages) rpm package gluster-swift-proxy not found
>Thread-969619::DEBUG::2014-01-28 00:13:16,638::caps::453::root::(_getKeyPackages) rpm package gluster-swift-doc not found
>Thread-969619::DEBUG::2014-01-28 00:13:16,640::caps::453::root::(_getKeyPackages) rpm package glusterfs-server not found
>Thread-969619::DEBUG::2014-01-28 00:13:16,643::caps::453::root::(_getKeyPackages) rpm package gluster-swift-container not found
>Thread-969619::DEBUG::2014-01-28 00:13:16,645::caps::453::root::(_getKeyPackages) rpm package glusterfs-geo-replication not found
>Thread-969619::DEBUG::2014-01-28 00:13:16,657::BindingXMLRPC::976::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.3.1.el6.x86_64', 'buildtime': 1388745567.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '33.gitf029545.el6', 'buildtime': 1390094004L, 'version': '4.14.0'}, 'qemu-kvm': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.2', 'buildtime': 1387360004L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'mom': {'release': '20140117.gitfd877c5.el6', 'buildtime': 1389966957L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           L5320  @ 1.86GHz', 'hooks': {'before_vm_start': {'50_hostedengine': {'md5': '45dde62155b5412eafbfff5ef265acc2'}}}, 'cpuSockets': '2', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'gateway': '172.16.0.1', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.2', 'vnet0']}, 'os_net': {'iface': 'os_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.10']}, 'r2o_net': {'iface': 'r2o_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}, 'storage_network': {'iface': u'bond0.3', 'addr': '172.16.1.5', 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000', 'netmask': '255.255.255.255', 'bridged': False, 'interface': u'bond0.3', 'ipv6gateway': '::', 'gateway': ''}}, 'bridges': {'ovirtmgmt': {'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '172.16.0.1', 'ports': ['bond0.2', 'vnet0']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}, 'os_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.10']}, 'r2o_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}}, 'uuid': '44454C4C-3100-104D-8046-B9C04F583153', 'lastClientIface': 'ovirtmgmt', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'cb327caa-ca77-4efe-b7b8-63aba3cda84d', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:85', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth2', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:85', 'speed': 1000, 'mtu': '4000'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'e401a6a5-8773-4971-9213-b3c2bb52f16c', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:83', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:83', 'speed': 1000, 'mtu': '4000'}, 'eth0': {'netmask': '', 'addr': '', 'hwaddr': '00:e0:4c:80:13:02', 'cfg': {'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': '00:e0:4c:80:13:02', 'STP': 'no', 'DEVICE': 'eth0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'speed': 1000, 'mtu': '1500'}}, 'software_revision': '33', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,tm2,ssse3,cx16,xtpr,pdcm,dca,lahf_lm,dts,tpr_shadow,vnmi,flexpriority,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': ['eth1', 'eth2'], 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'MTU': '4000', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'BONDING_OPTS': 'miimon=100 mode=balance-rr', 'DEVICE': 'bond0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '32105', 'cpuSpeed': '1861.936', 'version_name': 'Snow Man', 'vlans': {'bond0.2': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'ovirtmgmt', 'VLAN': 'yes', 'MTU': '4000', 'BOOTPROTO': 'none', 'DEVICE': 'bond0.2', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 2, 'mtu': '4000'}, 'bond0.3': {'netmask': '255.255.255.255', 'iface': 'bond0', 'addr': '172.16.1.5', 'cfg': {'VLAN': 'yes', 'DEFROUTE': 'no', 'MTU': '4000', 'NM_CONTROLLED': 'no', 'DEVICE': 'bond0.3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 3, 'mtu': '4000'}, 'eth0.11': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'r2o_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.11', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 11, 'mtu': '1500'}, 'eth0.10': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'os_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.10', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 10, 'mtu': '1500'}}, 'cpuCores': '8', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '8', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '172.16.0.11'}}
>Thread-12::DEBUG::2014-01-28 00:13:17,351::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:13:17,393::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>libvirtEventLoop::INFO::2014-01-28 00:13:17,614::vm::4507::vm.Vm::(_onAbnormalStop) vmId=`b3cde33c-f4c5-4b42-8e9d-820b80abc2fb`::abnormal vm stop device virtio-disk0 error eother
>libvirtEventLoop::DEBUG::2014-01-28 00:13:17,615::vm::5098::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`b3cde33c-f4c5-4b42-8e9d-820b80abc2fb`::event Suspended detail 2 opaque None
>libvirtEventLoop::INFO::2014-01-28 00:13:17,859::vm::4507::vm.Vm::(_onAbnormalStop) vmId=`b3cde33c-f4c5-4b42-8e9d-820b80abc2fb`::abnormal vm stop device virtio-disk0 error eother
>libvirtEventLoop::INFO::2014-01-28 00:13:17,860::vm::4507::vm.Vm::(_onAbnormalStop) vmId=`b3cde33c-f4c5-4b42-8e9d-820b80abc2fb`::abnormal vm stop device virtio-disk0 error eother
>libvirtEventLoop::INFO::2014-01-28 00:13:17,860::vm::4507::vm.Vm::(_onAbnormalStop) vmId=`b3cde33c-f4c5-4b42-8e9d-820b80abc2fb`::abnormal vm stop device virtio-disk0 error eother
>libvirtEventLoop::INFO::2014-01-28 00:13:17,861::vm::4507::vm.Vm::(_onAbnormalStop) vmId=`b3cde33c-f4c5-4b42-8e9d-820b80abc2fb`::abnormal vm stop device virtio-disk0 error eother
>Thread-969625::DEBUG::2014-01-28 00:13:18,562::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call vmGetStats with ('2d425c4c-91b1-4564-b276-f27cc581e6a1',) {}
>Thread-969625::DEBUG::2014-01-28 00:13:18,563::BindingXMLRPC::976::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'root', 'memUsage': '53', 'acpiEnable': 'true', 'guestFQDN': u'engine.melb.example.net', 'pid': '18113', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'/', u'total': '25323499520', u'used': '2592124928', u'fs': u'ext4'}, {u'path': u'/boot', u'total': '507744256', u'used': '55643136', u'fs': u'ext4'}], 'network': {u'vnet0': {'macAddr': '00:16:3e:0a:b7:55', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmType': 'kvm', 'guestName': u'engine.melb.example.net', 'elapsedTime': '610783', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'mem_free': '1970652', u'swap_in': '0', u'pageflt': '633', u'mem_total': '3922728', u'mem_unused': '1436932'}, 'cpuSys': '1.06', 'appsList': [u'ovirt-guest-agent-1.0.8-1.el6', u'kernel-2.6.32-431.el6', u'kernel-2.6.32-431.3.1.el6'], 'guestOs': u'2.6.32-431.3.1.el6.x86_64', 'displayType': 'vnc', 'status': 'Up', 'hash': '2950633987041661405', 'vmId': '2d425c4c-91b1-4564-b276-f27cc581e6a1', 'displayIp': '0', 'displayPort': u'5900', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '26843545600', 'writeLatency': '3946838', 'imageID': 'daa7fa08-1354-4b19-a80f-8694ea63f914', 'flushLatency': '78638', 'readRate': '0.00', 'truesize': '3920576512', 'writeRate': '24859.40'}, 'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.96', 'cpuUser': '12.43', 'lastLogin': 1390828070.405062, 'guestIPs': u'172.16.0.10', 'netIfaces': [{u'hw': u'00:16:3e:0a:b7:55', u'name': u'eth0', u'inet': [u'172.16.0.10'], u'inet6': [u'fe80::216:3eff:fe0a:b755']}]}]}
>Thread-969626::DEBUG::2014-01-28 00:13:18,870::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {}
>Thread-969626::DEBUG::2014-01-28 00:13:18,926::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'OFFLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-12::DEBUG::2014-01-28 00:13:19,409::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:13:19,450::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969628::DEBUG::2014-01-28 00:13:19,970::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call vmGetStats with ('b3cde33c-f4c5-4b42-8e9d-820b80abc2fb',) {}
>Thread-969628::DEBUG::2014-01-28 00:13:19,973::BindingXMLRPC::976::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '43', 'acpiEnable': 'true', 'guestFQDN': u'ipa1.melb.example.net', 'pid': '20640', 'lastLogout': 1390354272.4671021, 'session': 'Unknown', 'displaySecurePort': u'5904', 'timeOffset': 39600L, 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'mem_free': '1181192', u'swap_in': '0', u'pageflt': '127', u'mem_total': '1922356', u'mem_unused': '140484'}, 'balloonInfo': {'balloon_max': '2097152', 'balloon_target': '2097152', 'balloon_cur': '2097152', 'balloon_min': '2097152'}, 'pauseCode': 'EOTHER', 'disksUsage': [{u'path': u'/', u'total': '25893892096', u'used': '2064711680', u'fs': u'ext4'}, {u'path': u'/boot', u'total': '253869056', u'used': '51469312', u'fs': u'ext4'}], 'network': {u'vnet2': {'macAddr': '00:1a:4a:19:42:ce', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet2'}, u'vnet7': {'macAddr': '00:1a:4a:19:42:e1', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet7'}}, 'displayType': 'qxl', 'guestName': u'ipa1.melb.example.net', 'elapsedTime': '474127', 'vmType': 'kvm', 'cpuSys': '0.85', 'appsList': [u'ovirt-guest-agent-1.0.8-1.el6', u'kernel-2.6.32-431.el6', u'kernel-2.6.32-431.3.1.el6'], 'guestOs': u'2.6.32-431.3.1.el6.x86_64', 'status': 'Paused', 'hash': '-3441195388054660225', 'vmId': 'b3cde33c-f4c5-4b42-8e9d-820b80abc2fb', 'displayIp': '0', 'displayPort': u'5903', 'lastUser': 'Unknown', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '2946564096', 'writeLatency': '5224832', 'imageID': '50cabd29-7058-4510-b61f-db21862d99df', 'flushLatency': '2005044', 'readRate': '0.00', 'truesize': '2946564096', 'writeRate': '7775.53'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.43', 'cpuUser': '1.13', 'lastLogin': 1390445304.978739, 'guestIPs': u'10.0.1.10 10.0.1.9 192.168.168.148', 'netIfaces': [{u'hw': u'00:1a:4a:19:42:ce', u'name': u'eth0', u'inet': [u'10.0.1.10', u'10.0.1.9'], u'inet6': [u'fe80::21a:4aff:fe19:42ce']}, {u'hw': u'00:1a:4a:19:42:e1', u'name': u'eth1', u'inet': [u'192.168.168.148'], u'inet6': [u'fe80::21a:4aff:fe19:42e1']}]}]}
>Thread-36::DEBUG::2014-01-28 00:13:21,175::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-36::DEBUG::2014-01-28 00:13:21,240::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n364 bytes (364 B) copied, 0.000342152 s, 1.1 MB/s\n'; <rc> = 0
>Thread-35::ERROR::2014-01-28 00:13:21,261::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain ff3a0c94-11dc-446d-b093-5f68ad81520d monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
>    self.domain.selftest()
>  File "/usr/share/vdsm/storage/nfsSD.py", line 113, in selftest
>    fileSD.FileStorageDomain.selftest(self)
>  File "/usr/share/vdsm/storage/fileSD.py", line 552, in selftest
>    self.oop.os.statvfs(self.domaindir)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 199, in callCrabRPCFunction
>    raise err
>OSError: [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d'
>Thread-12::DEBUG::2014-01-28 00:13:21,472::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:13:21,514::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969630::DEBUG::2014-01-28 00:13:23,132::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-969630::INFO::2014-01-28 00:13:23,133::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-969630::INFO::2014-01-28 00:13:23,134::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-969631::DEBUG::2014-01-28 00:13:23,140::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-969631::INFO::2014-01-28 00:13:23,141::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-969631::INFO::2014-01-28 00:13:23,148::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': 'ff3a0c94-11dc-446d-b093-5f68ad81520d', 'name': 'NextDC_M1', 'version': '3', 'domains': 'ff3a0c94-11dc-446d-b093-5f68ad81520d:Active,436c344c-bc57-441b-b311-c9595c6039e1:Active', 'pool_status': 'connected', 'isoprefix': '/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/images/11111111-1111-1111-1111-111111111111', 'type': 'GLUSTERFS', 'master_ver': 1, 'lver': 1}, 'dominfo': {'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '436c344c-bc57-441b-b311-c9595c6039e1': {'status': 'Active', 'diskfree': '21444952064', 'isoprefix': '/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '25323634688', 'version': 0}}}
>Thread-12::DEBUG::2014-01-28 00:13:23,530::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:13:23,573::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969632::DEBUG::2014-01-28 00:13:23,736::BindingXMLRPC::159::vds::(wrapper) client [127.0.0.1]
>Thread-969632::INFO::2014-01-28 00:13:23,737::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='7e0cc075-eee7-4268-946e-33a79dd08748', conList=[{'port': '', 'connection': '172.16.1.5:/HOSTED-ENGINE', 'iqn': '', 'portal': '', 'user': 'kvm', 'protocol_version': '3', 'password': '******', 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}], options=None)
>Thread-969632::DEBUG::2014-01-28 00:13:23,742::hsm::2273::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE
>Thread-969632::DEBUG::2014-01-28 00:13:23,744::hsm::2297::Storage.HSM::(__prefetchDomains) Found SD uuids: ('562f0160-7b80-42e2-b248-5754455c40fc',)
>Thread-969632::DEBUG::2014-01-28 00:13:23,745::hsm::2353::Storage.HSM::(connectStorageServer) knownSDs: {562f0160-7b80-42e2-b248-5754455c40fc: storage.nfsSD.findDomain, ff3a0c94-11dc-446d-b093-5f68ad81520d: storage.glusterSD.findDomain, 436c344c-bc57-441b-b311-c9595c6039e1: storage.nfsSD.findDomain}
>Thread-969632::INFO::2014-01-28 00:13:23,745::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}]}
>Thread-969633::DEBUG::2014-01-28 00:13:24,016::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.11]
>Thread-969633::INFO::2014-01-28 00:13:24,018::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-969633::INFO::2014-01-28 00:13:24,018::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000672607', 'lastCheck': '7.5', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000342152', 'lastCheck': '2.8', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '2.8', 'valid': False}}
>Thread-969634::DEBUG::2014-01-28 00:13:24,034::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {}
>Thread-969634::DEBUG::2014-01-28 00:13:24,072::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'OFFLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-969635::INFO::2014-01-28 00:13:24,503::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-969635::INFO::2014-01-28 00:13:24,504::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000672607', 'lastCheck': '7.9', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000342152', 'lastCheck': '3.3', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.2', 'valid': False}}
>Thread-969636::DEBUG::2014-01-28 00:13:25,571::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call getCapabilities with () {}
>Thread-969636::DEBUG::2014-01-28 00:13:25,573::utils::555::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:13:25,613::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-969636::DEBUG::2014-01-28 00:13:25,660::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-12::DEBUG::2014-01-28 00:13:25,675::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969636::DEBUG::2014-01-28 00:13:25,724::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-969636::DEBUG::2014-01-28 00:13:25,768::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969636::DEBUG::2014-01-28 00:13:25,790::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.2' (cwd None)
>Thread-969636::DEBUG::2014-01-28 00:13:25,840::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969636::DEBUG::2014-01-28 00:13:25,841::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.2' (cwd None)
>Thread-969636::DEBUG::2014-01-28 00:13:25,881::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969636::DEBUG::2014-01-28 00:13:25,885::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.3' (cwd None)
>Thread-969636::DEBUG::2014-01-28 00:13:25,926::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969636::DEBUG::2014-01-28 00:13:25,927::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.3' (cwd None)
>Thread-969636::DEBUG::2014-01-28 00:13:25,972::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969636::DEBUG::2014-01-28 00:13:25,975::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.10' (cwd None)
>Thread-969636::DEBUG::2014-01-28 00:13:26,017::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969636::DEBUG::2014-01-28 00:13:26,018::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.10' (cwd None)
>Thread-969636::DEBUG::2014-01-28 00:13:26,061::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969636::DEBUG::2014-01-28 00:13:26,066::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.11' (cwd None)
>Thread-969636::DEBUG::2014-01-28 00:13:26,114::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969636::DEBUG::2014-01-28 00:13:26,115::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.11' (cwd None)
>Thread-969636::DEBUG::2014-01-28 00:13:26,159::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969636::DEBUG::2014-01-28 00:13:26,183::caps::453::root::(_getKeyPackages) rpm package glusterfs-rdma not found
>Thread-969636::DEBUG::2014-01-28 00:13:26,184::caps::453::root::(_getKeyPackages) rpm package glusterfs-fuse not found
>Thread-969636::DEBUG::2014-01-28 00:13:26,185::caps::453::root::(_getKeyPackages) rpm package gluster-swift not found
>Thread-969636::DEBUG::2014-01-28 00:13:26,191::caps::453::root::(_getKeyPackages) rpm package gluster-swift-object not found
>Thread-969636::DEBUG::2014-01-28 00:13:26,192::caps::453::root::(_getKeyPackages) rpm package glusterfs not found
>Thread-969636::DEBUG::2014-01-28 00:13:26,192::caps::453::root::(_getKeyPackages) rpm package gluster-swift-plugin not found
>Thread-969636::DEBUG::2014-01-28 00:13:26,199::caps::453::root::(_getKeyPackages) rpm package gluster-swift-account not found
>Thread-969636::DEBUG::2014-01-28 00:13:26,200::caps::453::root::(_getKeyPackages) rpm package gluster-swift-proxy not found
>Thread-969636::DEBUG::2014-01-28 00:13:26,200::caps::453::root::(_getKeyPackages) rpm package gluster-swift-doc not found
>Thread-969636::DEBUG::2014-01-28 00:13:26,200::caps::453::root::(_getKeyPackages) rpm package glusterfs-server not found
>Thread-969636::DEBUG::2014-01-28 00:13:26,201::caps::453::root::(_getKeyPackages) rpm package gluster-swift-container not found
>Thread-969636::DEBUG::2014-01-28 00:13:26,201::caps::453::root::(_getKeyPackages) rpm package glusterfs-geo-replication not found
>Thread-969636::DEBUG::2014-01-28 00:13:26,207::BindingXMLRPC::976::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.3.1.el6.x86_64', 'buildtime': 1388745567.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '33.gitf029545.el6', 'buildtime': 1390094004L, 'version': '4.14.0'}, 'qemu-kvm': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.2', 'buildtime': 1387360004L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'mom': {'release': '20140117.gitfd877c5.el6', 'buildtime': 1389966957L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           L5320  @ 1.86GHz', 'hooks': {'before_vm_start': {'50_hostedengine': {'md5': '45dde62155b5412eafbfff5ef265acc2'}}}, 'cpuSockets': '2', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'gateway': '172.16.0.1', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.2', 'vnet0']}, 'os_net': {'iface': 'os_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.10']}, 'r2o_net': {'iface': 'r2o_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}, 'storage_network': {'iface': u'bond0.3', 'addr': '172.16.1.5', 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000', 'netmask': '255.255.255.255', 'bridged': False, 'interface': u'bond0.3', 'ipv6gateway': '::', 'gateway': ''}}, 'bridges': {'ovirtmgmt': {'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '172.16.0.1', 'ports': ['bond0.2', 'vnet0']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}, 'os_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.10']}, 'r2o_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}}, 'uuid': '44454C4C-3100-104D-8046-B9C04F583153', 'lastClientIface': 'ovirtmgmt', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'cb327caa-ca77-4efe-b7b8-63aba3cda84d', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:85', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth2', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:85', 'speed': 1000, 'mtu': '4000'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'e401a6a5-8773-4971-9213-b3c2bb52f16c', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:83', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:83', 'speed': 1000, 'mtu': '4000'}, 'eth0': {'netmask': '', 'addr': '', 'hwaddr': '00:e0:4c:80:13:02', 'cfg': {'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': '00:e0:4c:80:13:02', 'STP': 'no', 'DEVICE': 'eth0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'speed': 1000, 'mtu': '1500'}}, 'software_revision': '33', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,tm2,ssse3,cx16,xtpr,pdcm,dca,lahf_lm,dts,tpr_shadow,vnmi,flexpriority,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': ['eth1', 'eth2'], 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'MTU': '4000', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'BONDING_OPTS': 'miimon=100 mode=balance-rr', 'DEVICE': 'bond0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '32105', 'cpuSpeed': '1861.936', 'version_name': 'Snow Man', 'vlans': {'bond0.2': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'ovirtmgmt', 'VLAN': 'yes', 'MTU': '4000', 'BOOTPROTO': 'none', 'DEVICE': 'bond0.2', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 2, 'mtu': '4000'}, 'bond0.3': {'netmask': '255.255.255.255', 'iface': 'bond0', 'addr': '172.16.1.5', 'cfg': {'VLAN': 'yes', 'DEFROUTE': 'no', 'MTU': '4000', 'NM_CONTROLLED': 'no', 'DEVICE': 'bond0.3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 3, 'mtu': '4000'}, 'eth0.11': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'r2o_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.11', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 11, 'mtu': '1500'}, 'eth0.10': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'os_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.10', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 10, 'mtu': '1500'}}, 'cpuCores': '8', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '8', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '172.16.0.11'}}
>Thread-129::DEBUG::2014-01-28 00:13:26,564::iscsi::393::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-129::DEBUG::2014-01-28 00:13:26,564::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:13:26,625::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:13:26,775::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-129::DEBUG::2014-01-28 00:13:26,804::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc
>Thread-129::DEBUG::2014-01-28 00:13:26,806::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-129::DEBUG::2014-01-28 00:13:26,816::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=562f0160-7b80-42e2-b248-5754455c40fc:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=7e0cc075-eee7-4268-946e-33a79dd08748', 'REMOTE_PATH=172.16.1.5:/HOSTED-ENGINE', 'ROLE=Master', 'SDUUID=562f0160-7b80-42e2-b248-5754455c40fc', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=14825cb0b944c5c2ae9ed025c82e06730a2d91c4']
>Thread-129::DEBUG::2014-01-28 00:13:26,819::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-129::INFO::2014-01-28 00:13:26,819::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_imageNS already registered
>Thread-129::INFO::2014-01-28 00:13:26,819::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_volumeNS already registered
>Thread-129::DEBUG::2014-01-28 00:13:26,832::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:13:26,875::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n478 bytes (478 B) copied, 0.000490921 s, 974 kB/s\n'; <rc> = 0
>Thread-12::DEBUG::2014-01-28 00:13:27,723::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:13:27,766::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969641::DEBUG::2014-01-28 00:13:28,570::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call vmGetStats with ('2d425c4c-91b1-4564-b276-f27cc581e6a1',) {}
>Thread-969641::DEBUG::2014-01-28 00:13:28,572::BindingXMLRPC::976::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'root', 'memUsage': '53', 'acpiEnable': 'true', 'guestFQDN': u'engine.melb.example.net', 'pid': '18113', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'/', u'total': '25323499520', u'used': '2592124928', u'fs': u'ext4'}, {u'path': u'/boot', u'total': '507744256', u'used': '55643136', u'fs': u'ext4'}], 'network': {u'vnet0': {'macAddr': '00:16:3e:0a:b7:55', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmType': 'kvm', 'guestName': u'engine.melb.example.net', 'elapsedTime': '610793', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'mem_free': '1968516', u'swap_in': '0', u'pageflt': '146', u'mem_total': '3922728', u'mem_unused': '1434760'}, 'cpuSys': '1.06', 'appsList': [u'ovirt-guest-agent-1.0.8-1.el6', u'kernel-2.6.32-431.el6', u'kernel-2.6.32-431.3.1.el6'], 'guestOs': u'2.6.32-431.3.1.el6.x86_64', 'displayType': 'vnc', 'status': 'Up', 'hash': '2950633987041661405', 'vmId': '2d425c4c-91b1-4564-b276-f27cc581e6a1', 'displayIp': '0', 'displayPort': u'5900', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '26843545600', 'writeLatency': '3946838', 'imageID': 'daa7fa08-1354-4b19-a80f-8694ea63f914', 'flushLatency': '78638', 'readRate': '0.00', 'truesize': '3920576512', 'writeRate': '24859.40'}, 'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.95', 'cpuUser': '12.57', 'lastLogin': 1390828070.405062, 'guestIPs': u'172.16.0.10', 'netIfaces': [{u'hw': u'00:16:3e:0a:b7:55', u'name': u'eth0', u'inet': [u'172.16.0.10'], u'inet6': [u'fe80::216:3eff:fe0a:b755']}]}]}
>Thread-12::DEBUG::2014-01-28 00:13:29,783::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-969643::DEBUG::2014-01-28 00:13:31,067::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {}
>Thread-12::DEBUG::2014-01-28 00:13:31,074::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969643::DEBUG::2014-01-28 00:13:31,127::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'OFFLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-36::DEBUG::2014-01-28 00:13:31,318::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-35::ERROR::2014-01-28 00:13:31,358::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain ff3a0c94-11dc-446d-b093-5f68ad81520d monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
>    self.domain.selftest()
>  File "/usr/share/vdsm/storage/nfsSD.py", line 113, in selftest
>    fileSD.FileStorageDomain.selftest(self)
>  File "/usr/share/vdsm/storage/fileSD.py", line 552, in selftest
>    self.oop.os.statvfs(self.domaindir)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 199, in callCrabRPCFunction
>    raise err
>OSError: [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d'
>Thread-36::DEBUG::2014-01-28 00:13:31,363::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n364 bytes (364 B) copied, 0.00034749 s, 1.0 MB/s\n'; <rc> = 0
>Thread-12::DEBUG::2014-01-28 00:13:33,093::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:13:33,136::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969644::DEBUG::2014-01-28 00:13:33,240::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-969644::INFO::2014-01-28 00:13:33,242::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-969644::INFO::2014-01-28 00:13:33,243::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-969645::DEBUG::2014-01-28 00:13:33,251::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-969645::INFO::2014-01-28 00:13:33,252::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-969645::INFO::2014-01-28 00:13:33,260::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': 'ff3a0c94-11dc-446d-b093-5f68ad81520d', 'name': 'NextDC_M1', 'version': '3', 'domains': 'ff3a0c94-11dc-446d-b093-5f68ad81520d:Active,436c344c-bc57-441b-b311-c9595c6039e1:Active', 'pool_status': 'connected', 'isoprefix': '/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/images/11111111-1111-1111-1111-111111111111', 'type': 'GLUSTERFS', 'master_ver': 1, 'lver': 1}, 'dominfo': {'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '436c344c-bc57-441b-b311-c9595c6039e1': {'status': 'Active', 'diskfree': '21444952064', 'isoprefix': '/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '25323634688', 'version': 0}}}
>VM Channels Listener::DEBUG::2014-01-28 00:13:34,044::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 104.
>Thread-969646::INFO::2014-01-28 00:13:34,095::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-969646::INFO::2014-01-28 00:13:34,095::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000490921', 'lastCheck': '7.2', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00034749', 'lastCheck': '2.7', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '2.7', 'valid': False}}
>Thread-969648::DEBUG::2014-01-28 00:13:34,366::BindingXMLRPC::159::vds::(wrapper) client [127.0.0.1]
>Thread-969648::INFO::2014-01-28 00:13:34,368::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='7e0cc075-eee7-4268-946e-33a79dd08748', conList=[{'port': '', 'connection': '172.16.1.5:/HOSTED-ENGINE', 'iqn': '', 'portal': '', 'user': 'kvm', 'protocol_version': '3', 'password': '******', 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}], options=None)
>Thread-969648::DEBUG::2014-01-28 00:13:34,372::hsm::2273::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE
>Thread-969648::DEBUG::2014-01-28 00:13:34,374::hsm::2297::Storage.HSM::(__prefetchDomains) Found SD uuids: ('562f0160-7b80-42e2-b248-5754455c40fc',)
>Thread-969648::DEBUG::2014-01-28 00:13:34,374::hsm::2353::Storage.HSM::(connectStorageServer) knownSDs: {562f0160-7b80-42e2-b248-5754455c40fc: storage.nfsSD.findDomain, ff3a0c94-11dc-446d-b093-5f68ad81520d: storage.glusterSD.findDomain, 436c344c-bc57-441b-b311-c9595c6039e1: storage.nfsSD.findDomain}
>Thread-969648::INFO::2014-01-28 00:13:34,374::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}]}
>Thread-969649::INFO::2014-01-28 00:13:34,510::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-969649::INFO::2014-01-28 00:13:34,511::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000490921', 'lastCheck': '7.6', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00034749', 'lastCheck': '3.1', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.2', 'valid': False}}
>Thread-969650::DEBUG::2014-01-28 00:13:34,640::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.11]
>Thread-969650::INFO::2014-01-28 00:13:34,641::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-969650::INFO::2014-01-28 00:13:34,641::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000490921', 'lastCheck': '7.8', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00034749', 'lastCheck': '3.3', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.3', 'valid': False}}
>VM Channels Listener::DEBUG::2014-01-28 00:13:35,046::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 138.
>Thread-12::DEBUG::2014-01-28 00:13:35,153::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:13:35,193::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969651::DEBUG::2014-01-28 00:13:35,581::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call getCapabilities with () {}
>Thread-969651::DEBUG::2014-01-28 00:13:35,583::utils::555::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-969651::DEBUG::2014-01-28 00:13:35,622::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969651::DEBUG::2014-01-28 00:13:35,660::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-969651::DEBUG::2014-01-28 00:13:35,701::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969651::DEBUG::2014-01-28 00:13:35,721::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.2' (cwd None)
>Thread-969651::DEBUG::2014-01-28 00:13:35,762::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969651::DEBUG::2014-01-28 00:13:35,763::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.2' (cwd None)
>Thread-969651::DEBUG::2014-01-28 00:13:35,804::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969651::DEBUG::2014-01-28 00:13:35,808::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.3' (cwd None)
>Thread-969651::DEBUG::2014-01-28 00:13:35,849::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969651::DEBUG::2014-01-28 00:13:35,850::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.3' (cwd None)
>Thread-969651::DEBUG::2014-01-28 00:13:35,893::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969651::DEBUG::2014-01-28 00:13:35,896::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.10' (cwd None)
>Thread-969651::DEBUG::2014-01-28 00:13:35,936::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969651::DEBUG::2014-01-28 00:13:35,937::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.10' (cwd None)
>Thread-969651::DEBUG::2014-01-28 00:13:35,979::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969651::DEBUG::2014-01-28 00:13:35,984::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.11' (cwd None)
>Thread-969651::DEBUG::2014-01-28 00:13:36,025::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969651::DEBUG::2014-01-28 00:13:36,026::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.11' (cwd None)
>Thread-969651::DEBUG::2014-01-28 00:13:36,068::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969651::DEBUG::2014-01-28 00:13:36,080::caps::453::root::(_getKeyPackages) rpm package glusterfs-rdma not found
>Thread-969651::DEBUG::2014-01-28 00:13:36,080::caps::453::root::(_getKeyPackages) rpm package glusterfs-fuse not found
>Thread-969651::DEBUG::2014-01-28 00:13:36,082::caps::453::root::(_getKeyPackages) rpm package gluster-swift not found
>Thread-969651::DEBUG::2014-01-28 00:13:36,088::caps::453::root::(_getKeyPackages) rpm package gluster-swift-object not found
>Thread-969651::DEBUG::2014-01-28 00:13:36,088::caps::453::root::(_getKeyPackages) rpm package glusterfs not found
>Thread-969651::DEBUG::2014-01-28 00:13:36,089::caps::453::root::(_getKeyPackages) rpm package gluster-swift-plugin not found
>Thread-969651::DEBUG::2014-01-28 00:13:36,096::caps::453::root::(_getKeyPackages) rpm package gluster-swift-account not found
>Thread-969651::DEBUG::2014-01-28 00:13:36,096::caps::453::root::(_getKeyPackages) rpm package gluster-swift-proxy not found
>Thread-969651::DEBUG::2014-01-28 00:13:36,097::caps::453::root::(_getKeyPackages) rpm package gluster-swift-doc not found
>Thread-969651::DEBUG::2014-01-28 00:13:36,097::caps::453::root::(_getKeyPackages) rpm package glusterfs-server not found
>Thread-969651::DEBUG::2014-01-28 00:13:36,097::caps::453::root::(_getKeyPackages) rpm package gluster-swift-container not found
>Thread-969651::DEBUG::2014-01-28 00:13:36,098::caps::453::root::(_getKeyPackages) rpm package glusterfs-geo-replication not found
>Thread-969651::DEBUG::2014-01-28 00:13:36,103::BindingXMLRPC::976::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.3.1.el6.x86_64', 'buildtime': 1388745567.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '33.gitf029545.el6', 'buildtime': 1390094004L, 'version': '4.14.0'}, 'qemu-kvm': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.2', 'buildtime': 1387360004L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'mom': {'release': '20140117.gitfd877c5.el6', 'buildtime': 1389966957L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           L5320  @ 1.86GHz', 'hooks': {'before_vm_start': {'50_hostedengine': {'md5': '45dde62155b5412eafbfff5ef265acc2'}}}, 'cpuSockets': '2', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'gateway': '172.16.0.1', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.2', 'vnet0']}, 'os_net': {'iface': 'os_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.10']}, 'r2o_net': {'iface': 'r2o_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}, 'storage_network': {'iface': u'bond0.3', 'addr': '172.16.1.5', 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000', 'netmask': '255.255.255.255', 'bridged': False, 'interface': u'bond0.3', 'ipv6gateway': '::', 'gateway': ''}}, 'bridges': {'ovirtmgmt': {'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '172.16.0.1', 'ports': ['bond0.2', 'vnet0']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}, 'os_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.10']}, 'r2o_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}}, 'uuid': '44454C4C-3100-104D-8046-B9C04F583153', 'lastClientIface': 'ovirtmgmt', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'cb327caa-ca77-4efe-b7b8-63aba3cda84d', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:85', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth2', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:85', 'speed': 1000, 'mtu': '4000'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'e401a6a5-8773-4971-9213-b3c2bb52f16c', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:83', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:83', 'speed': 1000, 'mtu': '4000'}, 'eth0': {'netmask': '', 'addr': '', 'hwaddr': '00:e0:4c:80:13:02', 'cfg': {'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': '00:e0:4c:80:13:02', 'STP': 'no', 'DEVICE': 'eth0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'speed': 1000, 'mtu': '1500'}}, 'software_revision': '33', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,tm2,ssse3,cx16,xtpr,pdcm,dca,lahf_lm,dts,tpr_shadow,vnmi,flexpriority,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': ['eth1', 'eth2'], 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'MTU': '4000', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'BONDING_OPTS': 'miimon=100 mode=balance-rr', 'DEVICE': 'bond0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '32105', 'cpuSpeed': '1861.936', 'version_name': 'Snow Man', 'vlans': {'bond0.2': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'ovirtmgmt', 'VLAN': 'yes', 'MTU': '4000', 'BOOTPROTO': 'none', 'DEVICE': 'bond0.2', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 2, 'mtu': '4000'}, 'bond0.3': {'netmask': '255.255.255.255', 'iface': 'bond0', 'addr': '172.16.1.5', 'cfg': {'VLAN': 'yes', 'DEFROUTE': 'no', 'MTU': '4000', 'NM_CONTROLLED': 'no', 'DEVICE': 'bond0.3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 3, 'mtu': '4000'}, 'eth0.11': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'r2o_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.11', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 11, 'mtu': '1500'}, 'eth0.10': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'os_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.10', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 10, 'mtu': '1500'}}, 'cpuCores': '8', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '8', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '172.16.0.11'}}
>Thread-969652::DEBUG::2014-01-28 00:13:36,276::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {}
>Thread-969652::DEBUG::2014-01-28 00:13:36,315::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'OFFLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-129::DEBUG::2014-01-28 00:13:36,884::iscsi::393::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-129::DEBUG::2014-01-28 00:13:36,886::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:13:36,959::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>VM Channels Listener::DEBUG::2014-01-28 00:13:37,067::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 70.
>Thread-129::DEBUG::2014-01-28 00:13:37,111::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-129::DEBUG::2014-01-28 00:13:37,134::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc
>Thread-129::DEBUG::2014-01-28 00:13:37,136::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-129::DEBUG::2014-01-28 00:13:37,147::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=562f0160-7b80-42e2-b248-5754455c40fc:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=7e0cc075-eee7-4268-946e-33a79dd08748', 'REMOTE_PATH=172.16.1.5:/HOSTED-ENGINE', 'ROLE=Master', 'SDUUID=562f0160-7b80-42e2-b248-5754455c40fc', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=14825cb0b944c5c2ae9ed025c82e06730a2d91c4']
>Thread-129::DEBUG::2014-01-28 00:13:37,149::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-129::INFO::2014-01-28 00:13:37,150::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_imageNS already registered
>Thread-129::INFO::2014-01-28 00:13:37,150::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_volumeNS already registered
>Thread-129::DEBUG::2014-01-28 00:13:37,164::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:13:37,205::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n478 bytes (478 B) copied, 0.000870163 s, 549 kB/s\n'; <rc> = 0
>Thread-12::DEBUG::2014-01-28 00:13:37,216::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:13:37,262::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>VM Channels Listener::DEBUG::2014-01-28 00:13:38,071::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 130.
>Thread-969657::DEBUG::2014-01-28 00:13:38,572::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call vmGetStats with ('2d425c4c-91b1-4564-b276-f27cc581e6a1',) {}
>Thread-969657::DEBUG::2014-01-28 00:13:38,573::BindingXMLRPC::976::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'root', 'memUsage': '53', 'acpiEnable': 'true', 'guestFQDN': u'engine.melb.example.net', 'pid': '18113', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'/', u'total': '25323499520', u'used': '2592124928', u'fs': u'ext4'}, {u'path': u'/boot', u'total': '507744256', u'used': '55643136', u'fs': u'ext4'}], 'network': {u'vnet0': {'macAddr': '00:16:3e:0a:b7:55', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmType': 'kvm', 'guestName': u'engine.melb.example.net', 'elapsedTime': '610803', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'mem_free': '1968788', u'swap_in': '0', u'pageflt': '143', u'mem_total': '3922728', u'mem_unused': '1435008'}, 'cpuSys': '1.06', 'appsList': [u'ovirt-guest-agent-1.0.8-1.el6', u'kernel-2.6.32-431.el6', u'kernel-2.6.32-431.3.1.el6'], 'guestOs': u'2.6.32-431.3.1.el6.x86_64', 'displayType': 'vnc', 'status': 'Up', 'hash': '2950633987041661405', 'vmId': '2d425c4c-91b1-4564-b276-f27cc581e6a1', 'displayIp': '0', 'displayPort': u'5900', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '26843545600', 'writeLatency': '3946838', 'imageID': 'daa7fa08-1354-4b19-a80f-8694ea63f914', 'flushLatency': '78638', 'readRate': '0.00', 'truesize': '3920576512', 'writeRate': '24859.40'}, 'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.95', 'cpuUser': '12.57', 'lastLogin': 1390828070.405062, 'guestIPs': u'172.16.0.10', 'netIfaces': [{u'hw': u'00:16:3e:0a:b7:55', u'name': u'eth0', u'inet': [u'172.16.0.10'], u'inet6': [u'fe80::216:3eff:fe0a:b755']}]}]}
>Thread-12::DEBUG::2014-01-28 00:13:39,282::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:13:39,324::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>VM Channels Listener::DEBUG::2014-01-28 00:13:40,074::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 132.
>Thread-12::DEBUG::2014-01-28 00:13:41,341::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:13:41,442::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-35::ERROR::2014-01-28 00:13:41,450::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain ff3a0c94-11dc-446d-b093-5f68ad81520d monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
>    self.domain.selftest()
>  File "/usr/share/vdsm/storage/nfsSD.py", line 113, in selftest
>    fileSD.FileStorageDomain.selftest(self)
>  File "/usr/share/vdsm/storage/fileSD.py", line 552, in selftest
>    self.oop.os.statvfs(self.domaindir)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 199, in callCrabRPCFunction
>    raise err
>OSError: [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d'
>Thread-969659::DEBUG::2014-01-28 00:13:41,480::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {}
>Thread-36::DEBUG::2014-01-28 00:13:41,522::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-969659::DEBUG::2014-01-28 00:13:41,561::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'OFFLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-36::DEBUG::2014-01-28 00:13:41,617::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n364 bytes (364 B) copied, 0.000328037 s, 1.1 MB/s\n'; <rc> = 0
>Thread-969660::DEBUG::2014-01-28 00:13:43,303::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10] flowID [221e4d0a]
>Thread-969660::INFO::2014-01-28 00:13:43,305::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-969660::INFO::2014-01-28 00:13:43,305::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-969661::DEBUG::2014-01-28 00:13:43,312::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10] flowID [221e4d0a]
>Thread-969661::INFO::2014-01-28 00:13:43,313::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-969661::INFO::2014-01-28 00:13:43,320::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': 'ff3a0c94-11dc-446d-b093-5f68ad81520d', 'name': 'NextDC_M1', 'version': '3', 'domains': 'ff3a0c94-11dc-446d-b093-5f68ad81520d:Active,436c344c-bc57-441b-b311-c9595c6039e1:Active', 'pool_status': 'connected', 'isoprefix': '/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/images/11111111-1111-1111-1111-111111111111', 'type': 'GLUSTERFS', 'master_ver': 1, 'lver': 1}, 'dominfo': {'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '436c344c-bc57-441b-b311-c9595c6039e1': {'status': 'Active', 'diskfree': '21444952064', 'isoprefix': '/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '25323634688', 'version': 0}}}
>Thread-12::DEBUG::2014-01-28 00:13:43,608::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:13:43,652::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969663::INFO::2014-01-28 00:13:44,521::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-969663::INFO::2014-01-28 00:13:44,522::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000870163', 'lastCheck': '7.3', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000328037', 'lastCheck': '2.9', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.1', 'valid': False}}
>Thread-969664::DEBUG::2014-01-28 00:13:44,964::BindingXMLRPC::159::vds::(wrapper) client [127.0.0.1]
>Thread-969664::INFO::2014-01-28 00:13:44,965::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='7e0cc075-eee7-4268-946e-33a79dd08748', conList=[{'port': '', 'connection': '172.16.1.5:/HOSTED-ENGINE', 'iqn': '', 'portal': '', 'user': 'kvm', 'protocol_version': '3', 'password': '******', 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}], options=None)
>Thread-969664::DEBUG::2014-01-28 00:13:44,970::hsm::2273::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE
>Thread-969664::DEBUG::2014-01-28 00:13:44,972::hsm::2297::Storage.HSM::(__prefetchDomains) Found SD uuids: ('562f0160-7b80-42e2-b248-5754455c40fc',)
>Thread-969664::DEBUG::2014-01-28 00:13:44,972::hsm::2353::Storage.HSM::(connectStorageServer) knownSDs: {562f0160-7b80-42e2-b248-5754455c40fc: storage.nfsSD.findDomain, ff3a0c94-11dc-446d-b093-5f68ad81520d: storage.glusterSD.findDomain, 436c344c-bc57-441b-b311-c9595c6039e1: storage.nfsSD.findDomain}
>Thread-969664::INFO::2014-01-28 00:13:44,973::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}]}
>Thread-969665::DEBUG::2014-01-28 00:13:45,216::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.11]
>Thread-969665::INFO::2014-01-28 00:13:45,224::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-969665::INFO::2014-01-28 00:13:45,225::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000870163', 'lastCheck': '8.0', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000328037', 'lastCheck': '3.6', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.8', 'valid': False}}
>Thread-969666::DEBUG::2014-01-28 00:13:45,590::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call getCapabilities with () {}
>Thread-969666::DEBUG::2014-01-28 00:13:45,592::utils::555::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-969666::DEBUG::2014-01-28 00:13:45,631::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969666::DEBUG::2014-01-28 00:13:45,655::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:13:45,691::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-969666::DEBUG::2014-01-28 00:13:45,773::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-12::DEBUG::2014-01-28 00:13:45,773::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969666::DEBUG::2014-01-28 00:13:45,875::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.2' (cwd None)
>Thread-969666::DEBUG::2014-01-28 00:13:45,914::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969666::DEBUG::2014-01-28 00:13:45,915::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.2' (cwd None)
>Thread-969666::DEBUG::2014-01-28 00:13:45,954::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969666::DEBUG::2014-01-28 00:13:45,957::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.3' (cwd None)
>Thread-969666::DEBUG::2014-01-28 00:13:45,997::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969666::DEBUG::2014-01-28 00:13:45,998::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.3' (cwd None)
>Thread-969666::DEBUG::2014-01-28 00:13:46,040::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969666::DEBUG::2014-01-28 00:13:46,043::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.10' (cwd None)
>VM Channels Listener::DEBUG::2014-01-28 00:13:46,085::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 97.
>Thread-969666::DEBUG::2014-01-28 00:13:46,091::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969666::DEBUG::2014-01-28 00:13:46,092::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.10' (cwd None)
>Thread-969666::DEBUG::2014-01-28 00:13:46,131::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969666::DEBUG::2014-01-28 00:13:46,136::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.11' (cwd None)
>Thread-969666::DEBUG::2014-01-28 00:13:46,176::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969666::DEBUG::2014-01-28 00:13:46,177::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.11' (cwd None)
>Thread-969666::DEBUG::2014-01-28 00:13:46,217::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969666::DEBUG::2014-01-28 00:13:46,227::caps::453::root::(_getKeyPackages) rpm package glusterfs-rdma not found
>Thread-969666::DEBUG::2014-01-28 00:13:46,228::caps::453::root::(_getKeyPackages) rpm package glusterfs-fuse not found
>Thread-969666::DEBUG::2014-01-28 00:13:46,230::caps::453::root::(_getKeyPackages) rpm package gluster-swift not found
>Thread-969666::DEBUG::2014-01-28 00:13:46,235::caps::453::root::(_getKeyPackages) rpm package gluster-swift-object not found
>Thread-969666::DEBUG::2014-01-28 00:13:46,236::caps::453::root::(_getKeyPackages) rpm package glusterfs not found
>Thread-969666::DEBUG::2014-01-28 00:13:46,236::caps::453::root::(_getKeyPackages) rpm package gluster-swift-plugin not found
>Thread-969666::DEBUG::2014-01-28 00:13:46,243::caps::453::root::(_getKeyPackages) rpm package gluster-swift-account not found
>Thread-969666::DEBUG::2014-01-28 00:13:46,244::caps::453::root::(_getKeyPackages) rpm package gluster-swift-proxy not found
>Thread-969666::DEBUG::2014-01-28 00:13:46,244::caps::453::root::(_getKeyPackages) rpm package gluster-swift-doc not found
>Thread-969666::DEBUG::2014-01-28 00:13:46,245::caps::453::root::(_getKeyPackages) rpm package glusterfs-server not found
>Thread-969666::DEBUG::2014-01-28 00:13:46,245::caps::453::root::(_getKeyPackages) rpm package gluster-swift-container not found
>Thread-969666::DEBUG::2014-01-28 00:13:46,245::caps::453::root::(_getKeyPackages) rpm package glusterfs-geo-replication not found
>Thread-969666::DEBUG::2014-01-28 00:13:46,259::BindingXMLRPC::976::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.3.1.el6.x86_64', 'buildtime': 1388745567.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '33.gitf029545.el6', 'buildtime': 1390094004L, 'version': '4.14.0'}, 'qemu-kvm': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.2', 'buildtime': 1387360004L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'mom': {'release': '20140117.gitfd877c5.el6', 'buildtime': 1389966957L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           L5320  @ 1.86GHz', 'hooks': {'before_vm_start': {'50_hostedengine': {'md5': '45dde62155b5412eafbfff5ef265acc2'}}}, 'cpuSockets': '2', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'gateway': '172.16.0.1', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.2', 'vnet0']}, 'os_net': {'iface': 'os_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.10']}, 'r2o_net': {'iface': 'r2o_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}, 'storage_network': {'iface': u'bond0.3', 'addr': '172.16.1.5', 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000', 'netmask': '255.255.255.255', 'bridged': False, 'interface': u'bond0.3', 'ipv6gateway': '::', 'gateway': ''}}, 'bridges': {'ovirtmgmt': {'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '172.16.0.1', 'ports': ['bond0.2', 'vnet0']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}, 'os_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.10']}, 'r2o_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}}, 'uuid': '44454C4C-3100-104D-8046-B9C04F583153', 'lastClientIface': 'ovirtmgmt', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'cb327caa-ca77-4efe-b7b8-63aba3cda84d', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:85', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth2', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:85', 'speed': 1000, 'mtu': '4000'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'e401a6a5-8773-4971-9213-b3c2bb52f16c', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:83', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:83', 'speed': 1000, 'mtu': '4000'}, 'eth0': {'netmask': '', 'addr': '', 'hwaddr': '00:e0:4c:80:13:02', 'cfg': {'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': '00:e0:4c:80:13:02', 'STP': 'no', 'DEVICE': 'eth0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'speed': 1000, 'mtu': '1500'}}, 'software_revision': '33', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,tm2,ssse3,cx16,xtpr,pdcm,dca,lahf_lm,dts,tpr_shadow,vnmi,flexpriority,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': ['eth1', 'eth2'], 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'MTU': '4000', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'BONDING_OPTS': 'miimon=100 mode=balance-rr', 'DEVICE': 'bond0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '32105', 'cpuSpeed': '1861.936', 'version_name': 'Snow Man', 'vlans': {'bond0.2': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'ovirtmgmt', 'VLAN': 'yes', 'MTU': '4000', 'BOOTPROTO': 'none', 'DEVICE': 'bond0.2', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 2, 'mtu': '4000'}, 'bond0.3': {'netmask': '255.255.255.255', 'iface': 'bond0', 'addr': '172.16.1.5', 'cfg': {'VLAN': 'yes', 'DEFROUTE': 'no', 'MTU': '4000', 'NM_CONTROLLED': 'no', 'DEVICE': 'bond0.3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 3, 'mtu': '4000'}, 'eth0.11': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'r2o_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.11', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 11, 'mtu': '1500'}, 'eth0.10': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'os_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.10', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 10, 'mtu': '1500'}}, 'cpuCores': '8', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '8', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '172.16.0.11'}}
>Thread-969668::DEBUG::2014-01-28 00:13:46,723::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {}
>Thread-969668::DEBUG::2014-01-28 00:13:46,769::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'OFFLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-129::DEBUG::2014-01-28 00:13:47,214::iscsi::393::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-129::DEBUG::2014-01-28 00:13:47,214::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:13:47,281::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:13:47,425::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-129::DEBUG::2014-01-28 00:13:47,451::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc
>Thread-129::DEBUG::2014-01-28 00:13:47,453::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-129::DEBUG::2014-01-28 00:13:47,464::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=562f0160-7b80-42e2-b248-5754455c40fc:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=7e0cc075-eee7-4268-946e-33a79dd08748', 'REMOTE_PATH=172.16.1.5:/HOSTED-ENGINE', 'ROLE=Master', 'SDUUID=562f0160-7b80-42e2-b248-5754455c40fc', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=14825cb0b944c5c2ae9ed025c82e06730a2d91c4']
>Thread-129::DEBUG::2014-01-28 00:13:47,466::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-129::INFO::2014-01-28 00:13:47,467::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_imageNS already registered
>Thread-129::INFO::2014-01-28 00:13:47,467::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_volumeNS already registered
>Thread-129::DEBUG::2014-01-28 00:13:47,481::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:13:47,524::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n478 bytes (478 B) copied, 0.000830335 s, 576 kB/s\n'; <rc> = 0
>Thread-12::DEBUG::2014-01-28 00:13:47,874::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:13:47,915::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969672::DEBUG::2014-01-28 00:13:48,575::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call vmGetStats with ('2d425c4c-91b1-4564-b276-f27cc581e6a1',) {}
>Thread-969672::DEBUG::2014-01-28 00:13:48,576::BindingXMLRPC::976::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'root', 'memUsage': '53', 'acpiEnable': 'true', 'guestFQDN': u'engine.melb.example.net', 'pid': '18113', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'/', u'total': '25323499520', u'used': '2592124928', u'fs': u'ext4'}, {u'path': u'/boot', u'total': '507744256', u'used': '55643136', u'fs': u'ext4'}], 'network': {u'vnet0': {'macAddr': '00:16:3e:0a:b7:55', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmType': 'kvm', 'guestName': u'engine.melb.example.net', 'elapsedTime': '610813', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'mem_free': '1968788', u'swap_in': '0', u'pageflt': '143', u'mem_total': '3922728', u'mem_unused': '1435008'}, 'cpuSys': '1.07', 'appsList': [u'ovirt-guest-agent-1.0.8-1.el6', u'kernel-2.6.32-431.el6', u'kernel-2.6.32-431.3.1.el6'], 'guestOs': u'2.6.32-431.3.1.el6.x86_64', 'displayType': 'vnc', 'status': 'Up', 'hash': '2950633987041661405', 'vmId': '2d425c4c-91b1-4564-b276-f27cc581e6a1', 'displayIp': '0', 'displayPort': u'5900', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '26843545600', 'writeLatency': '3946838', 'imageID': 'daa7fa08-1354-4b19-a80f-8694ea63f914', 'flushLatency': '78638', 'readRate': '0.00', 'truesize': '3920576512', 'writeRate': '24859.40'}, 'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.95', 'cpuUser': '8.81', 'lastLogin': 1390828070.405062, 'guestIPs': u'172.16.0.10', 'netIfaces': [{u'hw': u'00:16:3e:0a:b7:55', u'name': u'eth0', u'inet': [u'172.16.0.10'], u'inet6': [u'fe80::216:3eff:fe0a:b755']}]}]}
>Thread-969673::INFO::2014-01-28 00:13:49,550::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-969673::INFO::2014-01-28 00:13:49,550::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000830335', 'lastCheck': '2.0', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000328037', 'lastCheck': '7.9', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '8.1', 'valid': False}}
>Thread-12::DEBUG::2014-01-28 00:13:49,938::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:13:49,981::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-229881::INFO::2014-01-28 00:13:50,851::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='5af8ecbb-2530-48c9-8c74-704d51d34f70', volUUID='8d040005-63a7-43f8-9f67-fb8a4ac94193', options=None)
>Thread-229881::ERROR::2014-01-28 00:13:50,860::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/5af8ecbb-2530-48c9-8c74-704d51d34f70/8d040005-63a7-43f8-9f67-fb8a4ac94193'
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>OSError: [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/5af8ecbb-2530-48c9-8c74-704d51d34f70/8d040005-63a7-43f8-9f67-fb8a4ac94193'
>Thread-229881::ERROR::2014-01-28 00:13:50,861::vm::3781::vm.Vm::(updateDriveVolume) vmId=`9c9befe0-9e17-462e-9bce-27ddd684a6d3`::Unable to update the volume 8d040005-63a7-43f8-9f67-fb8a4ac94193 (domain: ff3a0c94-11dc-446d-b093-5f68ad81520d image: 5af8ecbb-2530-48c9-8c74-704d51d34f70) for the drive vda
>Thread-237204::INFO::2014-01-28 00:13:50,934::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='6b90cf80-28ea-48a6-9a11-bc9dcc835138', volUUID='487cfec8-3f1a-4c04-9b54-8aa7b4a5ae68', options=None)
>Thread-237204::ERROR::2014-01-28 00:13:50,937::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/6b90cf80-28ea-48a6-9a11-bc9dcc835138/487cfec8-3f1a-4c04-9b54-8aa7b4a5ae68'
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>OSError: [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/6b90cf80-28ea-48a6-9a11-bc9dcc835138/487cfec8-3f1a-4c04-9b54-8aa7b4a5ae68'
>Thread-237204::ERROR::2014-01-28 00:13:50,937::vm::3781::vm.Vm::(updateDriveVolume) vmId=`e85be056-7715-485c-9cb8-69f9980ed3a3`::Unable to update the volume 487cfec8-3f1a-4c04-9b54-8aa7b4a5ae68 (domain: ff3a0c94-11dc-446d-b093-5f68ad81520d image: 6b90cf80-28ea-48a6-9a11-bc9dcc835138) for the drive vda
>Thread-35::ERROR::2014-01-28 00:13:51,475::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain ff3a0c94-11dc-446d-b093-5f68ad81520d monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
>    self.domain.selftest()
>  File "/usr/share/vdsm/storage/nfsSD.py", line 113, in selftest
>    fileSD.FileStorageDomain.selftest(self)
>  File "/usr/share/vdsm/storage/fileSD.py", line 552, in selftest
>    self.oop.os.statvfs(self.domaindir)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 199, in callCrabRPCFunction
>    raise err
>OSError: [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d'
>Thread-36::DEBUG::2014-01-28 00:13:51,631::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-36::DEBUG::2014-01-28 00:13:51,672::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n364 bytes (364 B) copied, 0.000418801 s, 869 kB/s\n'; <rc> = 0
>Thread-969675::DEBUG::2014-01-28 00:13:51,917::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {}
>Thread-969675::DEBUG::2014-01-28 00:13:51,968::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'OFFLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-12::DEBUG::2014-01-28 00:13:51,997::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:13:52,039::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969677::DEBUG::2014-01-28 00:13:53,361::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-969677::INFO::2014-01-28 00:13:53,362::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-969677::INFO::2014-01-28 00:13:53,363::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-969678::DEBUG::2014-01-28 00:13:53,369::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-969678::INFO::2014-01-28 00:13:53,370::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-969678::INFO::2014-01-28 00:13:53,377::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': 'ff3a0c94-11dc-446d-b093-5f68ad81520d', 'name': 'NextDC_M1', 'version': '3', 'domains': 'ff3a0c94-11dc-446d-b093-5f68ad81520d:Active,436c344c-bc57-441b-b311-c9595c6039e1:Active', 'pool_status': 'connected', 'isoprefix': '/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/images/11111111-1111-1111-1111-111111111111', 'type': 'GLUSTERFS', 'master_ver': 1, 'lver': 1}, 'dominfo': {'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '436c344c-bc57-441b-b311-c9595c6039e1': {'status': 'Active', 'diskfree': '21444952064', 'isoprefix': '/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '25323634688', 'version': 0}}}
>Thread-12::DEBUG::2014-01-28 00:13:54,067::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:13:54,113::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969679::INFO::2014-01-28 00:13:54,531::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-969679::INFO::2014-01-28 00:13:54,532::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000830335', 'lastCheck': '7.0', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000418801', 'lastCheck': '2.9', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.1', 'valid': False}}
>Thread-2585::INFO::2014-01-28 00:13:54,634::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='562f0160-7b80-42e2-b248-5754455c40fc', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='daa7fa08-1354-4b19-a80f-8694ea63f914', volUUID='8d9c27b1-8b26-4a44-8be2-79f024a56e0f', options=None)
>Thread-2585::INFO::2014-01-28 00:13:54,639::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '3920584704', 'apparentsize': '26843545600'}
>Thread-969680::DEBUG::2014-01-28 00:13:55,565::BindingXMLRPC::159::vds::(wrapper) client [127.0.0.1]
>Thread-969680::INFO::2014-01-28 00:13:55,566::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='7e0cc075-eee7-4268-946e-33a79dd08748', conList=[{'port': '', 'connection': '172.16.1.5:/HOSTED-ENGINE', 'iqn': '', 'portal': '', 'user': 'kvm', 'protocol_version': '3', 'password': '******', 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}], options=None)
>Thread-969680::DEBUG::2014-01-28 00:13:55,571::hsm::2273::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE
>Thread-969680::DEBUG::2014-01-28 00:13:55,573::hsm::2297::Storage.HSM::(__prefetchDomains) Found SD uuids: ('562f0160-7b80-42e2-b248-5754455c40fc',)
>Thread-969680::DEBUG::2014-01-28 00:13:55,574::hsm::2353::Storage.HSM::(connectStorageServer) knownSDs: {562f0160-7b80-42e2-b248-5754455c40fc: storage.nfsSD.findDomain, ff3a0c94-11dc-446d-b093-5f68ad81520d: storage.glusterSD.findDomain, 436c344c-bc57-441b-b311-c9595c6039e1: storage.nfsSD.findDomain}
>Thread-969680::INFO::2014-01-28 00:13:55,574::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}]}
>Thread-969681::DEBUG::2014-01-28 00:13:55,595::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call getCapabilities with () {}
>Thread-969681::DEBUG::2014-01-28 00:13:55,598::utils::555::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-969681::DEBUG::2014-01-28 00:13:55,639::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969681::DEBUG::2014-01-28 00:13:55,664::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-969681::DEBUG::2014-01-28 00:13:55,709::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969681::DEBUG::2014-01-28 00:13:55,731::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.2' (cwd None)
>Thread-969681::DEBUG::2014-01-28 00:13:55,779::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969681::DEBUG::2014-01-28 00:13:55,780::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.2' (cwd None)
>Thread-969681::DEBUG::2014-01-28 00:13:55,818::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969681::DEBUG::2014-01-28 00:13:55,841::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.3' (cwd None)
>Thread-969682::DEBUG::2014-01-28 00:13:55,883::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.11]
>Thread-969682::INFO::2014-01-28 00:13:55,899::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-969681::DEBUG::2014-01-28 00:13:55,910::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969682::INFO::2014-01-28 00:13:55,911::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000830335', 'lastCheck': '8.4', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000418801', 'lastCheck': '4.2', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.4', 'valid': False}}
>Thread-969681::DEBUG::2014-01-28 00:13:55,918::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.3' (cwd None)
>Thread-969681::DEBUG::2014-01-28 00:13:55,969::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969681::DEBUG::2014-01-28 00:13:55,982::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.10' (cwd None)
>Thread-969681::DEBUG::2014-01-28 00:13:56,022::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969681::DEBUG::2014-01-28 00:13:56,023::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.10' (cwd None)
>Thread-969681::DEBUG::2014-01-28 00:13:56,066::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969681::DEBUG::2014-01-28 00:13:56,071::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.11' (cwd None)
>Thread-969681::DEBUG::2014-01-28 00:13:56,110::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969681::DEBUG::2014-01-28 00:13:56,111::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.11' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:13:56,154::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-969681::DEBUG::2014-01-28 00:13:56,200::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-12::DEBUG::2014-01-28 00:13:56,236::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969681::DEBUG::2014-01-28 00:13:56,266::caps::453::root::(_getKeyPackages) rpm package glusterfs-rdma not found
>Thread-969681::DEBUG::2014-01-28 00:13:56,270::caps::453::root::(_getKeyPackages) rpm package glusterfs-fuse not found
>Thread-969681::DEBUG::2014-01-28 00:13:56,275::caps::453::root::(_getKeyPackages) rpm package gluster-swift not found
>Thread-969681::DEBUG::2014-01-28 00:13:56,285::caps::453::root::(_getKeyPackages) rpm package gluster-swift-object not found
>Thread-969681::DEBUG::2014-01-28 00:13:56,289::caps::453::root::(_getKeyPackages) rpm package glusterfs not found
>Thread-969681::DEBUG::2014-01-28 00:13:56,293::caps::453::root::(_getKeyPackages) rpm package gluster-swift-plugin not found
>Thread-969681::DEBUG::2014-01-28 00:13:56,304::caps::453::root::(_getKeyPackages) rpm package gluster-swift-account not found
>Thread-969681::DEBUG::2014-01-28 00:13:56,305::caps::453::root::(_getKeyPackages) rpm package gluster-swift-proxy not found
>Thread-969681::DEBUG::2014-01-28 00:13:56,306::caps::453::root::(_getKeyPackages) rpm package gluster-swift-doc not found
>Thread-969681::DEBUG::2014-01-28 00:13:56,312::caps::453::root::(_getKeyPackages) rpm package glusterfs-server not found
>Thread-969681::DEBUG::2014-01-28 00:13:56,315::caps::453::root::(_getKeyPackages) rpm package gluster-swift-container not found
>Thread-969681::DEBUG::2014-01-28 00:13:56,320::caps::453::root::(_getKeyPackages) rpm package glusterfs-geo-replication not found
>Thread-969681::DEBUG::2014-01-28 00:13:56,334::BindingXMLRPC::976::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.3.1.el6.x86_64', 'buildtime': 1388745567.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '33.gitf029545.el6', 'buildtime': 1390094004L, 'version': '4.14.0'}, 'qemu-kvm': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.2', 'buildtime': 1387360004L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'mom': {'release': '20140117.gitfd877c5.el6', 'buildtime': 1389966957L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           L5320  @ 1.86GHz', 'hooks': {'before_vm_start': {'50_hostedengine': {'md5': '45dde62155b5412eafbfff5ef265acc2'}}}, 'cpuSockets': '2', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'gateway': '172.16.0.1', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.2', 'vnet0']}, 'os_net': {'iface': 'os_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.10']}, 'r2o_net': {'iface': 'r2o_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}, 'storage_network': {'iface': u'bond0.3', 'addr': '172.16.1.5', 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000', 'netmask': '255.255.255.255', 'bridged': False, 'interface': u'bond0.3', 'ipv6gateway': '::', 'gateway': ''}}, 'bridges': {'ovirtmgmt': {'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '172.16.0.1', 'ports': ['bond0.2', 'vnet0']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}, 'os_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.10']}, 'r2o_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}}, 'uuid': '44454C4C-3100-104D-8046-B9C04F583153', 'lastClientIface': 'ovirtmgmt', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'cb327caa-ca77-4efe-b7b8-63aba3cda84d', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:85', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth2', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:85', 'speed': 1000, 'mtu': '4000'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'e401a6a5-8773-4971-9213-b3c2bb52f16c', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:83', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:83', 'speed': 1000, 'mtu': '4000'}, 'eth0': {'netmask': '', 'addr': '', 'hwaddr': '00:e0:4c:80:13:02', 'cfg': {'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': '00:e0:4c:80:13:02', 'STP': 'no', 'DEVICE': 'eth0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'speed': 1000, 'mtu': '1500'}}, 'software_revision': '33', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,tm2,ssse3,cx16,xtpr,pdcm,dca,lahf_lm,dts,tpr_shadow,vnmi,flexpriority,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': ['eth1', 'eth2'], 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'MTU': '4000', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'BONDING_OPTS': 'miimon=100 mode=balance-rr', 'DEVICE': 'bond0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '32105', 'cpuSpeed': '1861.936', 'version_name': 'Snow Man', 'vlans': {'bond0.2': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'ovirtmgmt', 'VLAN': 'yes', 'MTU': '4000', 'BOOTPROTO': 'none', 'DEVICE': 'bond0.2', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 2, 'mtu': '4000'}, 'bond0.3': {'netmask': '255.255.255.255', 'iface': 'bond0', 'addr': '172.16.1.5', 'cfg': {'VLAN': 'yes', 'DEFROUTE': 'no', 'MTU': '4000', 'NM_CONTROLLED': 'no', 'DEVICE': 'bond0.3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 3, 'mtu': '4000'}, 'eth0.11': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'r2o_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.11', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 11, 'mtu': '1500'}, 'eth0.10': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'os_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.10', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 10, 'mtu': '1500'}}, 'cpuCores': '8', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '8', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '172.16.0.11'}}
>Thread-969684::DEBUG::2014-01-28 00:13:57,116::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {}
>Thread-969684::DEBUG::2014-01-28 00:13:57,168::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'OFFLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-129::DEBUG::2014-01-28 00:13:57,543::iscsi::393::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-129::DEBUG::2014-01-28 00:13:57,543::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:13:57,603::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:13:57,739::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-129::DEBUG::2014-01-28 00:13:57,762::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc
>Thread-129::DEBUG::2014-01-28 00:13:57,765::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-129::DEBUG::2014-01-28 00:13:57,780::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=562f0160-7b80-42e2-b248-5754455c40fc:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=7e0cc075-eee7-4268-946e-33a79dd08748', 'REMOTE_PATH=172.16.1.5:/HOSTED-ENGINE', 'ROLE=Master', 'SDUUID=562f0160-7b80-42e2-b248-5754455c40fc', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=14825cb0b944c5c2ae9ed025c82e06730a2d91c4']
>Thread-129::DEBUG::2014-01-28 00:13:57,783::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-129::INFO::2014-01-28 00:13:57,783::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_imageNS already registered
>Thread-129::INFO::2014-01-28 00:13:57,783::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_volumeNS already registered
>Thread-129::DEBUG::2014-01-28 00:13:57,798::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:13:57,842::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n478 bytes (478 B) copied, 0.000710283 s, 673 kB/s\n'; <rc> = 0
>Thread-12::DEBUG::2014-01-28 00:13:58,368::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:13:58,418::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969688::DEBUG::2014-01-28 00:13:58,595::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call vmGetStats with ('2d425c4c-91b1-4564-b276-f27cc581e6a1',) {}
>Thread-969688::DEBUG::2014-01-28 00:13:58,597::BindingXMLRPC::976::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'root', 'memUsage': '53', 'acpiEnable': 'true', 'guestFQDN': u'engine.melb.example.net', 'pid': '18113', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'/', u'total': '25323499520', u'used': '2592124928', u'fs': u'ext4'}, {u'path': u'/boot', u'total': '507744256', u'used': '55643136', u'fs': u'ext4'}], 'network': {u'vnet0': {'macAddr': '00:16:3e:0a:b7:55', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmType': 'kvm', 'guestName': u'engine.melb.example.net', 'elapsedTime': '610823', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'mem_free': '1968568', u'swap_in': '0', u'pageflt': '423', u'mem_total': '3922728', u'mem_unused': '1434760'}, 'cpuSys': '0.87', 'appsList': [u'ovirt-guest-agent-1.0.8-1.el6', u'kernel-2.6.32-431.el6', u'kernel-2.6.32-431.3.1.el6'], 'guestOs': u'2.6.32-431.3.1.el6.x86_64', 'displayType': 'vnc', 'status': 'Up', 'hash': '2950633987041661405', 'vmId': '2d425c4c-91b1-4564-b276-f27cc581e6a1', 'displayIp': '0', 'displayPort': u'5900', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '26843545600', 'writeLatency': '719527899', 'imageID': 'daa7fa08-1354-4b19-a80f-8694ea63f914', 'flushLatency': '74081', 'readRate': '0.00', 'truesize': '3920584704', 'writeRate': '27315.21'}, 'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.89', 'cpuUser': '10.97', 'lastLogin': 1390828070.405062, 'guestIPs': u'172.16.0.10', 'netIfaces': [{u'hw': u'00:16:3e:0a:b7:55', u'name': u'eth0', u'inet': [u'172.16.0.10'], u'inet6': [u'fe80::216:3eff:fe0a:b755']}]}]}
>Thread-12::DEBUG::2014-01-28 00:14:00,434::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:14:00,476::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-35::ERROR::2014-01-28 00:14:01,491::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain ff3a0c94-11dc-446d-b093-5f68ad81520d monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
>    self.domain.selftest()
>  File "/usr/share/vdsm/storage/nfsSD.py", line 113, in selftest
>    fileSD.FileStorageDomain.selftest(self)
>  File "/usr/share/vdsm/storage/fileSD.py", line 552, in selftest
>    self.oop.os.statvfs(self.domaindir)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 199, in callCrabRPCFunction
>    raise err
>OSError: [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d'
>Thread-36::DEBUG::2014-01-28 00:14:01,689::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-36::DEBUG::2014-01-28 00:14:01,734::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n364 bytes (364 B) copied, 0.000532541 s, 684 kB/s\n'; <rc> = 0
>Thread-969691::DEBUG::2014-01-28 00:14:02,311::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {}
>Thread-969691::DEBUG::2014-01-28 00:14:02,370::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'OFFLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-12::DEBUG::2014-01-28 00:14:02,492::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:14:02,534::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969692::DEBUG::2014-01-28 00:14:03,428::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-969692::INFO::2014-01-28 00:14:03,429::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-969692::INFO::2014-01-28 00:14:03,430::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-969693::DEBUG::2014-01-28 00:14:03,435::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-969693::INFO::2014-01-28 00:14:03,436::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-969693::INFO::2014-01-28 00:14:03,442::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': 'ff3a0c94-11dc-446d-b093-5f68ad81520d', 'name': 'NextDC_M1', 'version': '3', 'domains': 'ff3a0c94-11dc-446d-b093-5f68ad81520d:Active,436c344c-bc57-441b-b311-c9595c6039e1:Active', 'pool_status': 'connected', 'isoprefix': '/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/images/11111111-1111-1111-1111-111111111111', 'type': 'GLUSTERFS', 'master_ver': 1, 'lver': 1}, 'dominfo': {'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '436c344c-bc57-441b-b311-c9595c6039e1': {'status': 'Active', 'diskfree': '21444952064', 'isoprefix': '/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '25323634688', 'version': 0}}}
>VM Channels Listener::DEBUG::2014-01-28 00:14:04,140::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 104.
>Thread-969694::INFO::2014-01-28 00:14:04,542::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-969694::INFO::2014-01-28 00:14:04,543::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000710283', 'lastCheck': '6.7', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000532541', 'lastCheck': '2.8', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.1', 'valid': False}}
>Thread-12::DEBUG::2014-01-28 00:14:04,555::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:14:04,633::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969695::INFO::2014-01-28 00:14:05,053::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-969695::INFO::2014-01-28 00:14:05,054::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000710283', 'lastCheck': '7.2', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000532541', 'lastCheck': '3.3', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.6', 'valid': False}}
>Thread-969697::DEBUG::2014-01-28 00:14:05,597::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call getCapabilities with () {}
>Thread-969697::DEBUG::2014-01-28 00:14:05,599::utils::555::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-969697::DEBUG::2014-01-28 00:14:05,637::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969697::DEBUG::2014-01-28 00:14:05,671::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-969697::DEBUG::2014-01-28 00:14:05,713::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969697::DEBUG::2014-01-28 00:14:05,737::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.2' (cwd None)
>Thread-969697::DEBUG::2014-01-28 00:14:05,775::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969697::DEBUG::2014-01-28 00:14:05,776::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.2' (cwd None)
>VM Channels Listener::DEBUG::2014-01-28 00:14:05,816::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 138.
>Thread-969697::DEBUG::2014-01-28 00:14:05,822::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969697::DEBUG::2014-01-28 00:14:05,825::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.3' (cwd None)
>Thread-969697::DEBUG::2014-01-28 00:14:05,864::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969697::DEBUG::2014-01-28 00:14:05,865::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.3' (cwd None)
>Thread-969697::DEBUG::2014-01-28 00:14:05,905::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969697::DEBUG::2014-01-28 00:14:05,908::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.10' (cwd None)
>Thread-969697::DEBUG::2014-01-28 00:14:05,948::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969697::DEBUG::2014-01-28 00:14:05,949::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.10' (cwd None)
>Thread-969697::DEBUG::2014-01-28 00:14:05,990::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969697::DEBUG::2014-01-28 00:14:05,995::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.11' (cwd None)
>Thread-969697::DEBUG::2014-01-28 00:14:06,037::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969697::DEBUG::2014-01-28 00:14:06,039::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.11' (cwd None)
>Thread-366769::INFO::2014-01-28 00:14:06,085::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='a746f514-51eb-4926-80d5-545108438f01', volUUID='ba8685ec-668a-4780-88ba-0b114e1a7e7d', options=None)
>Thread-969697::DEBUG::2014-01-28 00:14:06,088::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-366769::ERROR::2014-01-28 00:14:06,116::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/a746f514-51eb-4926-80d5-545108438f01/ba8685ec-668a-4780-88ba-0b114e1a7e7d'
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>OSError: [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/a746f514-51eb-4926-80d5-545108438f01/ba8685ec-668a-4780-88ba-0b114e1a7e7d'
>Thread-969697::DEBUG::2014-01-28 00:14:06,128::caps::453::root::(_getKeyPackages) rpm package glusterfs-rdma not found
>Thread-969697::DEBUG::2014-01-28 00:14:06,128::caps::453::root::(_getKeyPackages) rpm package glusterfs-fuse not found
>Thread-366769::ERROR::2014-01-28 00:14:06,119::vm::3781::vm.Vm::(updateDriveVolume) vmId=`01d705e3-6b62-4796-b9e4-4de1c477401a`::Unable to update the volume ba8685ec-668a-4780-88ba-0b114e1a7e7d (domain: ff3a0c94-11dc-446d-b093-5f68ad81520d image: a746f514-51eb-4926-80d5-545108438f01) for the drive vda
>Thread-969697::DEBUG::2014-01-28 00:14:06,130::caps::453::root::(_getKeyPackages) rpm package gluster-swift not found
>Thread-969697::DEBUG::2014-01-28 00:14:06,140::caps::453::root::(_getKeyPackages) rpm package gluster-swift-object not found
>Thread-969697::DEBUG::2014-01-28 00:14:06,141::caps::453::root::(_getKeyPackages) rpm package glusterfs not found
>Thread-969697::DEBUG::2014-01-28 00:14:06,141::caps::453::root::(_getKeyPackages) rpm package gluster-swift-plugin not found
>Thread-969697::DEBUG::2014-01-28 00:14:06,149::caps::453::root::(_getKeyPackages) rpm package gluster-swift-account not found
>Thread-969697::DEBUG::2014-01-28 00:14:06,150::caps::453::root::(_getKeyPackages) rpm package gluster-swift-proxy not found
>Thread-969697::DEBUG::2014-01-28 00:14:06,150::caps::453::root::(_getKeyPackages) rpm package gluster-swift-doc not found
>Thread-969697::DEBUG::2014-01-28 00:14:06,151::caps::453::root::(_getKeyPackages) rpm package glusterfs-server not found
>Thread-969697::DEBUG::2014-01-28 00:14:06,151::caps::453::root::(_getKeyPackages) rpm package gluster-swift-container not found
>Thread-969697::DEBUG::2014-01-28 00:14:06,151::caps::453::root::(_getKeyPackages) rpm package glusterfs-geo-replication not found
>Thread-969697::DEBUG::2014-01-28 00:14:06,157::BindingXMLRPC::976::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.3.1.el6.x86_64', 'buildtime': 1388745567.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '33.gitf029545.el6', 'buildtime': 1390094004L, 'version': '4.14.0'}, 'qemu-kvm': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.2', 'buildtime': 1387360004L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'mom': {'release': '20140117.gitfd877c5.el6', 'buildtime': 1389966957L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           L5320  @ 1.86GHz', 'hooks': {'before_vm_start': {'50_hostedengine': {'md5': '45dde62155b5412eafbfff5ef265acc2'}}}, 'cpuSockets': '2', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'gateway': '172.16.0.1', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.2', 'vnet0']}, 'os_net': {'iface': 'os_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.10']}, 'r2o_net': {'iface': 'r2o_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}, 'storage_network': {'iface': u'bond0.3', 'addr': '172.16.1.5', 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000', 'netmask': '255.255.255.255', 'bridged': False, 'interface': u'bond0.3', 'ipv6gateway': '::', 'gateway': ''}}, 'bridges': {'ovirtmgmt': {'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '172.16.0.1', 'ports': ['bond0.2', 'vnet0']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}, 'os_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.10']}, 'r2o_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}}, 'uuid': '44454C4C-3100-104D-8046-B9C04F583153', 'lastClientIface': 'ovirtmgmt', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'cb327caa-ca77-4efe-b7b8-63aba3cda84d', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:85', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth2', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:85', 'speed': 1000, 'mtu': '4000'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'e401a6a5-8773-4971-9213-b3c2bb52f16c', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:83', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:83', 'speed': 1000, 'mtu': '4000'}, 'eth0': {'netmask': '', 'addr': '', 'hwaddr': '00:e0:4c:80:13:02', 'cfg': {'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': '00:e0:4c:80:13:02', 'STP': 'no', 'DEVICE': 'eth0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'speed': 1000, 'mtu': '1500'}}, 'software_revision': '33', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,tm2,ssse3,cx16,xtpr,pdcm,dca,lahf_lm,dts,tpr_shadow,vnmi,flexpriority,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': ['eth1', 'eth2'], 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'MTU': '4000', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'BONDING_OPTS': 'miimon=100 mode=balance-rr', 'DEVICE': 'bond0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '32105', 'cpuSpeed': '1861.936', 'version_name': 'Snow Man', 'vlans': {'bond0.2': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'ovirtmgmt', 'VLAN': 'yes', 'MTU': '4000', 'BOOTPROTO': 'none', 'DEVICE': 'bond0.2', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 2, 'mtu': '4000'}, 'bond0.3': {'netmask': '255.255.255.255', 'iface': 'bond0', 'addr': '172.16.1.5', 'cfg': {'VLAN': 'yes', 'DEFROUTE': 'no', 'MTU': '4000', 'NM_CONTROLLED': 'no', 'DEVICE': 'bond0.3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 3, 'mtu': '4000'}, 'eth0.11': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'r2o_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.11', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 11, 'mtu': '1500'}, 'eth0.10': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'os_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.10', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 10, 'mtu': '1500'}}, 'cpuCores': '8', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '8', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '172.16.0.11'}}
>Thread-213624::INFO::2014-01-28 00:14:06,314::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='91386b87-5fe0-4368-9e87-a28896024015', volUUID='b0db0737-8168-4502-a236-1c3162f98f6a', options=None)
>Thread-213624::ERROR::2014-01-28 00:14:06,317::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/91386b87-5fe0-4368-9e87-a28896024015/b0db0737-8168-4502-a236-1c3162f98f6a'
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>OSError: [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/91386b87-5fe0-4368-9e87-a28896024015/b0db0737-8168-4502-a236-1c3162f98f6a'
>Thread-213624::ERROR::2014-01-28 00:14:06,317::vm::3781::vm.Vm::(updateDriveVolume) vmId=`bbbd4efa-022d-44d3-8274-a0506bda2d5e`::Unable to update the volume b0db0737-8168-4502-a236-1c3162f98f6a (domain: ff3a0c94-11dc-446d-b093-5f68ad81520d image: 91386b87-5fe0-4368-9e87-a28896024015) for the drive vda
>Thread-219052::INFO::2014-01-28 00:14:06,321::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='00d2f5e6-0263-4ac7-aa12-98ea66f02d00', volUUID='98d6fa79-aecc-429e-8624-4b6f706bdf4f', options=None)
>Thread-219052::ERROR::2014-01-28 00:14:06,325::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/00d2f5e6-0263-4ac7-aa12-98ea66f02d00/98d6fa79-aecc-429e-8624-4b6f706bdf4f'
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>OSError: [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/00d2f5e6-0263-4ac7-aa12-98ea66f02d00/98d6fa79-aecc-429e-8624-4b6f706bdf4f'
>Thread-219052::ERROR::2014-01-28 00:14:06,326::vm::3781::vm.Vm::(updateDriveVolume) vmId=`546b9733-3453-4210-90fd-2b79cf08d53b`::Unable to update the volume 98d6fa79-aecc-429e-8624-4b6f706bdf4f (domain: ff3a0c94-11dc-446d-b093-5f68ad81520d image: 00d2f5e6-0263-4ac7-aa12-98ea66f02d00) for the drive vda
>Thread-969698::DEBUG::2014-01-28 00:14:06,332::BindingXMLRPC::159::vds::(wrapper) client [127.0.0.1]
>Thread-969698::INFO::2014-01-28 00:14:06,334::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='7e0cc075-eee7-4268-946e-33a79dd08748', conList=[{'port': '', 'connection': '172.16.1.5:/HOSTED-ENGINE', 'iqn': '', 'portal': '', 'user': 'kvm', 'protocol_version': '3', 'password': '******', 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}], options=None)
>Thread-969698::DEBUG::2014-01-28 00:14:06,339::hsm::2273::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE
>Thread-969698::DEBUG::2014-01-28 00:14:06,341::hsm::2297::Storage.HSM::(__prefetchDomains) Found SD uuids: ('562f0160-7b80-42e2-b248-5754455c40fc',)
>Thread-969698::DEBUG::2014-01-28 00:14:06,341::hsm::2353::Storage.HSM::(connectStorageServer) knownSDs: {562f0160-7b80-42e2-b248-5754455c40fc: storage.nfsSD.findDomain, ff3a0c94-11dc-446d-b093-5f68ad81520d: storage.glusterSD.findDomain, 436c344c-bc57-441b-b311-c9595c6039e1: storage.nfsSD.findDomain}
>Thread-969698::INFO::2014-01-28 00:14:06,342::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}]}
>Thread-969699::DEBUG::2014-01-28 00:14:06,609::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.11]
>Thread-969699::INFO::2014-01-28 00:14:06,610::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-969699::INFO::2014-01-28 00:14:06,611::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000710283', 'lastCheck': '8.8', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000532541', 'lastCheck': '4.9', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.1', 'valid': False}}
>Thread-12::DEBUG::2014-01-28 00:14:06,691::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:14:06,735::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>VM Channels Listener::DEBUG::2014-01-28 00:14:07,170::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 70.
>Thread-969700::DEBUG::2014-01-28 00:14:07,520::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {}
>Thread-969700::DEBUG::2014-01-28 00:14:07,557::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'OFFLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-969701::DEBUG::2014-01-28 00:14:07,647::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call tasksList with () {}
>Thread-969701::ERROR::2014-01-28 00:14:07,676::BindingXMLRPC::985::vds::(wrapper) vdsm exception occured
>Traceback (most recent call last):
>  File "/usr/share/vdsm/BindingXMLRPC.py", line 972, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/gluster/api.py", line 54, in wrapper
>    rv = func(*args, **kwargs)
>  File "/usr/share/vdsm/gluster/api.py", line 306, in tasksList
>    status = self.svdsmProxy.glusterTasksList(taskIds)
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in glusterTasksList
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 740, in _callmethod
>    raise convert_to_error(kind, result)
>GlusterCmdExecFailedException: Command execution failed
>error: tasks is not a valid status option
>Usage: volume status [all | <VOLNAME> [nfs|shd|<BRICK>]] [detail|clients|mem|inode|fd|callpool]
>return code: 1
>Thread-250271::INFO::2014-01-28 00:14:07,734::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='51206945-475a-48b1-96e1-689ca96312a5', volUUID='1732793c-bf4f-4f8d-a5b9-4f41e99154ba', options=None)
>Thread-218532::INFO::2014-01-28 00:14:07,742::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='50cabd29-7058-4510-b61f-db21862d99df', volUUID='96d9eaab-9dff-45f6-8fe7-b4fd992ee23e', options=None)
>Thread-250271::ERROR::2014-01-28 00:14:07,768::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/51206945-475a-48b1-96e1-689ca96312a5/1732793c-bf4f-4f8d-a5b9-4f41e99154ba'
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>OSError: [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/51206945-475a-48b1-96e1-689ca96312a5/1732793c-bf4f-4f8d-a5b9-4f41e99154ba'
>Thread-218532::ERROR::2014-01-28 00:14:07,773::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/50cabd29-7058-4510-b61f-db21862d99df/96d9eaab-9dff-45f6-8fe7-b4fd992ee23e'
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>OSError: [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/50cabd29-7058-4510-b61f-db21862d99df/96d9eaab-9dff-45f6-8fe7-b4fd992ee23e'
>Thread-250271::ERROR::2014-01-28 00:14:07,772::vm::3781::vm.Vm::(updateDriveVolume) vmId=`48ad952b-4457-4ae1-bf85-93f869382298`::Unable to update the volume 1732793c-bf4f-4f8d-a5b9-4f41e99154ba (domain: ff3a0c94-11dc-446d-b093-5f68ad81520d image: 51206945-475a-48b1-96e1-689ca96312a5) for the drive vda
>Thread-218532::ERROR::2014-01-28 00:14:07,776::vm::3781::vm.Vm::(updateDriveVolume) vmId=`b3cde33c-f4c5-4b42-8e9d-820b80abc2fb`::Unable to update the volume 96d9eaab-9dff-45f6-8fe7-b4fd992ee23e (domain: ff3a0c94-11dc-446d-b093-5f68ad81520d image: 50cabd29-7058-4510-b61f-db21862d99df) for the drive vda
>Thread-129::DEBUG::2014-01-28 00:14:07,853::iscsi::393::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-129::DEBUG::2014-01-28 00:14:07,854::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:14:07,910::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:14:08,072::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-129::DEBUG::2014-01-28 00:14:08,103::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc
>Thread-129::DEBUG::2014-01-28 00:14:08,105::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-129::DEBUG::2014-01-28 00:14:08,116::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=562f0160-7b80-42e2-b248-5754455c40fc:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=7e0cc075-eee7-4268-946e-33a79dd08748', 'REMOTE_PATH=172.16.1.5:/HOSTED-ENGINE', 'ROLE=Master', 'SDUUID=562f0160-7b80-42e2-b248-5754455c40fc', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=14825cb0b944c5c2ae9ed025c82e06730a2d91c4']
>Thread-129::DEBUG::2014-01-28 00:14:08,118::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-129::INFO::2014-01-28 00:14:08,118::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_imageNS already registered
>Thread-129::INFO::2014-01-28 00:14:08,119::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_volumeNS already registered
>Thread-129::DEBUG::2014-01-28 00:14:08,133::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc/dom_md/metadata bs=4096 count=1' (cwd None)
>VM Channels Listener::DEBUG::2014-01-28 00:14:08,180::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 130.
>Thread-129::DEBUG::2014-01-28 00:14:08,193::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n478 bytes (478 B) copied, 0.001101 s, 434 kB/s\n'; <rc> = 0
>Thread-969706::DEBUG::2014-01-28 00:14:08,593::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call vmGetStats with ('2d425c4c-91b1-4564-b276-f27cc581e6a1',) {}
>Thread-969706::DEBUG::2014-01-28 00:14:08,594::BindingXMLRPC::976::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'root', 'memUsage': '53', 'acpiEnable': 'true', 'guestFQDN': u'engine.melb.example.net', 'pid': '18113', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'/', u'total': '25323499520', u'used': '2592124928', u'fs': u'ext4'}, {u'path': u'/boot', u'total': '507744256', u'used': '55643136', u'fs': u'ext4'}], 'network': {u'vnet0': {'macAddr': '00:16:3e:0a:b7:55', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmType': 'kvm', 'guestName': u'engine.melb.example.net', 'elapsedTime': '610833', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'mem_free': '1968836', u'swap_in': '0', u'pageflt': '148', u'mem_total': '3922728', u'mem_unused': '1435008'}, 'cpuSys': '0.87', 'appsList': [u'ovirt-guest-agent-1.0.8-1.el6', u'kernel-2.6.32-431.el6', u'kernel-2.6.32-431.3.1.el6'], 'guestOs': u'2.6.32-431.3.1.el6.x86_64', 'displayType': 'vnc', 'status': 'Up', 'hash': '2950633987041661405', 'vmId': '2d425c4c-91b1-4564-b276-f27cc581e6a1', 'displayIp': '0', 'displayPort': u'5900', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '26843545600', 'writeLatency': '719527899', 'imageID': 'daa7fa08-1354-4b19-a80f-8694ea63f914', 'flushLatency': '74081', 'readRate': '0.00', 'truesize': '3920584704', 'writeRate': '27315.21'}, 'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.86', 'cpuUser': '10.97', 'lastLogin': 1390828070.405062, 'guestIPs': u'172.16.0.10', 'netIfaces': [{u'hw': u'00:16:3e:0a:b7:55', u'name': u'eth0', u'inet': [u'172.16.0.10'], u'inet6': [u'fe80::216:3eff:fe0a:b755']}]}]}
>Thread-12::DEBUG::2014-01-28 00:14:08,751::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:14:08,796::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969707::DEBUG::2014-01-28 00:14:09,544::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call vmSetTicket with ('9c9befe0-9e17-462e-9bce-27ddd684a6d3', 'cJRhRjxGzEou', '120', 'disconnect', {'userName': 'admin@internal', 'userId': 'fdfc627c-d875-11e0-90f0-83df133b58cc'}) {} flowID [6980b72e]
>Thread-969707::DEBUG::2014-01-28 00:14:09,589::BindingXMLRPC::976::vds::(wrapper) return vmSetTicket with {'status': {'message': 'Done', 'code': 0}}
>VM Channels Listener::DEBUG::2014-01-28 00:14:10,790::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 132.
>Thread-12::DEBUG::2014-01-28 00:14:10,817::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:14:10,859::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-35::ERROR::2014-01-28 00:14:11,508::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain ff3a0c94-11dc-446d-b093-5f68ad81520d monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
>    self.domain.selftest()
>  File "/usr/share/vdsm/storage/nfsSD.py", line 113, in selftest
>    fileSD.FileStorageDomain.selftest(self)
>  File "/usr/share/vdsm/storage/fileSD.py", line 552, in selftest
>    self.oop.os.statvfs(self.domaindir)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 199, in callCrabRPCFunction
>    raise err
>OSError: [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d'
>Thread-36::DEBUG::2014-01-28 00:14:11,749::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-36::DEBUG::2014-01-28 00:14:11,790::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n364 bytes (364 B) copied, 0.00038234 s, 952 kB/s\n'; <rc> = 0
>Thread-969709::DEBUG::2014-01-28 00:14:12,696::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {}
>Thread-969709::DEBUG::2014-01-28 00:14:12,754::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'OFFLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-12::DEBUG::2014-01-28 00:14:12,876::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:14:12,917::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969710::DEBUG::2014-01-28 00:14:13,492::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-969710::INFO::2014-01-28 00:14:13,493::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-969710::INFO::2014-01-28 00:14:13,494::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-969711::DEBUG::2014-01-28 00:14:13,500::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-969711::INFO::2014-01-28 00:14:13,501::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-969711::INFO::2014-01-28 00:14:13,508::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': 'ff3a0c94-11dc-446d-b093-5f68ad81520d', 'name': 'NextDC_M1', 'version': '3', 'domains': 'ff3a0c94-11dc-446d-b093-5f68ad81520d:Active,436c344c-bc57-441b-b311-c9595c6039e1:Active', 'pool_status': 'connected', 'isoprefix': '/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/images/11111111-1111-1111-1111-111111111111', 'type': 'GLUSTERFS', 'master_ver': 1, 'lver': 1}, 'dominfo': {'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '436c344c-bc57-441b-b311-c9595c6039e1': {'status': 'Active', 'diskfree': '21444952064', 'isoprefix': '/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '25323634688', 'version': 0}}}
>Thread-969713::INFO::2014-01-28 00:14:14,555::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-969713::INFO::2014-01-28 00:14:14,556::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.001101', 'lastCheck': '6.4', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00038234', 'lastCheck': '2.8', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.0', 'valid': False}}
>Thread-12::DEBUG::2014-01-28 00:14:14,933::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:14:14,974::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>libvirtEventLoop::DEBUG::2014-01-28 00:14:15,531::clientIF::550::vm.Vm::(dispatchLibvirtEvents) vmId=`9c9befe0-9e17-462e-9bce-27ddd684a6d3`::graphics event phase 0 localAddr {'node': '172.16.0.11', 'service': '', 'family': 0} remoteAddr {'node': '172.16.0.10', 'service': '', 'family': 0}authScheme  subject []
>Thread-969714::DEBUG::2014-01-28 00:14:15,608::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call getCapabilities with () {}
>Thread-969714::DEBUG::2014-01-28 00:14:15,611::utils::555::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-969714::DEBUG::2014-01-28 00:14:15,649::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969714::DEBUG::2014-01-28 00:14:15,678::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-969714::DEBUG::2014-01-28 00:14:15,720::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969714::DEBUG::2014-01-28 00:14:15,741::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.2' (cwd None)
>Thread-969714::DEBUG::2014-01-28 00:14:15,779::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969714::DEBUG::2014-01-28 00:14:15,780::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.2' (cwd None)
>Thread-969714::DEBUG::2014-01-28 00:14:15,820::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969714::DEBUG::2014-01-28 00:14:15,824::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.3' (cwd None)
>Thread-969714::DEBUG::2014-01-28 00:14:15,864::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969714::DEBUG::2014-01-28 00:14:15,865::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.3' (cwd None)
>Thread-969714::DEBUG::2014-01-28 00:14:15,907::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969714::DEBUG::2014-01-28 00:14:15,910::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.10' (cwd None)
>Thread-969714::DEBUG::2014-01-28 00:14:15,949::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969714::DEBUG::2014-01-28 00:14:15,950::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.10' (cwd None)
>Thread-969714::DEBUG::2014-01-28 00:14:15,990::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969714::DEBUG::2014-01-28 00:14:15,995::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.11' (cwd None)
>Thread-969714::DEBUG::2014-01-28 00:14:16,034::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969714::DEBUG::2014-01-28 00:14:16,035::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.11' (cwd None)
>Thread-969714::DEBUG::2014-01-28 00:14:16,074::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-969714::DEBUG::2014-01-28 00:14:16,084::caps::453::root::(_getKeyPackages) rpm package glusterfs-rdma not found
>Thread-969714::DEBUG::2014-01-28 00:14:16,085::caps::453::root::(_getKeyPackages) rpm package glusterfs-fuse not found
>Thread-969714::DEBUG::2014-01-28 00:14:16,087::caps::453::root::(_getKeyPackages) rpm package gluster-swift not found
>Thread-969714::DEBUG::2014-01-28 00:14:16,093::caps::453::root::(_getKeyPackages) rpm package gluster-swift-object not found
>Thread-969714::DEBUG::2014-01-28 00:14:16,094::caps::453::root::(_getKeyPackages) rpm package glusterfs not found
>Thread-969714::DEBUG::2014-01-28 00:14:16,094::caps::453::root::(_getKeyPackages) rpm package gluster-swift-plugin not found
>Thread-969714::DEBUG::2014-01-28 00:14:16,101::caps::453::root::(_getKeyPackages) rpm package gluster-swift-account not found
>Thread-969714::DEBUG::2014-01-28 00:14:16,102::caps::453::root::(_getKeyPackages) rpm package gluster-swift-proxy not found
>Thread-969714::DEBUG::2014-01-28 00:14:16,102::caps::453::root::(_getKeyPackages) rpm package gluster-swift-doc not found
>Thread-969714::DEBUG::2014-01-28 00:14:16,103::caps::453::root::(_getKeyPackages) rpm package glusterfs-server not found
>Thread-969714::DEBUG::2014-01-28 00:14:16,103::caps::453::root::(_getKeyPackages) rpm package gluster-swift-container not found
>Thread-969714::DEBUG::2014-01-28 00:14:16,104::caps::453::root::(_getKeyPackages) rpm package glusterfs-geo-replication not found
>Thread-969714::DEBUG::2014-01-28 00:14:16,109::BindingXMLRPC::976::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.3.1.el6.x86_64', 'buildtime': 1388745567.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '33.gitf029545.el6', 'buildtime': 1390094004L, 'version': '4.14.0'}, 'qemu-kvm': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.2', 'buildtime': 1387360004L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'mom': {'release': '20140117.gitfd877c5.el6', 'buildtime': 1389966957L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           L5320  @ 1.86GHz', 'hooks': {'before_vm_start': {'50_hostedengine': {'md5': '45dde62155b5412eafbfff5ef265acc2'}}}, 'cpuSockets': '2', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'gateway': '172.16.0.1', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.2', 'vnet0']}, 'os_net': {'iface': 'os_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.10']}, 'r2o_net': {'iface': 'r2o_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}, 'storage_network': {'iface': u'bond0.3', 'addr': '172.16.1.5', 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000', 'netmask': '255.255.255.255', 'bridged': False, 'interface': u'bond0.3', 'ipv6gateway': '::', 'gateway': ''}}, 'bridges': {'ovirtmgmt': {'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '172.16.0.1', 'ports': ['bond0.2', 'vnet0']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}, 'os_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.10']}, 'r2o_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}}, 'uuid': '44454C4C-3100-104D-8046-B9C04F583153', 'lastClientIface': 'ovirtmgmt', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'cb327caa-ca77-4efe-b7b8-63aba3cda84d', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:85', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth2', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:85', 'speed': 1000, 'mtu': '4000'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'e401a6a5-8773-4971-9213-b3c2bb52f16c', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:83', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:83', 'speed': 1000, 'mtu': '4000'}, 'eth0': {'netmask': '', 'addr': '', 'hwaddr': '00:e0:4c:80:13:02', 'cfg': {'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': '00:e0:4c:80:13:02', 'STP': 'no', 'DEVICE': 'eth0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'speed': 1000, 'mtu': '1500'}}, 'software_revision': '33', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,tm2,ssse3,cx16,xtpr,pdcm,dca,lahf_lm,dts,tpr_shadow,vnmi,flexpriority,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': ['eth1', 'eth2'], 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'MTU': '4000', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'BONDING_OPTS': 'miimon=100 mode=balance-rr', 'DEVICE': 'bond0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '32105', 'cpuSpeed': '1861.936', 'version_name': 'Snow Man', 'vlans': {'bond0.2': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'ovirtmgmt', 'VLAN': 'yes', 'MTU': '4000', 'BOOTPROTO': 'none', 'DEVICE': 'bond0.2', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 2, 'mtu': '4000'}, 'bond0.3': {'netmask': '255.255.255.255', 'iface': 'bond0', 'addr': '172.16.1.5', 'cfg': {'VLAN': 'yes', 'DEFROUTE': 'no', 'MTU': '4000', 'NM_CONTROLLED': 'no', 'DEVICE': 'bond0.3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 3, 'mtu': '4000'}, 'eth0.11': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'r2o_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.11', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 11, 'mtu': '1500'}, 'eth0.10': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'os_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.10', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 10, 'mtu': '1500'}}, 'cpuCores': '8', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '8', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '172.16.0.11'}}
>VM Channels Listener::DEBUG::2014-01-28 00:14:16,820::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 97.
>Thread-969715::DEBUG::2014-01-28 00:14:16,945::BindingXMLRPC::159::vds::(wrapper) client [127.0.0.1]
>Thread-969715::INFO::2014-01-28 00:14:16,947::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='7e0cc075-eee7-4268-946e-33a79dd08748', conList=[{'port': '', 'connection': '172.16.1.5:/HOSTED-ENGINE', 'iqn': '', 'portal': '', 'user': 'kvm', 'protocol_version': '3', 'password': '******', 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}], options=None)
>Thread-969715::DEBUG::2014-01-28 00:14:16,951::hsm::2273::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE
>Thread-969715::DEBUG::2014-01-28 00:14:16,954::hsm::2297::Storage.HSM::(__prefetchDomains) Found SD uuids: ('562f0160-7b80-42e2-b248-5754455c40fc',)
>Thread-969715::DEBUG::2014-01-28 00:14:16,954::hsm::2353::Storage.HSM::(connectStorageServer) knownSDs: {562f0160-7b80-42e2-b248-5754455c40fc: storage.nfsSD.findDomain, ff3a0c94-11dc-446d-b093-5f68ad81520d: storage.glusterSD.findDomain, 436c344c-bc57-441b-b311-c9595c6039e1: storage.nfsSD.findDomain}
>Thread-969715::INFO::2014-01-28 00:14:16,955::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}]}
>Thread-12::DEBUG::2014-01-28 00:14:16,990::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:14:17,043::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>libvirtEventLoop::DEBUG::2014-01-28 00:14:17,087::clientIF::550::vm.Vm::(dispatchLibvirtEvents) vmId=`9c9befe0-9e17-462e-9bce-27ddd684a6d3`::graphics event phase 1 localAddr {'node': '172.16.0.11', 'service': '', 'family': 0} remoteAddr {'node': '172.16.0.10', 'service': '', 'family': 0}authScheme spice subject []
>Thread-969716::DEBUG::2014-01-28 00:14:17,215::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.11]
>Thread-969716::INFO::2014-01-28 00:14:17,218::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-969716::INFO::2014-01-28 00:14:17,219::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.001101', 'lastCheck': '9.0', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00038234', 'lastCheck': '5.4', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.7', 'valid': False}}
>Thread-969718::DEBUG::2014-01-28 00:14:17,868::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {}
>Thread-969718::DEBUG::2014-01-28 00:14:17,910::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'OFFLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-129::DEBUG::2014-01-28 00:14:18,202::iscsi::393::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-129::DEBUG::2014-01-28 00:14:18,203::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:14:18,270::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:14:18,412::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-129::DEBUG::2014-01-28 00:14:18,438::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc
>Thread-129::DEBUG::2014-01-28 00:14:18,439::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-129::DEBUG::2014-01-28 00:14:18,451::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=562f0160-7b80-42e2-b248-5754455c40fc:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=7e0cc075-eee7-4268-946e-33a79dd08748', 'REMOTE_PATH=172.16.1.5:/HOSTED-ENGINE', 'ROLE=Master', 'SDUUID=562f0160-7b80-42e2-b248-5754455c40fc', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=14825cb0b944c5c2ae9ed025c82e06730a2d91c4']
>Thread-129::DEBUG::2014-01-28 00:14:18,454::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-129::INFO::2014-01-28 00:14:18,454::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_imageNS already registered
>Thread-129::INFO::2014-01-28 00:14:18,455::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_volumeNS already registered
>Thread-129::DEBUG::2014-01-28 00:14:18,469::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:14:18,512::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n478 bytes (478 B) copied, 0.000704314 s, 679 kB/s\n'; <rc> = 0
>Thread-969722::DEBUG::2014-01-28 00:14:18,610::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call vmGetStats with ('2d425c4c-91b1-4564-b276-f27cc581e6a1',) {}
>Thread-969722::DEBUG::2014-01-28 00:14:18,611::BindingXMLRPC::976::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'root', 'memUsage': '53', 'acpiEnable': 'true', 'guestFQDN': u'engine.melb.example.net', 'pid': '18113', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'/', u'total': '25323499520', u'used': '2592124928', u'fs': u'ext4'}, {u'path': u'/boot', u'total': '507744256', u'used': '55643136', u'fs': u'ext4'}], 'network': {u'vnet0': {'macAddr': '00:16:3e:0a:b7:55', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmType': 'kvm', 'guestName': u'engine.melb.example.net', 'elapsedTime': '610843', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'mem_free': '1966828', u'swap_in': '0', u'pageflt': '156', u'mem_total': '3922728', u'mem_unused': '1432960'}, 'cpuSys': '1.06', 'appsList': [u'ovirt-guest-agent-1.0.8-1.el6', u'kernel-2.6.32-431.el6', u'kernel-2.6.32-431.3.1.el6'], 'guestOs': u'2.6.32-431.3.1.el6.x86_64', 'displayType': 'vnc', 'status': 'Up', 'hash': '2950633987041661405', 'vmId': '2d425c4c-91b1-4564-b276-f27cc581e6a1', 'displayIp': '0', 'displayPort': u'5900', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '26843545600', 'writeLatency': '719527899', 'imageID': 'daa7fa08-1354-4b19-a80f-8694ea63f914', 'flushLatency': '74081', 'readRate': '0.00', 'truesize': '3920584704', 'writeRate': '27315.21'}, 'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.85', 'cpuUser': '13.66', 'lastLogin': 1390828070.405062, 'guestIPs': u'172.16.0.10', 'netIfaces': [{u'hw': u'00:16:3e:0a:b7:55', u'name': u'eth0', u'inet': [u'172.16.0.10'], u'inet6': [u'fe80::216:3eff:fe0a:b755']}]}]}
>libvirtEventLoop::DEBUG::2014-01-28 00:14:19,015::clientIF::550::vm.Vm::(dispatchLibvirtEvents) vmId=`9c9befe0-9e17-462e-9bce-27ddd684a6d3`::graphics event phase 0 localAddr {'node': '172.16.0.11', 'service': '', 'family': 0} remoteAddr {'node': '172.16.0.10', 'service': '', 'family': 0}authScheme  subject []
>Thread-12::DEBUG::2014-01-28 00:14:19,060::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:14:19,107::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>MainThread::DEBUG::2014-01-28 00:14:19,329::vdsm::55::vds::(sigtermHandler) Received signal 15
>MainThread::INFO::2014-01-28 00:14:19,573::vmChannels::183::vds::(stop) VM channels listener was stopped.
>MainThread::INFO::2014-01-28 00:14:19,574::momIF::84::MOM::(stop) Shutting down MOM
>MainThread::INFO::2014-01-28 00:14:19,577::logUtils::44::dispatcher::(wrapper) Run and protect: prepareForShutdown(options=None)
>libvirtEventLoop::DEBUG::2014-01-28 00:14:19,817::clientIF::550::vm.Vm::(dispatchLibvirtEvents) vmId=`9c9befe0-9e17-462e-9bce-27ddd684a6d3`::graphics event phase 1 localAddr {'node': '172.16.0.11', 'service': '', 'family': 0} remoteAddr {'node': '172.16.0.10', 'service': '', 'family': 0}authScheme spice subject []
>VM Channels Listener::INFO::2014-01-28 00:14:19,846::vmChannels::178::vds::(run) VM channels listener thread has ended.
>MainThread::DEBUG::2014-01-28 00:14:20,187::vdsm::55::vds::(sigtermHandler) Received signal 15
>MainThread::DEBUG::2014-01-28 00:14:20,188::clientIF::207::vds::(prepareForShutdown) cannot run prepareForShutdown concurrently
>libvirtEventLoop::DEBUG::2014-01-28 00:14:20,592::clientIF::550::vm.Vm::(dispatchLibvirtEvents) vmId=`9c9befe0-9e17-462e-9bce-27ddd684a6d3`::graphics event phase 0 localAddr {'node': '172.16.0.11', 'service': '', 'family': 0} remoteAddr {'node': '172.16.0.10', 'service': '', 'family': 0}authScheme  subject []
>libvirtEventLoop::DEBUG::2014-01-28 00:14:20,598::clientIF::550::vm.Vm::(dispatchLibvirtEvents) vmId=`9c9befe0-9e17-462e-9bce-27ddd684a6d3`::graphics event phase 0 localAddr {'node': '172.16.0.11', 'service': '', 'family': 0} remoteAddr {'node': '172.16.0.10', 'service': '', 'family': 0}authScheme  subject []
>MainThread::INFO::2014-01-28 00:14:20,984::domainMonitor::125::Storage.DomainMonitor::(close) Stopping domain monitors
>MainThread::INFO::2014-01-28 00:14:20,984::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring 562f0160-7b80-42e2-b248-5754455c40fc
>Thread-129::DEBUG::2014-01-28 00:14:20,985::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for 562f0160-7b80-42e2-b248-5754455c40fc
>Thread-129::DEBUG::2014-01-28 00:14:20,988::domainMonitor::184::Storage.DomainMonitorThread::(_monitorLoop) Unable to release the host id 1 for domain 562f0160-7b80-42e2-b248-5754455c40fc
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 181, in _monitorLoop
>    self.domain.releaseHostId(self.hostId, unused=True)
>  File "/usr/share/vdsm/storage/sd.py", line 461, in releaseHostId
>    self._clusterLock.releaseHostId(hostId, async, unused)
>  File "/usr/share/vdsm/storage/clusterlock.py", line 204, in releaseHostId
>    raise se.ReleaseHostIdFailure(self._sdUUID, e)
>ReleaseHostIdFailure: Cannot release host id: ('562f0160-7b80-42e2-b248-5754455c40fc', SanlockException(16, 'Sanlock lockspace remove failure', 'Device or resource busy'))
>MainThread::INFO::2014-01-28 00:14:21,001::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring ff3a0c94-11dc-446d-b093-5f68ad81520d
>Thread-35::DEBUG::2014-01-28 00:14:21,001::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for ff3a0c94-11dc-446d-b093-5f68ad81520d
>Thread-35::DEBUG::2014-01-28 00:14:21,002::domainMonitor::184::Storage.DomainMonitorThread::(_monitorLoop) Unable to release the host id 1 for domain ff3a0c94-11dc-446d-b093-5f68ad81520d
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 181, in _monitorLoop
>    self.domain.releaseHostId(self.hostId, unused=True)
>  File "/usr/share/vdsm/storage/sd.py", line 461, in releaseHostId
>    self._clusterLock.releaseHostId(hostId, async, unused)
>  File "/usr/share/vdsm/storage/clusterlock.py", line 204, in releaseHostId
>    raise se.ReleaseHostIdFailure(self._sdUUID, e)
>ReleaseHostIdFailure: Cannot release host id: ('ff3a0c94-11dc-446d-b093-5f68ad81520d', SanlockException(16, 'Sanlock lockspace remove failure', 'Device or resource busy'))
>MainThread::INFO::2014-01-28 00:14:21,003::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring 436c344c-bc57-441b-b311-c9595c6039e1
>Thread-36::DEBUG::2014-01-28 00:14:21,003::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for 436c344c-bc57-441b-b311-c9595c6039e1
>MainThread::INFO::2014-01-28 00:14:21,004::logUtils::47::dispatcher::(wrapper) Run and protect: prepareForShutdown, Return response: None
>MainThread::INFO::2014-01-28 00:14:21,005::vdsm::125::vds::(run) VDSM main thread ended. Waiting for 25 other threads...
>MainThread::INFO::2014-01-28 00:14:21,006::vdsm::129::vds::(run) <_MainThread(MainThread, started 140507188819712)>
>MainThread::INFO::2014-01-28 00:14:21,006::vdsm::129::vds::(run) <Thread(libvirtEventLoop, started daemon 140506921940736)>
>MainThread::INFO::2014-01-28 00:14:21,006::vdsm::129::vds::(run) <WorkerThread(a5c591db-5c54-474e-a7b0-4225ca91c442, started daemon 140506835187456)>
>MainThread::INFO::2014-01-28 00:14:21,006::vdsm::129::vds::(run) <WorkerThread(e24324ea-2e5e-4c4d-813b-7b4b4b877f6f, started daemon 140506401064704)>
>MainThread::DEBUG::2014-01-28 00:14:21,006::sampling::292::vm.Vm::(stop) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Stop statistics collection
>Thread-2585::DEBUG::2014-01-28 00:14:21,007::sampling::323::vm.Vm::(run) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Stats thread finished
>MainThread::INFO::2014-01-28 00:14:21,007::vdsm::129::vds::(run) <VmStatsThread(Thread-2585, started daemon 140505451050752)>
>MainThread::INFO::2014-01-28 00:14:21,008::vdsm::129::vds::(run) <GuestMonitor(GuestMonitor-dev1-melb-example-net, started daemon 140505069377280)>
>MainThread::INFO::2014-01-28 00:14:21,008::vdsm::129::vds::(run) <WorkerThread(c4fed4d2-5ba1-497e-815f-921833741811, started daemon 140506782738176)>
>MainThread::INFO::2014-01-28 00:14:21,008::vdsm::129::vds::(run) <WorkerThread(ed49fd1d-8d94-4b12-89ce-3da6997d3e60, started daemon 140506793228032)>
>MainThread::INFO::2014-01-28 00:14:21,008::vdsm::129::vds::(run) <WorkerThread(f9e3f98d-b363-4328-8c91-442d741c2c72, started daemon 140506824697600)>
>MainThread::INFO::2014-01-28 00:14:21,008::vdsm::129::vds::(run) <WorkerThread(9cd9ab3a-4706-4a7a-bc2a-52c3c3568fbc, started daemon 140506803717888)>
>MainThread::INFO::2014-01-28 00:14:21,009::vdsm::129::vds::(run) <WorkerThread(2e70b102-b7be-417b-8436-72e56bee1526, started daemon 140506411554560)>
>MainThread::DEBUG::2014-01-28 00:14:21,009::sampling::292::vm.Vm::(stop) vmId=`01d705e3-6b62-4796-b9e4-4de1c477401a`::Stop statistics collection
>MainThread::INFO::2014-01-28 00:14:21,009::vdsm::129::vds::(run) <VmStatsThread(Thread-366769, started daemon 140504129853184)>
>Thread-366769::DEBUG::2014-01-28 00:14:21,009::sampling::323::vm.Vm::(run) vmId=`01d705e3-6b62-4796-b9e4-4de1c477401a`::Stats thread finished
>MainThread::INFO::2014-01-28 00:14:21,010::vdsm::129::vds::(run) <WorkerThread(0cc9e3c5-c5bf-4feb-9ced-2453a1091115, started daemon 140506422044416)>
>MainThread::INFO::2014-01-28 00:14:21,011::vdsm::129::vds::(run) <HostMonitor(HostMonitor, started daemon 140505493010176)>
>MainThread::DEBUG::2014-01-28 00:14:21,011::sampling::292::vm.Vm::(stop) vmId=`bbbd4efa-022d-44d3-8274-a0506bda2d5e`::Stop statistics collection
>MainThread::INFO::2014-01-28 00:14:21,011::vdsm::129::vds::(run) <VmStatsThread(Thread-213624, started daemon 140505941792512)>
>Thread-213624::DEBUG::2014-01-28 00:14:21,011::sampling::323::vm.Vm::(run) vmId=`bbbd4efa-022d-44d3-8274-a0506bda2d5e`::Stats thread finished
>MainThread::INFO::2014-01-28 00:14:21,012::vdsm::129::vds::(run) <GuestManager(GuestManager, started daemon 140505482520320)>
>MainThread::INFO::2014-01-28 00:14:21,012::vdsm::129::vds::(run) <PolicyEngine(PolicyEngine, started daemon 140506380084992)>
>MainThread::INFO::2014-01-28 00:14:21,013::vdsm::129::vds::(run) <WorkerThread(939e7424-ab30-47e0-92a8-5050f69f64c6, started daemon 140506432534272)>
>MainThread::DEBUG::2014-01-28 00:14:21,013::sampling::292::vm.Vm::(stop) vmId=`48ad952b-4457-4ae1-bf85-93f869382298`::Stop statistics collection
>MainThread::INFO::2014-01-28 00:14:21,013::vdsm::129::vds::(run) <VmStatsThread(Thread-250271, started daemon 140504108873472)>
>Thread-250271::DEBUG::2014-01-28 00:14:21,013::sampling::323::vm.Vm::(run) vmId=`48ad952b-4457-4ae1-bf85-93f869382298`::Stats thread finished
>MainThread::DEBUG::2014-01-28 00:14:21,014::sampling::292::vm.Vm::(stop) vmId=`e85be056-7715-485c-9cb8-69f9980ed3a3`::Stop statistics collection
>MainThread::INFO::2014-01-28 00:14:21,015::vdsm::129::vds::(run) <VmStatsThread(Thread-237204, started daemon 140504610105088)>
>Thread-237204::DEBUG::2014-01-28 00:14:21,015::sampling::323::vm.Vm::(run) vmId=`e85be056-7715-485c-9cb8-69f9980ed3a3`::Stats thread finished
>MainThread::INFO::2014-01-28 00:14:21,015::vdsm::129::vds::(run) <GuestMonitor(GuestMonitor-HostedEngine, started daemon 140505058887424)>
>MainThread::INFO::2014-01-28 00:14:21,015::momIF::84::MOM::(stop) Shutting down MOM
>MainThread::INFO::2014-01-28 00:14:21,016::vdsm::129::vds::(run) <MomThread(MOM, started 140505952282368)>
>MainThread::DEBUG::2014-01-28 00:14:21,016::sampling::292::vm.Vm::(stop) vmId=`b3cde33c-f4c5-4b42-8e9d-820b80abc2fb`::Stop statistics collection
>MainThread::INFO::2014-01-28 00:14:21,016::vdsm::129::vds::(run) <VmStatsThread(Thread-218532, started daemon 140505037907712)>
>Thread-218532::DEBUG::2014-01-28 00:14:21,016::sampling::323::vm.Vm::(run) vmId=`b3cde33c-f4c5-4b42-8e9d-820b80abc2fb`::Stats thread finished
>MainThread::DEBUG::2014-01-28 00:14:21,017::sampling::292::vm.Vm::(stop) vmId=`9c9befe0-9e17-462e-9bce-27ddd684a6d3`::Stop statistics collection
>Thread-229881::DEBUG::2014-01-28 00:14:21,018::sampling::323::vm.Vm::(run) vmId=`9c9befe0-9e17-462e-9bce-27ddd684a6d3`::Stats thread finished
>MainThread::INFO::2014-01-28 00:14:21,018::vdsm::129::vds::(run) <VmStatsThread(Thread-229881, started daemon 140504599615232)>
>MainThread::DEBUG::2014-01-28 00:14:21,019::sampling::292::vm.Vm::(stop) vmId=`546b9733-3453-4210-90fd-2b79cf08d53b`::Stop statistics collection
>MainThread::INFO::2014-01-28 00:14:21,019::vdsm::129::vds::(run) <VmStatsThread(Thread-219052, started daemon 140505048397568)>
>Thread-219052::DEBUG::2014-01-28 00:14:21,019::sampling::323::vm.Vm::(run) vmId=`546b9733-3453-4210-90fd-2b79cf08d53b`::Stats thread finished
>MainThread::INFO::2014-01-28 00:14:21,019::vdsm::129::vds::(run) <WorkerThread(d29e821b-2187-4d58-b0cf-815c14d303c1, started daemon 140506814207744)>
>libvirtEventLoop::DEBUG::2014-01-28 00:14:21,274::clientIF::550::vm.Vm::(dispatchLibvirtEvents) vmId=`9c9befe0-9e17-462e-9bce-27ddd684a6d3`::graphics event phase 1 localAddr {'node': '172.16.0.11', 'service': '', 'family': 0} remoteAddr {'node': '172.16.0.10', 'service': '', 'family': 0}authScheme spice subject []
>libvirtEventLoop::DEBUG::2014-01-28 00:14:21,365::clientIF::550::vm.Vm::(dispatchLibvirtEvents) vmId=`9c9befe0-9e17-462e-9bce-27ddd684a6d3`::graphics event phase 1 localAddr {'node': '172.16.0.11', 'service': '', 'family': 0} remoteAddr {'node': '172.16.0.10', 'service': '', 'family': 0}authScheme spice subject []
>MainThread::DEBUG::2014-01-28 00:14:27,622::vdsm::55::vds::(sigtermHandler) Received signal 15
>MainThread::DEBUG::2014-01-28 00:14:27,623::clientIF::211::vds::(prepareForShutdown) cannot run prepareForShutdown twice
>MainThread::INFO::2014-01-28 00:14:29,234::vdsm::119::vds::(run) (PID: 7788) I am the actual vdsm 4.14.0-33.gitf029545.el6 hv01.melb.example.net (2.6.32-431.3.1.el6.x86_64)
>MainThread::WARNING::2014-01-28 00:14:29,243::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt already exists
>MainThread::DEBUG::2014-01-28 00:14:30,152::multipath::142::Storage.Multipath::(isEnabled) Current revision of multipath.conf detected, preserving
>MainThread::DEBUG::2014-01-28 00:14:30,153::hsm::420::Storage.Misc.excCmd::(__validateLvmLockingType) '/usr/bin/sudo -n /sbin/lvm dumpconfig global/locking_type' (cwd None)
>MainThread::DEBUG::2014-01-28 00:14:30,412::hsm::420::Storage.Misc.excCmd::(__validateLvmLockingType) SUCCESS: <err> = ''; <rc> = 0
>MainThread::DEBUG::2014-01-28 00:14:30,413::hsm::444::Storage.HSM::(__cleanStorageRepository) Started cleaning storage repository at '/rhev/data-center'
>MainThread::DEBUG::2014-01-28 00:14:30,419::hsm::476::Storage.HSM::(__cleanStorageRepository) White list: ['/rhev/data-center/hsm-tasks', '/rhev/data-center/hsm-tasks/*', '/rhev/data-center/mnt']
>MainThread::DEBUG::2014-01-28 00:14:30,419::hsm::477::Storage.HSM::(__cleanStorageRepository) Mount list: ['/rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE', '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA', '/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso']
>MainThread::DEBUG::2014-01-28 00:14:30,420::hsm::479::Storage.HSM::(__cleanStorageRepository) Cleaning leftovers
>MainThread::DEBUG::2014-01-28 00:14:30,424::hsm::522::Storage.HSM::(__cleanStorageRepository) Finished cleaning storage repository at '/rhev/data-center'
>MainThread::INFO::2014-01-28 00:14:30,426::dispatcher::99::Storage.Dispatcher::(__init__) Starting StorageDispatcher...
>storageRefresh::DEBUG::2014-01-28 00:14:30,427::iscsi::393::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>storageRefresh::DEBUG::2014-01-28 00:14:30,496::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>MainThread::INFO::2014-01-28 00:14:30,631::logUtils::44::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback(callbackFunc=<bound method clientIF.contEIOVms of <clientIF.clientIF instance at 0x29afa70>>)
>MainThread::INFO::2014-01-28 00:14:30,640::logUtils::47::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback, Return response: None
>MainThread::DEBUG::2014-01-28 00:14:30,642::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>MainThread::DEBUG::2014-01-28 00:14:30,658::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>MainThread::INFO::2014-01-28 00:14:30,665::momIF::47::MOM::(__init__) Starting up MOM
>MainThread::INFO::2014-01-28 00:14:30,677::vmChannels::187::vds::(settimeout) Setting channels' timeout to 30 seconds.
>VM Channels Listener::INFO::2014-01-28 00:14:30,753::vmChannels::170::vds::(run) Starting VM channels listener thread.
>clientIFinit::DEBUG::2014-01-28 00:14:30,818::libvirtconnection::145::root::(get) trying to connect libvirt
>MainThread::WARNING::2014-01-28 00:14:30,823::clientIF::181::vds::(_prepareBindings) Unable to load the json rpc server module. Please make sure it is installed.
>Thread-13::DEBUG::2014-01-28 00:14:31,041::BindingXMLRPC::159::vds::(wrapper) client [127.0.0.1]
>Thread-13::INFO::2014-01-28 00:14:31,062::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='7e0cc075-eee7-4268-946e-33a79dd08748', conList=[{'port': '', 'connection': '172.16.1.5:/HOSTED-ENGINE', 'iqn': '', 'portal': '', 'user': 'kvm', 'protocol_version': '3', 'password': '******', 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}], options=None)
>Thread-13::DEBUG::2014-01-28 00:14:31,138::hsm::2273::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE
>Thread-13::DEBUG::2014-01-28 00:14:31,441::hsm::2297::Storage.HSM::(__prefetchDomains) Found SD uuids: ('562f0160-7b80-42e2-b248-5754455c40fc',)
>Thread-13::DEBUG::2014-01-28 00:14:31,446::hsm::2353::Storage.HSM::(connectStorageServer) knownSDs: {562f0160-7b80-42e2-b248-5754455c40fc: storage.nfsSD.findDomain}
>Thread-13::INFO::2014-01-28 00:14:31,449::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}]}
>Thread-14::DEBUG::2014-01-28 00:14:31,540::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call getCapabilities with () {}
>Thread-14::DEBUG::2014-01-28 00:14:31,546::BindingXMLRPC::976::vds::(wrapper) return getCapabilities with {'status': {'message': 'Recovering from crash or Initializing', 'code': 99}}
>Thread-15::DEBUG::2014-01-28 00:14:31,890::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.11]
>Thread-15::INFO::2014-01-28 00:14:31,912::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15::INFO::2014-01-28 00:14:31,921::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
>Thread-16::DEBUG::2014-01-28 00:14:32,029::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.11]
>Thread-16::INFO::2014-01-28 00:14:32,040::logUtils::44::dispatcher::(wrapper) Run and protect: startMonitoringDomain(sdUUID='562f0160-7b80-42e2-b248-5754455c40fc', hostID=1, options=None)
>Thread-16::INFO::2014-01-28 00:14:32,069::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 562f0160-7b80-42e2-b248-5754455c40fc
>Thread-17::DEBUG::2014-01-28 00:14:32,079::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 562f0160-7b80-42e2-b248-5754455c40fc
>Thread-16::INFO::2014-01-28 00:14:32,100::logUtils::47::dispatcher::(wrapper) Run and protect: startMonitoringDomain, Return response: None
>Thread-18::DEBUG::2014-01-28 00:14:32,211::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.11]
>Thread-18::INFO::2014-01-28 00:14:32,231::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-18::INFO::2014-01-28 00:14:32,245::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.2', 'valid': True}}
>clientIFinit::DEBUG::2014-01-28 00:14:32,474::clientIF::493::vds::(_recoverVm) Trying to recover 9c9befe0-9e17-462e-9bce-27ddd684a6d3
>clientIFinit::INFO::2014-01-28 00:14:32,475::clientIF::361::vds::(createVm) vmContainerLock acquired by vm 9c9befe0-9e17-462e-9bce-27ddd684a6d3
>clientIFinit::DEBUG::2014-01-28 00:14:32,475::libvirtconnection::145::root::(get) trying to connect libvirt
>clientIFinit::DEBUG::2014-01-28 00:14:32,529::clientIF::374::vds::(createVm) Total desktops after creation of 9c9befe0-9e17-462e-9bce-27ddd684a6d3 is 1
>Thread-19::DEBUG::2014-01-28 00:14:32,528::vm::2226::vm.Vm::(_startUnderlyingVm) vmId=`9c9befe0-9e17-462e-9bce-27ddd684a6d3`::Start
>Thread-19::DEBUG::2014-01-28 00:14:32,532::vm::2230::vm.Vm::(_startUnderlyingVm) vmId=`9c9befe0-9e17-462e-9bce-27ddd684a6d3`::_ongoingCreations acquired
>Thread-19::INFO::2014-01-28 00:14:32,535::vm::3091::vm.Vm::(_run) vmId=`9c9befe0-9e17-462e-9bce-27ddd684a6d3`::VM wrapper has started
>Thread-19::WARNING::2014-01-28 00:14:32,539::vm::2053::vm.Vm::(buildConfDevices) vmId=`9c9befe0-9e17-462e-9bce-27ddd684a6d3`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel0', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'1'}}' found
>Thread-19::WARNING::2014-01-28 00:14:32,547::vm::2053::vm.Vm::(buildConfDevices) vmId=`9c9befe0-9e17-462e-9bce-27ddd684a6d3`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel1', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'2'}}' found
>storageRefresh::DEBUG::2014-01-28 00:14:32,549::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-19::WARNING::2014-01-28 00:14:32,571::vm::2053::vm.Vm::(buildConfDevices) vmId=`9c9befe0-9e17-462e-9bce-27ddd684a6d3`::Unknown type found, device: '{'device': u'spicevmc', 'alias': u'channel2', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'3'}}' found
>Thread-19::INFO::2014-01-28 00:14:32,590::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='5af8ecbb-2530-48c9-8c74-704d51d34f70', volUUID='8d040005-63a7-43f8-9f67-fb8a4ac94193', options=None)
>clientIFinit::DEBUG::2014-01-28 00:14:32,598::clientIF::493::vds::(_recoverVm) Trying to recover e85be056-7715-485c-9cb8-69f9980ed3a3
>clientIFinit::INFO::2014-01-28 00:14:32,599::clientIF::361::vds::(createVm) vmContainerLock acquired by vm e85be056-7715-485c-9cb8-69f9980ed3a3
>Thread-21::DEBUG::2014-01-28 00:14:32,612::vm::2226::vm.Vm::(_startUnderlyingVm) vmId=`e85be056-7715-485c-9cb8-69f9980ed3a3`::Start
>clientIFinit::DEBUG::2014-01-28 00:14:32,612::clientIF::374::vds::(createVm) Total desktops after creation of e85be056-7715-485c-9cb8-69f9980ed3a3 is 2
>Thread-21::DEBUG::2014-01-28 00:14:32,616::vm::2230::vm.Vm::(_startUnderlyingVm) vmId=`e85be056-7715-485c-9cb8-69f9980ed3a3`::_ongoingCreations acquired
>Thread-21::INFO::2014-01-28 00:14:32,617::vm::3091::vm.Vm::(_run) vmId=`e85be056-7715-485c-9cb8-69f9980ed3a3`::VM wrapper has started
>Thread-21::WARNING::2014-01-28 00:14:32,617::vm::2053::vm.Vm::(buildConfDevices) vmId=`e85be056-7715-485c-9cb8-69f9980ed3a3`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel0', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'1'}}' found
>Thread-21::WARNING::2014-01-28 00:14:32,617::vm::2053::vm.Vm::(buildConfDevices) vmId=`e85be056-7715-485c-9cb8-69f9980ed3a3`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel1', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'2'}}' found
>Thread-21::WARNING::2014-01-28 00:14:32,618::vm::2053::vm.Vm::(buildConfDevices) vmId=`e85be056-7715-485c-9cb8-69f9980ed3a3`::Unknown type found, device: '{'device': u'spicevmc', 'alias': u'channel2', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'3'}}' found
>Thread-21::INFO::2014-01-28 00:14:32,619::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='6b90cf80-28ea-48a6-9a11-bc9dcc835138', volUUID='487cfec8-3f1a-4c04-9b54-8aa7b4a5ae68', options=None)
>clientIFinit::DEBUG::2014-01-28 00:14:32,631::clientIF::493::vds::(_recoverVm) Trying to recover 01d705e3-6b62-4796-b9e4-4de1c477401a
>clientIFinit::INFO::2014-01-28 00:14:32,631::clientIF::361::vds::(createVm) vmContainerLock acquired by vm 01d705e3-6b62-4796-b9e4-4de1c477401a
>Thread-23::DEBUG::2014-01-28 00:14:32,643::vm::2226::vm.Vm::(_startUnderlyingVm) vmId=`01d705e3-6b62-4796-b9e4-4de1c477401a`::Start
>clientIFinit::DEBUG::2014-01-28 00:14:32,642::clientIF::374::vds::(createVm) Total desktops after creation of 01d705e3-6b62-4796-b9e4-4de1c477401a is 3
>Thread-23::DEBUG::2014-01-28 00:14:32,646::vm::2230::vm.Vm::(_startUnderlyingVm) vmId=`01d705e3-6b62-4796-b9e4-4de1c477401a`::_ongoingCreations acquired
>Thread-23::INFO::2014-01-28 00:14:32,647::vm::3091::vm.Vm::(_run) vmId=`01d705e3-6b62-4796-b9e4-4de1c477401a`::VM wrapper has started
>Thread-23::WARNING::2014-01-28 00:14:32,648::vm::2053::vm.Vm::(buildConfDevices) vmId=`01d705e3-6b62-4796-b9e4-4de1c477401a`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel0', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'1'}}' found
>Thread-23::WARNING::2014-01-28 00:14:32,652::vm::2053::vm.Vm::(buildConfDevices) vmId=`01d705e3-6b62-4796-b9e4-4de1c477401a`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel1', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'2'}}' found
>Thread-23::WARNING::2014-01-28 00:14:32,655::vm::2053::vm.Vm::(buildConfDevices) vmId=`01d705e3-6b62-4796-b9e4-4de1c477401a`::Unknown type found, device: '{'device': u'spicevmc', 'alias': u'channel2', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'3'}}' found
>Thread-23::INFO::2014-01-28 00:14:32,664::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='a746f514-51eb-4926-80d5-545108438f01', volUUID='ba8685ec-668a-4780-88ba-0b114e1a7e7d', options=None)
>Thread-12::DEBUG::2014-01-28 00:14:32,667::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>storageRefresh::DEBUG::2014-01-28 00:14:32,708::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-17::DEBUG::2014-01-28 00:14:32,765::iscsi::393::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-12::DEBUG::2014-01-28 00:14:32,778::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-17::DEBUG::2014-01-28 00:14:32,783::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>clientIFinit::DEBUG::2014-01-28 00:14:32,821::clientIF::493::vds::(_recoverVm) Trying to recover b3cde33c-f4c5-4b42-8e9d-820b80abc2fb
>clientIFinit::INFO::2014-01-28 00:14:32,829::clientIF::361::vds::(createVm) vmContainerLock acquired by vm b3cde33c-f4c5-4b42-8e9d-820b80abc2fb
>Thread-17::DEBUG::2014-01-28 00:14:32,843::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>storageRefresh::DEBUG::2014-01-28 00:14:32,896::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm pvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,size,vg_name,vg_uuid,pe_start,pe_count,pe_alloc_count,mda_count,dev_size' (cwd None)
>Thread-17::DEBUG::2014-01-28 00:14:32,945::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>clientIFinit::DEBUG::2014-01-28 00:14:33,000::clientIF::374::vds::(createVm) Total desktops after creation of b3cde33c-f4c5-4b42-8e9d-820b80abc2fb is 4
>Thread-25::DEBUG::2014-01-28 00:14:33,000::vm::2226::vm.Vm::(_startUnderlyingVm) vmId=`b3cde33c-f4c5-4b42-8e9d-820b80abc2fb`::Start
>Thread-25::DEBUG::2014-01-28 00:14:33,003::vm::2230::vm.Vm::(_startUnderlyingVm) vmId=`b3cde33c-f4c5-4b42-8e9d-820b80abc2fb`::_ongoingCreations acquired
>Thread-25::INFO::2014-01-28 00:14:33,003::vm::3091::vm.Vm::(_run) vmId=`b3cde33c-f4c5-4b42-8e9d-820b80abc2fb`::VM wrapper has started
>Thread-25::WARNING::2014-01-28 00:14:33,004::vm::2053::vm.Vm::(buildConfDevices) vmId=`b3cde33c-f4c5-4b42-8e9d-820b80abc2fb`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel0', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'1'}}' found
>Thread-25::WARNING::2014-01-28 00:14:33,004::vm::2053::vm.Vm::(buildConfDevices) vmId=`b3cde33c-f4c5-4b42-8e9d-820b80abc2fb`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel1', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'2'}}' found
>Thread-25::WARNING::2014-01-28 00:14:33,004::vm::2053::vm.Vm::(buildConfDevices) vmId=`b3cde33c-f4c5-4b42-8e9d-820b80abc2fb`::Unknown type found, device: '{'device': u'spicevmc', 'alias': u'channel2', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'3'}}' found
>Thread-25::INFO::2014-01-28 00:14:33,006::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='50cabd29-7058-4510-b61f-db21862d99df', volUUID='96d9eaab-9dff-45f6-8fe7-b4fd992ee23e', options=None)
>clientIFinit::DEBUG::2014-01-28 00:14:33,019::clientIF::493::vds::(_recoverVm) Trying to recover 546b9733-3453-4210-90fd-2b79cf08d53b
>clientIFinit::INFO::2014-01-28 00:14:33,019::clientIF::361::vds::(createVm) vmContainerLock acquired by vm 546b9733-3453-4210-90fd-2b79cf08d53b
>clientIFinit::DEBUG::2014-01-28 00:14:33,030::clientIF::374::vds::(createVm) Total desktops after creation of 546b9733-3453-4210-90fd-2b79cf08d53b is 5
>Thread-27::DEBUG::2014-01-28 00:14:33,030::vm::2226::vm.Vm::(_startUnderlyingVm) vmId=`546b9733-3453-4210-90fd-2b79cf08d53b`::Start
>clientIFinit::DEBUG::2014-01-28 00:14:33,044::clientIF::493::vds::(_recoverVm) Trying to recover 2d425c4c-91b1-4564-b276-f27cc581e6a1
>clientIFinit::INFO::2014-01-28 00:14:33,045::clientIF::361::vds::(createVm) vmContainerLock acquired by vm 2d425c4c-91b1-4564-b276-f27cc581e6a1
>clientIFinit::DEBUG::2014-01-28 00:14:33,055::clientIF::374::vds::(createVm) Total desktops after creation of 2d425c4c-91b1-4564-b276-f27cc581e6a1 is 6
>Thread-29::DEBUG::2014-01-28 00:14:33,055::vm::2226::vm.Vm::(_startUnderlyingVm) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Start
>clientIFinit::DEBUG::2014-01-28 00:14:33,063::clientIF::493::vds::(_recoverVm) Trying to recover bbbd4efa-022d-44d3-8274-a0506bda2d5e
>clientIFinit::INFO::2014-01-28 00:14:33,063::clientIF::361::vds::(createVm) vmContainerLock acquired by vm bbbd4efa-022d-44d3-8274-a0506bda2d5e
>clientIFinit::DEBUG::2014-01-28 00:14:33,074::clientIF::374::vds::(createVm) Total desktops after creation of bbbd4efa-022d-44d3-8274-a0506bda2d5e is 7
>Thread-31::DEBUG::2014-01-28 00:14:33,074::vm::2226::vm.Vm::(_startUnderlyingVm) vmId=`bbbd4efa-022d-44d3-8274-a0506bda2d5e`::Start
>clientIFinit::DEBUG::2014-01-28 00:14:33,089::clientIF::493::vds::(_recoverVm) Trying to recover 48ad952b-4457-4ae1-bf85-93f869382298
>clientIFinit::INFO::2014-01-28 00:14:33,090::clientIF::361::vds::(createVm) vmContainerLock acquired by vm 48ad952b-4457-4ae1-bf85-93f869382298
>clientIFinit::DEBUG::2014-01-28 00:14:33,102::clientIF::374::vds::(createVm) Total desktops after creation of 48ad952b-4457-4ae1-bf85-93f869382298 is 8
>Thread-33::DEBUG::2014-01-28 00:14:33,102::vm::2226::vm.Vm::(_startUnderlyingVm) vmId=`48ad952b-4457-4ae1-bf85-93f869382298`::Start
>storageRefresh::DEBUG::2014-01-28 00:14:33,192::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0
>storageRefresh::DEBUG::2014-01-28 00:14:33,200::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None)
>storageRefresh::DEBUG::2014-01-28 00:14:33,397::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  No volume groups found\n'; <rc> = 0
>storageRefresh::DEBUG::2014-01-28 00:14:33,410::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags' (cwd None)
>Thread-19::ERROR::2014-01-28 00:14:33,440::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain ff3a0c94-11dc-446d-b093-5f68ad81520d
>Thread-19::ERROR::2014-01-28 00:14:33,448::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain ff3a0c94-11dc-446d-b093-5f68ad81520d
>Thread-19::DEBUG::2014-01-28 00:14:33,533::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name ff3a0c94-11dc-446d-b093-5f68ad81520d' (cwd None)
>storageRefresh::DEBUG::2014-01-28 00:14:33,637::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  No volume groups found\n'; <rc> = 0
>storageRefresh::DEBUG::2014-01-28 00:14:33,639::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None)
>Thread-17::DEBUG::2014-01-28 00:14:33,759::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc
>Thread-19::DEBUG::2014-01-28 00:14:33,788::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "ff3a0c94-11dc-446d-b093-5f68ad81520d" not found\n  Skipping volume group ff3a0c94-11dc-446d-b093-5f68ad81520d\n'; <rc> = 5
>Thread-19::WARNING::2014-01-28 00:14:33,789::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "ff3a0c94-11dc-446d-b093-5f68ad81520d" not found', '  Skipping volume group ff3a0c94-11dc-446d-b093-5f68ad81520d']
>storageRefresh::DEBUG::2014-01-28 00:14:33,847::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  No volume groups found\n'; <rc> = 0
>storageRefresh::WARNING::2014-01-28 00:14:33,848::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/hsm-tasks already exists
>storageRefresh::DEBUG::2014-01-28 00:14:33,849::hsm::388::Storage.HSM::(storageRefresh) HSM is ready
>Thread-17::DEBUG::2014-01-28 00:14:33,998::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-17::DEBUG::2014-01-28 00:14:34,009::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=562f0160-7b80-42e2-b248-5754455c40fc:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=7e0cc075-eee7-4268-946e-33a79dd08748', 'REMOTE_PATH=172.16.1.5:/HOSTED-ENGINE', 'ROLE=Master', 'SDUUID=562f0160-7b80-42e2-b248-5754455c40fc', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=14825cb0b944c5c2ae9ed025c82e06730a2d91c4']
>Thread-17::DEBUG::2014-01-28 00:14:34,012::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-17::DEBUG::2014-01-28 00:14:34,027::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-19::ERROR::2014-01-28 00:14:34,092::sdc::143::Storage.StorageDomainCache::(_findDomain) domain ff3a0c94-11dc-446d-b093-5f68ad81520d not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('ff3a0c94-11dc-446d-b093-5f68ad81520d',)
>Thread-17::DEBUG::2014-01-28 00:14:34,096::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n478 bytes (478 B) copied, 0.00061236 s, 781 kB/s\n'; <rc> = 0
>Thread-21::ERROR::2014-01-28 00:14:34,110::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain ff3a0c94-11dc-446d-b093-5f68ad81520d
>Thread-21::ERROR::2014-01-28 00:14:34,116::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain ff3a0c94-11dc-446d-b093-5f68ad81520d
>Thread-19::ERROR::2014-01-28 00:14:34,162::dispatcher::67::Storage.Dispatcher.Protect::(run) {'status': {'message': "Storage domain does not exist: ('ff3a0c94-11dc-446d-b093-5f68ad81520d',)", 'code': 358}}
>Thread-21::DEBUG::2014-01-28 00:14:34,164::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name ff3a0c94-11dc-446d-b093-5f68ad81520d' (cwd None)
>Thread-17::DEBUG::2014-01-28 00:14:34,196::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 562f0160-7b80-42e2-b248-5754455c40fc changed its status to Valid
>Thread-19::ERROR::2014-01-28 00:14:34,169::vm::1991::vm.Vm::(_normalizeVdsmImg) vmId=`9c9befe0-9e17-462e-9bce-27ddd684a6d3`::Unable to get volume size for 8d040005-63a7-43f8-9f67-fb8a4ac94193
>Traceback (most recent call last):
>  File "/usr/share/vdsm/vm.py", line 1987, in _normalizeVdsmImg
>    drv['truesize'] = res['truesize']
>KeyError: 'truesize'
>Thread-19::INFO::2014-01-28 00:14:34,208::vm::2244::vm.Vm::(_startUnderlyingVm) vmId=`9c9befe0-9e17-462e-9bce-27ddd684a6d3`::Skipping errors on recovery
>Traceback (most recent call last):
>  File "/usr/share/vdsm/vm.py", line 2233, in _startUnderlyingVm
>    self._run()
>  File "/usr/share/vdsm/vm.py", line 3093, in _run
>    devices = self.buildConfDevices()
>  File "/usr/share/vdsm/vm.py", line 2061, in buildConfDevices
>    self._normalizeVdsmImg(drv)
>  File "/usr/share/vdsm/vm.py", line 1993, in _normalizeVdsmImg
>    drv['volumeID'])
>RuntimeError: Volume 8d040005-63a7-43f8-9f67-fb8a4ac94193 is corrupted or missing
>Thread-19::DEBUG::2014-01-28 00:14:34,209::vm::2247::vm.Vm::(_startUnderlyingVm) vmId=`9c9befe0-9e17-462e-9bce-27ddd684a6d3`::_ongoingCreations released
>Thread-27::DEBUG::2014-01-28 00:14:34,209::vm::2230::vm.Vm::(_startUnderlyingVm) vmId=`546b9733-3453-4210-90fd-2b79cf08d53b`::_ongoingCreations acquired
>Thread-27::INFO::2014-01-28 00:14:34,213::vm::3091::vm.Vm::(_run) vmId=`546b9733-3453-4210-90fd-2b79cf08d53b`::VM wrapper has started
>Thread-27::WARNING::2014-01-28 00:14:34,216::vm::2053::vm.Vm::(buildConfDevices) vmId=`546b9733-3453-4210-90fd-2b79cf08d53b`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel0', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'1'}}' found
>Thread-27::WARNING::2014-01-28 00:14:34,219::vm::2053::vm.Vm::(buildConfDevices) vmId=`546b9733-3453-4210-90fd-2b79cf08d53b`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel1', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'2'}}' found
>Thread-27::WARNING::2014-01-28 00:14:34,223::vm::2053::vm.Vm::(buildConfDevices) vmId=`546b9733-3453-4210-90fd-2b79cf08d53b`::Unknown type found, device: '{'device': u'spicevmc', 'alias': u'channel2', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'3'}}' found
>Thread-27::INFO::2014-01-28 00:14:34,230::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='00d2f5e6-0263-4ac7-aa12-98ea66f02d00', volUUID='98d6fa79-aecc-429e-8624-4b6f706bdf4f', options=None)
>Thread-21::DEBUG::2014-01-28 00:14:34,382::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "ff3a0c94-11dc-446d-b093-5f68ad81520d" not found\n  Skipping volume group ff3a0c94-11dc-446d-b093-5f68ad81520d\n'; <rc> = 5
>Thread-21::WARNING::2014-01-28 00:14:34,383::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "ff3a0c94-11dc-446d-b093-5f68ad81520d" not found', '  Skipping volume group ff3a0c94-11dc-446d-b093-5f68ad81520d']
>Thread-21::ERROR::2014-01-28 00:14:34,400::sdc::143::Storage.StorageDomainCache::(_findDomain) domain ff3a0c94-11dc-446d-b093-5f68ad81520d not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('ff3a0c94-11dc-446d-b093-5f68ad81520d',)
>Thread-25::ERROR::2014-01-28 00:14:34,403::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain ff3a0c94-11dc-446d-b093-5f68ad81520d
>Thread-25::ERROR::2014-01-28 00:14:34,405::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain ff3a0c94-11dc-446d-b093-5f68ad81520d
>Thread-21::ERROR::2014-01-28 00:14:34,419::dispatcher::67::Storage.Dispatcher.Protect::(run) {'status': {'message': "Storage domain does not exist: ('ff3a0c94-11dc-446d-b093-5f68ad81520d',)", 'code': 358}}
>Thread-25::DEBUG::2014-01-28 00:14:34,422::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name ff3a0c94-11dc-446d-b093-5f68ad81520d' (cwd None)
>Thread-21::ERROR::2014-01-28 00:14:34,421::vm::1991::vm.Vm::(_normalizeVdsmImg) vmId=`e85be056-7715-485c-9cb8-69f9980ed3a3`::Unable to get volume size for 487cfec8-3f1a-4c04-9b54-8aa7b4a5ae68
>Traceback (most recent call last):
>  File "/usr/share/vdsm/vm.py", line 1987, in _normalizeVdsmImg
>    drv['truesize'] = res['truesize']
>KeyError: 'truesize'
>Thread-21::INFO::2014-01-28 00:14:34,445::vm::2244::vm.Vm::(_startUnderlyingVm) vmId=`e85be056-7715-485c-9cb8-69f9980ed3a3`::Skipping errors on recovery
>Traceback (most recent call last):
>  File "/usr/share/vdsm/vm.py", line 2233, in _startUnderlyingVm
>    self._run()
>  File "/usr/share/vdsm/vm.py", line 3093, in _run
>    devices = self.buildConfDevices()
>  File "/usr/share/vdsm/vm.py", line 2061, in buildConfDevices
>    self._normalizeVdsmImg(drv)
>  File "/usr/share/vdsm/vm.py", line 1993, in _normalizeVdsmImg
>    drv['volumeID'])
>RuntimeError: Volume 487cfec8-3f1a-4c04-9b54-8aa7b4a5ae68 is corrupted or missing
>Thread-21::DEBUG::2014-01-28 00:14:34,447::vm::2247::vm.Vm::(_startUnderlyingVm) vmId=`e85be056-7715-485c-9cb8-69f9980ed3a3`::_ongoingCreations released
>Thread-29::DEBUG::2014-01-28 00:14:34,448::vm::2230::vm.Vm::(_startUnderlyingVm) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::_ongoingCreations acquired
>Thread-29::INFO::2014-01-28 00:14:34,451::vm::3091::vm.Vm::(_run) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::VM wrapper has started
>Thread-29::WARNING::2014-01-28 00:14:34,455::vm::2053::vm.Vm::(buildConfDevices) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel0', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'1'}}' found
>Thread-29::WARNING::2014-01-28 00:14:34,458::vm::2053::vm.Vm::(buildConfDevices) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel1', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'2'}}' found
>Thread-29::INFO::2014-01-28 00:14:34,466::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='562f0160-7b80-42e2-b248-5754455c40fc', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='daa7fa08-1354-4b19-a80f-8694ea63f914', volUUID='8d9c27b1-8b26-4a44-8be2-79f024a56e0f', options=None)
>Thread-29::INFO::2014-01-28 00:14:34,485::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '3920592896', 'apparentsize': '26843545600'}
>Thread-25::DEBUG::2014-01-28 00:14:34,703::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "ff3a0c94-11dc-446d-b093-5f68ad81520d" not found\n  Skipping volume group ff3a0c94-11dc-446d-b093-5f68ad81520d\n'; <rc> = 5
>Thread-25::WARNING::2014-01-28 00:14:34,728::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "ff3a0c94-11dc-446d-b093-5f68ad81520d" not found', '  Skipping volume group ff3a0c94-11dc-446d-b093-5f68ad81520d']
>Thread-47::DEBUG::2014-01-28 00:14:34,762::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call getCapabilities with () {} flowID [221e4d0a]
>Thread-47::DEBUG::2014-01-28 00:14:34,769::BindingXMLRPC::976::vds::(wrapper) return getCapabilities with {'status': {'message': 'Recovering from crash or Initializing', 'code': 99}}
>Thread-25::ERROR::2014-01-28 00:14:34,843::sdc::143::Storage.StorageDomainCache::(_findDomain) domain ff3a0c94-11dc-446d-b093-5f68ad81520d not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('ff3a0c94-11dc-446d-b093-5f68ad81520d',)
>Thread-27::ERROR::2014-01-28 00:14:34,854::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain ff3a0c94-11dc-446d-b093-5f68ad81520d
>Thread-27::ERROR::2014-01-28 00:14:34,861::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain ff3a0c94-11dc-446d-b093-5f68ad81520d
>Thread-25::ERROR::2014-01-28 00:14:34,917::dispatcher::67::Storage.Dispatcher.Protect::(run) {'status': {'message': "Storage domain does not exist: ('ff3a0c94-11dc-446d-b093-5f68ad81520d',)", 'code': 358}}
>Thread-27::DEBUG::2014-01-28 00:14:34,924::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name ff3a0c94-11dc-446d-b093-5f68ad81520d' (cwd None)
>Thread-25::ERROR::2014-01-28 00:14:34,926::vm::1991::vm.Vm::(_normalizeVdsmImg) vmId=`b3cde33c-f4c5-4b42-8e9d-820b80abc2fb`::Unable to get volume size for 96d9eaab-9dff-45f6-8fe7-b4fd992ee23e
>Traceback (most recent call last):
>  File "/usr/share/vdsm/vm.py", line 1987, in _normalizeVdsmImg
>    drv['truesize'] = res['truesize']
>KeyError: 'truesize'
>Thread-25::INFO::2014-01-28 00:14:34,964::vm::2244::vm.Vm::(_startUnderlyingVm) vmId=`b3cde33c-f4c5-4b42-8e9d-820b80abc2fb`::Skipping errors on recovery
>Traceback (most recent call last):
>  File "/usr/share/vdsm/vm.py", line 2233, in _startUnderlyingVm
>    self._run()
>  File "/usr/share/vdsm/vm.py", line 3093, in _run
>    devices = self.buildConfDevices()
>  File "/usr/share/vdsm/vm.py", line 2061, in buildConfDevices
>    self._normalizeVdsmImg(drv)
>  File "/usr/share/vdsm/vm.py", line 1993, in _normalizeVdsmImg
>    drv['volumeID'])
>RuntimeError: Volume 96d9eaab-9dff-45f6-8fe7-b4fd992ee23e is corrupted or missing
>Thread-25::DEBUG::2014-01-28 00:14:34,980::vm::2247::vm.Vm::(_startUnderlyingVm) vmId=`b3cde33c-f4c5-4b42-8e9d-820b80abc2fb`::_ongoingCreations released
>Thread-31::DEBUG::2014-01-28 00:14:34,980::vm::2230::vm.Vm::(_startUnderlyingVm) vmId=`bbbd4efa-022d-44d3-8274-a0506bda2d5e`::_ongoingCreations acquired
>Thread-31::INFO::2014-01-28 00:14:34,994::vm::3091::vm.Vm::(_run) vmId=`bbbd4efa-022d-44d3-8274-a0506bda2d5e`::VM wrapper has started
>Thread-12::DEBUG::2014-01-28 00:14:35,001::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-31::WARNING::2014-01-28 00:14:35,008::vm::2053::vm.Vm::(buildConfDevices) vmId=`bbbd4efa-022d-44d3-8274-a0506bda2d5e`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel0', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'1'}}' found
>Thread-31::WARNING::2014-01-28 00:14:35,045::vm::2053::vm.Vm::(buildConfDevices) vmId=`bbbd4efa-022d-44d3-8274-a0506bda2d5e`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel1', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'2'}}' found
>Thread-31::WARNING::2014-01-28 00:14:35,061::vm::2053::vm.Vm::(buildConfDevices) vmId=`bbbd4efa-022d-44d3-8274-a0506bda2d5e`::Unknown type found, device: '{'device': u'spicevmc', 'alias': u'channel2', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'3'}}' found
>Thread-31::INFO::2014-01-28 00:14:35,090::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='91386b87-5fe0-4368-9e87-a28896024015', volUUID='b0db0737-8168-4502-a236-1c3162f98f6a', options=None)
>Thread-12::DEBUG::2014-01-28 00:14:35,136::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-27::DEBUG::2014-01-28 00:14:35,192::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "ff3a0c94-11dc-446d-b093-5f68ad81520d" not found\n  Skipping volume group ff3a0c94-11dc-446d-b093-5f68ad81520d\n'; <rc> = 5
>Thread-27::WARNING::2014-01-28 00:14:35,222::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "ff3a0c94-11dc-446d-b093-5f68ad81520d" not found', '  Skipping volume group ff3a0c94-11dc-446d-b093-5f68ad81520d']
>Thread-27::ERROR::2014-01-28 00:14:35,361::sdc::143::Storage.StorageDomainCache::(_findDomain) domain ff3a0c94-11dc-446d-b093-5f68ad81520d not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('ff3a0c94-11dc-446d-b093-5f68ad81520d',)
>Thread-23::ERROR::2014-01-28 00:14:35,377::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain ff3a0c94-11dc-446d-b093-5f68ad81520d
>Thread-23::ERROR::2014-01-28 00:14:35,386::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain ff3a0c94-11dc-446d-b093-5f68ad81520d
>Thread-27::ERROR::2014-01-28 00:14:35,444::dispatcher::67::Storage.Dispatcher.Protect::(run) {'status': {'message': "Storage domain does not exist: ('ff3a0c94-11dc-446d-b093-5f68ad81520d',)", 'code': 358}}
>Thread-23::DEBUG::2014-01-28 00:14:35,449::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name ff3a0c94-11dc-446d-b093-5f68ad81520d' (cwd None)
>Thread-27::ERROR::2014-01-28 00:14:35,453::vm::1991::vm.Vm::(_normalizeVdsmImg) vmId=`546b9733-3453-4210-90fd-2b79cf08d53b`::Unable to get volume size for 98d6fa79-aecc-429e-8624-4b6f706bdf4f
>Traceback (most recent call last):
>  File "/usr/share/vdsm/vm.py", line 1987, in _normalizeVdsmImg
>    drv['truesize'] = res['truesize']
>KeyError: 'truesize'
>Thread-27::INFO::2014-01-28 00:14:35,488::vm::2244::vm.Vm::(_startUnderlyingVm) vmId=`546b9733-3453-4210-90fd-2b79cf08d53b`::Skipping errors on recovery
>Traceback (most recent call last):
>  File "/usr/share/vdsm/vm.py", line 2233, in _startUnderlyingVm
>    self._run()
>  File "/usr/share/vdsm/vm.py", line 3093, in _run
>    devices = self.buildConfDevices()
>  File "/usr/share/vdsm/vm.py", line 2061, in buildConfDevices
>    self._normalizeVdsmImg(drv)
>  File "/usr/share/vdsm/vm.py", line 1993, in _normalizeVdsmImg
>    drv['volumeID'])
>RuntimeError: Volume 98d6fa79-aecc-429e-8624-4b6f706bdf4f is corrupted or missing
>Thread-33::DEBUG::2014-01-28 00:14:35,501::vm::2230::vm.Vm::(_startUnderlyingVm) vmId=`48ad952b-4457-4ae1-bf85-93f869382298`::_ongoingCreations acquired
>Thread-27::DEBUG::2014-01-28 00:14:35,501::vm::2247::vm.Vm::(_startUnderlyingVm) vmId=`546b9733-3453-4210-90fd-2b79cf08d53b`::_ongoingCreations released
>Thread-33::INFO::2014-01-28 00:14:35,510::vm::3091::vm.Vm::(_run) vmId=`48ad952b-4457-4ae1-bf85-93f869382298`::VM wrapper has started
>Thread-33::WARNING::2014-01-28 00:14:35,520::vm::2053::vm.Vm::(buildConfDevices) vmId=`48ad952b-4457-4ae1-bf85-93f869382298`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel0', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'1'}}' found
>Thread-33::WARNING::2014-01-28 00:14:35,530::vm::2053::vm.Vm::(buildConfDevices) vmId=`48ad952b-4457-4ae1-bf85-93f869382298`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel1', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'2'}}' found
>Thread-33::WARNING::2014-01-28 00:14:35,541::vm::2053::vm.Vm::(buildConfDevices) vmId=`48ad952b-4457-4ae1-bf85-93f869382298`::Unknown type found, device: '{'device': u'spicevmc', 'alias': u'channel2', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'3'}}' found
>Thread-33::INFO::2014-01-28 00:14:35,567::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='51206945-475a-48b1-96e1-689ca96312a5', volUUID='1732793c-bf4f-4f8d-a5b9-4f41e99154ba', options=None)
>Thread-56::DEBUG::2014-01-28 00:14:35,632::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call getCapabilities with () {}
>Thread-56::DEBUG::2014-01-28 00:14:35,641::BindingXMLRPC::976::vds::(wrapper) return getCapabilities with {'status': {'message': 'Recovering from crash or Initializing', 'code': 99}}
>Thread-23::DEBUG::2014-01-28 00:14:35,725::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "ff3a0c94-11dc-446d-b093-5f68ad81520d" not found\n  Skipping volume group ff3a0c94-11dc-446d-b093-5f68ad81520d\n'; <rc> = 5
>Thread-23::WARNING::2014-01-28 00:14:35,750::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "ff3a0c94-11dc-446d-b093-5f68ad81520d" not found', '  Skipping volume group ff3a0c94-11dc-446d-b093-5f68ad81520d']
>Thread-23::ERROR::2014-01-28 00:14:35,878::sdc::143::Storage.StorageDomainCache::(_findDomain) domain ff3a0c94-11dc-446d-b093-5f68ad81520d not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('ff3a0c94-11dc-446d-b093-5f68ad81520d',)
>Thread-31::ERROR::2014-01-28 00:14:35,895::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain ff3a0c94-11dc-446d-b093-5f68ad81520d
>Thread-31::ERROR::2014-01-28 00:14:35,906::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain ff3a0c94-11dc-446d-b093-5f68ad81520d
>Thread-23::ERROR::2014-01-28 00:14:36,005::dispatcher::67::Storage.Dispatcher.Protect::(run) {'status': {'message': "Storage domain does not exist: ('ff3a0c94-11dc-446d-b093-5f68ad81520d',)", 'code': 358}}
>Thread-31::DEBUG::2014-01-28 00:14:36,022::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name ff3a0c94-11dc-446d-b093-5f68ad81520d' (cwd None)
>Thread-23::ERROR::2014-01-28 00:14:36,018::vm::1991::vm.Vm::(_normalizeVdsmImg) vmId=`01d705e3-6b62-4796-b9e4-4de1c477401a`::Unable to get volume size for ba8685ec-668a-4780-88ba-0b114e1a7e7d
>Traceback (most recent call last):
>  File "/usr/share/vdsm/vm.py", line 1987, in _normalizeVdsmImg
>    drv['truesize'] = res['truesize']
>KeyError: 'truesize'
>Thread-23::INFO::2014-01-28 00:14:36,071::vm::2244::vm.Vm::(_startUnderlyingVm) vmId=`01d705e3-6b62-4796-b9e4-4de1c477401a`::Skipping errors on recovery
>Traceback (most recent call last):
>  File "/usr/share/vdsm/vm.py", line 2233, in _startUnderlyingVm
>    self._run()
>  File "/usr/share/vdsm/vm.py", line 3093, in _run
>    devices = self.buildConfDevices()
>  File "/usr/share/vdsm/vm.py", line 2061, in buildConfDevices
>    self._normalizeVdsmImg(drv)
>  File "/usr/share/vdsm/vm.py", line 1993, in _normalizeVdsmImg
>    drv['volumeID'])
>RuntimeError: Volume ba8685ec-668a-4780-88ba-0b114e1a7e7d is corrupted or missing
>Thread-23::DEBUG::2014-01-28 00:14:36,087::vm::2247::vm.Vm::(_startUnderlyingVm) vmId=`01d705e3-6b62-4796-b9e4-4de1c477401a`::_ongoingCreations released
>Thread-31::DEBUG::2014-01-28 00:14:36,328::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "ff3a0c94-11dc-446d-b093-5f68ad81520d" not found\n  Skipping volume group ff3a0c94-11dc-446d-b093-5f68ad81520d\n'; <rc> = 5
>Thread-31::WARNING::2014-01-28 00:14:36,366::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "ff3a0c94-11dc-446d-b093-5f68ad81520d" not found', '  Skipping volume group ff3a0c94-11dc-446d-b093-5f68ad81520d']
>Thread-31::ERROR::2014-01-28 00:14:36,516::sdc::143::Storage.StorageDomainCache::(_findDomain) domain ff3a0c94-11dc-446d-b093-5f68ad81520d not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('ff3a0c94-11dc-446d-b093-5f68ad81520d',)
>Thread-33::ERROR::2014-01-28 00:14:36,531::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain ff3a0c94-11dc-446d-b093-5f68ad81520d
>Thread-33::ERROR::2014-01-28 00:14:36,540::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain ff3a0c94-11dc-446d-b093-5f68ad81520d
>Thread-31::ERROR::2014-01-28 00:14:36,603::dispatcher::67::Storage.Dispatcher.Protect::(run) {'status': {'message': "Storage domain does not exist: ('ff3a0c94-11dc-446d-b093-5f68ad81520d',)", 'code': 358}}
>Thread-31::ERROR::2014-01-28 00:14:36,614::vm::1991::vm.Vm::(_normalizeVdsmImg) vmId=`bbbd4efa-022d-44d3-8274-a0506bda2d5e`::Unable to get volume size for b0db0737-8168-4502-a236-1c3162f98f6a
>Traceback (most recent call last):
>  File "/usr/share/vdsm/vm.py", line 1987, in _normalizeVdsmImg
>    drv['truesize'] = res['truesize']
>KeyError: 'truesize'
>Thread-33::DEBUG::2014-01-28 00:14:36,628::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name ff3a0c94-11dc-446d-b093-5f68ad81520d' (cwd None)
>Thread-31::INFO::2014-01-28 00:14:36,636::vm::2244::vm.Vm::(_startUnderlyingVm) vmId=`bbbd4efa-022d-44d3-8274-a0506bda2d5e`::Skipping errors on recovery
>Traceback (most recent call last):
>  File "/usr/share/vdsm/vm.py", line 2233, in _startUnderlyingVm
>    self._run()
>  File "/usr/share/vdsm/vm.py", line 3093, in _run
>    devices = self.buildConfDevices()
>  File "/usr/share/vdsm/vm.py", line 2061, in buildConfDevices
>    self._normalizeVdsmImg(drv)
>  File "/usr/share/vdsm/vm.py", line 1993, in _normalizeVdsmImg
>    drv['volumeID'])
>RuntimeError: Volume b0db0737-8168-4502-a236-1c3162f98f6a is corrupted or missing
>Thread-31::DEBUG::2014-01-28 00:14:36,676::vm::2247::vm.Vm::(_startUnderlyingVm) vmId=`bbbd4efa-022d-44d3-8274-a0506bda2d5e`::_ongoingCreations released
>Thread-29::DEBUG::2014-01-28 00:14:36,846::vm::4972::vm.Vm::(_getUnderlyingDriveInfo) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Looking for drive with attributes {'name': u'hdc', 'bootOrder': '', 'boot': [], 'readonly': True, 'address': {u'bus': u'1', u'controller': u'0', u'type': u'drive', u'target': u'0', u'unit': u'0'}, 'path': '', 'type': u'cdrom'}
>Thread-29::DEBUG::2014-01-28 00:14:36,847::vm::4983::vm.Vm::(_getUnderlyingDriveInfo) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Matched {'name': (u'hdc', u'hdc'), 'bootOrder': ('', None), 'boot': ([], None), 'readonly': (True, True), 'address': ({u'bus': u'1', u'controller': u'0', u'type': u'drive', u'target': u'0', u'unit': u'0'}, {u'bus': u'1', u'controller': u'0', u'type': u'drive', u'target': u'0', u'unit': u'0'}), 'path': ('', ''), 'type': (u'cdrom', u'cdrom')}
>Thread-29::DEBUG::2014-01-28 00:14:36,850::vm::4994::vm.Vm::(_getUnderlyingDriveInfo) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Matched {'name': (u'hdc', u'hdc'), 'bootOrder': ('', None), 'boot': ([], None), 'readonly': (True, 'True'), 'address': ({u'bus': u'1', u'controller': u'0', u'type': u'drive', u'target': u'0', u'unit': u'0'}, {u'bus': u'1', u'controller': u'0', u'type': u'drive', u'target': u'0', u'unit': u'0'}), 'path': ('', ''), 'type': (u'cdrom', 'disk')}
>Thread-29::DEBUG::2014-01-28 00:14:36,851::vm::4972::vm.Vm::(_getUnderlyingDriveInfo) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Looking for drive with attributes {'name': u'vda', 'bootOrder': u'1', 'boot': [<DOM Element: boot at 0x7f19681653b0>], 'readonly': False, 'address': {u'slot': u'0x06', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, 'path': u'/var/run/vdsm/storage/562f0160-7b80-42e2-b248-5754455c40fc/daa7fa08-1354-4b19-a80f-8694ea63f914/8d9c27b1-8b26-4a44-8be2-79f024a56e0f', 'type': u'disk'}
>Thread-29::DEBUG::2014-01-28 00:14:36,852::vm::4983::vm.Vm::(_getUnderlyingDriveInfo) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Matched {'name': (u'vda', u'vda'), 'bootOrder': (u'1', u'1'), 'boot': ([<DOM Element: boot at 0x7f19681653b0>], None), 'readonly': (False, False), 'address': ({u'slot': u'0x06', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, {u'slot': u'0x06', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}), 'path': (u'/var/run/vdsm/storage/562f0160-7b80-42e2-b248-5754455c40fc/daa7fa08-1354-4b19-a80f-8694ea63f914/8d9c27b1-8b26-4a44-8be2-79f024a56e0f', '/var/run/vdsm/storage/562f0160-7b80-42e2-b248-5754455c40fc/daa7fa08-1354-4b19-a80f-8694ea63f914/8d9c27b1-8b26-4a44-8be2-79f024a56e0f'), 'type': (u'disk', u'disk')}
>Thread-29::DEBUG::2014-01-28 00:14:36,853::vm::4994::vm.Vm::(_getUnderlyingDriveInfo) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Matched {'name': (u'vda', u'vda'), 'bootOrder': (u'1', u'1'), 'boot': ([<DOM Element: boot at 0x7f19681653b0>], None), 'readonly': (False, 'False'), 'address': ({u'slot': u'0x06', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, {u'slot': u'0x06', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}), 'path': (u'/var/run/vdsm/storage/562f0160-7b80-42e2-b248-5754455c40fc/daa7fa08-1354-4b19-a80f-8694ea63f914/8d9c27b1-8b26-4a44-8be2-79f024a56e0f', '/var/run/vdsm/storage/562f0160-7b80-42e2-b248-5754455c40fc/daa7fa08-1354-4b19-a80f-8694ea63f914/8d9c27b1-8b26-4a44-8be2-79f024a56e0f'), 'type': (u'disk', 'disk')}
>Thread-33::DEBUG::2014-01-28 00:14:36,921::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "ff3a0c94-11dc-446d-b093-5f68ad81520d" not found\n  Skipping volume group ff3a0c94-11dc-446d-b093-5f68ad81520d\n'; <rc> = 5
>Thread-33::WARNING::2014-01-28 00:14:36,964::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "ff3a0c94-11dc-446d-b093-5f68ad81520d" not found', '  Skipping volume group ff3a0c94-11dc-446d-b093-5f68ad81520d']
>Thread-33::ERROR::2014-01-28 00:14:37,116::sdc::143::Storage.StorageDomainCache::(_findDomain) domain ff3a0c94-11dc-446d-b093-5f68ad81520d not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('ff3a0c94-11dc-446d-b093-5f68ad81520d',)
>Thread-33::ERROR::2014-01-28 00:14:37,214::dispatcher::67::Storage.Dispatcher.Protect::(run) {'status': {'message': "Storage domain does not exist: ('ff3a0c94-11dc-446d-b093-5f68ad81520d',)", 'code': 358}}
>Thread-33::ERROR::2014-01-28 00:14:37,225::vm::1991::vm.Vm::(_normalizeVdsmImg) vmId=`48ad952b-4457-4ae1-bf85-93f869382298`::Unable to get volume size for 1732793c-bf4f-4f8d-a5b9-4f41e99154ba
>Traceback (most recent call last):
>  File "/usr/share/vdsm/vm.py", line 1987, in _normalizeVdsmImg
>    drv['truesize'] = res['truesize']
>KeyError: 'truesize'
>Thread-33::INFO::2014-01-28 00:14:37,246::vm::2244::vm.Vm::(_startUnderlyingVm) vmId=`48ad952b-4457-4ae1-bf85-93f869382298`::Skipping errors on recovery
>Traceback (most recent call last):
>  File "/usr/share/vdsm/vm.py", line 2233, in _startUnderlyingVm
>    self._run()
>  File "/usr/share/vdsm/vm.py", line 3093, in _run
>    devices = self.buildConfDevices()
>  File "/usr/share/vdsm/vm.py", line 2061, in buildConfDevices
>    self._normalizeVdsmImg(drv)
>  File "/usr/share/vdsm/vm.py", line 1993, in _normalizeVdsmImg
>    drv['volumeID'])
>RuntimeError: Volume 1732793c-bf4f-4f8d-a5b9-4f41e99154ba is corrupted or missing
>Thread-33::DEBUG::2014-01-28 00:14:37,267::vm::2247::vm.Vm::(_startUnderlyingVm) vmId=`48ad952b-4457-4ae1-bf85-93f869382298`::_ongoingCreations released
>Thread-69::DEBUG::2014-01-28 00:14:37,370::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.11]
>Thread-12::DEBUG::2014-01-28 00:14:37,391::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-69::INFO::2014-01-28 00:14:37,437::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-69::INFO::2014-01-28 00:14:37,460::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00061236', 'lastCheck': '3.3', 'valid': True}}
>Thread-12::DEBUG::2014-01-28 00:14:37,605::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-70::DEBUG::2014-01-28 00:14:37,895::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call getCapabilities with () {}
>Thread-70::DEBUG::2014-01-28 00:14:37,909::BindingXMLRPC::976::vds::(wrapper) return getCapabilities with {'status': {'message': 'Recovering from crash or Initializing', 'code': 99}}
>Thread-29::DEBUG::2014-01-28 00:14:38,294::sampling::285::vm.Vm::(start) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Start statistics collection
>Thread-71::DEBUG::2014-01-28 00:14:38,296::sampling::314::vm.Vm::(run) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Stats thread started
>Thread-29::DEBUG::2014-01-28 00:14:38,324::vmChannels::194::vds::(register) Add fileno 19 to listener's channels.
>Thread-29::WARNING::2014-01-28 00:14:38,328::vm::3655::vm.Vm::(_readPauseCode) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::_readPauseCode unsupported by libvirt vm
>Thread-29::DEBUG::2014-01-28 00:14:38,345::vm::2247::vm.Vm::(_startUnderlyingVm) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::_ongoingCreations released
>Thread-72::DEBUG::2014-01-28 00:14:38,531::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10] flowID [195dc414]
>Thread-72::INFO::2014-01-28 00:14:38,532::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-72::ERROR::2014-01-28 00:14:38,534::dispatcher::67::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('5849b030-626e-47cb-ad90-3ce782d831b3',)", 'code': 309}}
>Thread-73::DEBUG::2014-01-28 00:14:38,544::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10] flowID [195dc414]
>Thread-73::INFO::2014-01-28 00:14:38,545::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None)
>Thread-73::ERROR::2014-01-28 00:14:38,546::dispatcher::67::Storage.Dispatcher.Protect::(run) {'status': {'message': 'Not SPM: ()', 'code': 654}}
>Thread-74::DEBUG::2014-01-28 00:14:38,620::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call vmGetStats with ('2d425c4c-91b1-4564-b276-f27cc581e6a1',) {}
>Thread-74::DEBUG::2014-01-28 00:14:38,620::BindingXMLRPC::976::vds::(wrapper) return vmGetStats with {'status': {'message': 'Recovering from crash or Initializing', 'code': 99}}
>VM Channels Listener::DEBUG::2014-01-28 00:14:38,854::vmChannels::102::vds::(_do_add_channels) fileno 19 was added to unconnected channels.
>VM Channels Listener::DEBUG::2014-01-28 00:14:38,855::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 19.
>VM Channels Listener::DEBUG::2014-01-28 00:14:38,855::guestIF::147::vm.Vm::(_connect) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Attempting connection to /var/lib/libvirt/qemu/channels/2d425c4c-91b1-4564-b276-f27cc581e6a1.com.redhat.rhevm.vdsm
>VM Channels Listener::DEBUG::2014-01-28 00:14:38,857::guestIF::150::vm.Vm::(_connect) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Connected to /var/lib/libvirt/qemu/channels/2d425c4c-91b1-4564-b276-f27cc581e6a1.com.redhat.rhevm.vdsm
>VM Channels Listener::DEBUG::2014-01-28 00:14:38,858::vmChannels::144::vds::(_handle_unconnected) Connecting to fileno 19 succeeded.
>clientIFinit::DEBUG::2014-01-28 00:14:39,116::clientIF::518::vds::(_cleanOldFiles) removing old file 053f5e53-7562-47de-a9a9-b44680a56257.recovery
>clientIFinit::DEBUG::2014-01-28 00:14:39,117::clientIF::518::vds::(_cleanOldFiles) removing old file 72a65e33-1dec-4557-bd76-73fed2da2939.recovery
>clientIFinit::DEBUG::2014-01-28 00:14:39,117::clientIF::518::vds::(_cleanOldFiles) removing old file d16e9ccd-1944-4860-b9fe-266d59faac3d.recovery
>clientIFinit::DEBUG::2014-01-28 00:14:39,117::clientIF::518::vds::(_cleanOldFiles) removing old file dde94c04-e899-4310-82c9-29652d8a9ff1.recovery
>clientIFinit::DEBUG::2014-01-28 00:14:39,118::clientIF::518::vds::(_cleanOldFiles) removing old file 2e74cd6f-f81c-471f-8e7a-22048270cb8e.recovery
>clientIFinit::DEBUG::2014-01-28 00:14:39,118::clientIF::518::vds::(_cleanOldFiles) removing old file 8e7e3da4-1bed-4d84-83a2-ebb308c827cb.recovery
>clientIFinit::DEBUG::2014-01-28 00:14:39,118::clientIF::518::vds::(_cleanOldFiles) removing old file 8bf37c46-8472-4fec-ae31-2821ea104df6.recovery
>clientIFinit::DEBUG::2014-01-28 00:14:39,118::clientIF::518::vds::(_cleanOldFiles) removing old file 99288fd4-97a8-4cbe-8d52-1abfbb960cfd.recovery
>clientIFinit::DEBUG::2014-01-28 00:14:39,119::clientIF::518::vds::(_cleanOldFiles) removing old file 0f69fdc7-895c-4113-bf6b-ba166bc98b11.recovery
>clientIFinit::DEBUG::2014-01-28 00:14:39,119::clientIF::518::vds::(_cleanOldFiles) removing old file cd7c5e5e-26b6-4f6a-94da-52c23f09be3b.recovery
>clientIFinit::DEBUG::2014-01-28 00:14:39,119::clientIF::518::vds::(_cleanOldFiles) removing old file ff9b04f1-aaf7-4d28-b86d-25ce882de4d6.recovery
>clientIFinit::DEBUG::2014-01-28 00:14:39,119::clientIF::518::vds::(_cleanOldFiles) removing old file 75de8ad7-bd7f-4142-bb5d-fb88b4b60823.recovery
>clientIFinit::DEBUG::2014-01-28 00:14:39,120::clientIF::518::vds::(_cleanOldFiles) removing old file f743c9ed-4fd6-4b23-9afa-1e4bde50e80d.recovery
>clientIFinit::DEBUG::2014-01-28 00:14:39,120::clientIF::518::vds::(_cleanOldFiles) removing old file f5e66f17-f97f-4566-b31f-87b80ccfb312.recovery
>clientIFinit::DEBUG::2014-01-28 00:14:39,120::clientIF::518::vds::(_cleanOldFiles) removing old file fc4c95be-dc58-462d-af27-b1ec233908a2.recovery
>clientIFinit::DEBUG::2014-01-28 00:14:39,120::clientIF::518::vds::(_cleanOldFiles) removing old file 64c80740-480e-4391-b13b-92d02e968539.recovery
>clientIFinit::DEBUG::2014-01-28 00:14:39,121::clientIF::518::vds::(_cleanOldFiles) removing old file bf4d43c0-8499-4241-8d14-65d57130b18b.recovery
>clientIFinit::DEBUG::2014-01-28 00:14:39,121::clientIF::518::vds::(_cleanOldFiles) removing old file 6cc9a918-51d6-431a-bc34-0217c94a4ebc.recovery
>clientIFinit::DEBUG::2014-01-28 00:14:39,121::clientIF::518::vds::(_cleanOldFiles) removing old file f77abc92-b0d7-4ee5-84cb-e880a6b3d813.recovery
>clientIFinit::DEBUG::2014-01-28 00:14:39,122::clientIF::518::vds::(_cleanOldFiles) removing old file 44b0e8a5-340f-4517-afdf-41bb9773d7db.recovery
>clientIFinit::DEBUG::2014-01-28 00:14:39,122::clientIF::518::vds::(_cleanOldFiles) removing old file f1e3dc65-d9c4-441e-89ff-7d0d921beee8.recovery
>clientIFinit::DEBUG::2014-01-28 00:14:39,122::clientIF::518::vds::(_cleanOldFiles) removing old file 2ca64185-267a-40df-a706-5c42c17de5ba.recovery
>clientIFinit::DEBUG::2014-01-28 00:14:39,122::clientIF::518::vds::(_cleanOldFiles) removing old file 12f80feb-7eb9-4041-bf40-8ac717bc1080.recovery
>clientIFinit::INFO::2014-01-28 00:14:39,123::logUtils::44::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList(options=None)
>clientIFinit::INFO::2014-01-28 00:14:39,123::logUtils::47::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList, Return response: {'poollist': []}
>VM Channels Listener::DEBUG::2014-01-28 00:14:39,279::guestIF::209::vm.Vm::(_handleMessage) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::username: u'root'
>Thread-12::DEBUG::2014-01-28 00:14:39,957::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:14:39,977::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:40,870::vm::587::vm.Vm::(_getCpuStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::CPU stats not available: 'NoneType' object is unsubscriptable
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:40,870::vm::621::vm.Vm::(_getNetworkStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Network stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:40,870::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:40,871::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:40,871::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:40,871::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:40,871::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:40,872::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:40,872::vm::587::vm.Vm::(_getCpuStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::CPU stats not available: 'NoneType' object is unsubscriptable
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:40,873::vm::621::vm.Vm::(_getNetworkStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Network stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:40,873::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:40,873::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:40,873::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:40,874::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:40,874::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:40,874::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-75::DEBUG::2014-01-28 00:14:41,026::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call getCapabilities with () {} flowID [430df2fc]
>Thread-75::DEBUG::2014-01-28 00:14:41,172::utils::555::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-75::DEBUG::2014-01-28 00:14:41,203::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-75::DEBUG::2014-01-28 00:14:41,230::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-75::DEBUG::2014-01-28 00:14:41,269::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-75::DEBUG::2014-01-28 00:14:41,291::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.2' (cwd None)
>Thread-75::DEBUG::2014-01-28 00:14:41,311::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-75::DEBUG::2014-01-28 00:14:41,312::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.2' (cwd None)
>Thread-75::DEBUG::2014-01-28 00:14:41,330::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-75::DEBUG::2014-01-28 00:14:41,333::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.3' (cwd None)
>Thread-75::DEBUG::2014-01-28 00:14:41,351::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-75::DEBUG::2014-01-28 00:14:41,352::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.3' (cwd None)
>Thread-75::DEBUG::2014-01-28 00:14:41,371::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-75::DEBUG::2014-01-28 00:14:41,374::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.10' (cwd None)
>Thread-75::DEBUG::2014-01-28 00:14:41,392::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-75::DEBUG::2014-01-28 00:14:41,393::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.10' (cwd None)
>Thread-75::DEBUG::2014-01-28 00:14:41,412::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-75::DEBUG::2014-01-28 00:14:41,416::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.11' (cwd None)
>Thread-75::DEBUG::2014-01-28 00:14:41,434::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-75::DEBUG::2014-01-28 00:14:41,435::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.11' (cwd None)
>Thread-75::DEBUG::2014-01-28 00:14:41,454::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-75::DEBUG::2014-01-28 00:14:41,519::caps::453::root::(_getKeyPackages) rpm package glusterfs-rdma not found
>Thread-75::DEBUG::2014-01-28 00:14:41,519::caps::453::root::(_getKeyPackages) rpm package glusterfs-fuse not found
>Thread-75::DEBUG::2014-01-28 00:14:41,521::caps::453::root::(_getKeyPackages) rpm package gluster-swift not found
>Thread-75::DEBUG::2014-01-28 00:14:41,527::caps::453::root::(_getKeyPackages) rpm package gluster-swift-object not found
>Thread-75::DEBUG::2014-01-28 00:14:41,527::caps::453::root::(_getKeyPackages) rpm package glusterfs not found
>Thread-75::DEBUG::2014-01-28 00:14:41,528::caps::453::root::(_getKeyPackages) rpm package gluster-swift-plugin not found
>Thread-75::DEBUG::2014-01-28 00:14:41,535::caps::453::root::(_getKeyPackages) rpm package gluster-swift-account not found
>Thread-75::DEBUG::2014-01-28 00:14:41,535::caps::453::root::(_getKeyPackages) rpm package gluster-swift-proxy not found
>Thread-75::DEBUG::2014-01-28 00:14:41,536::caps::453::root::(_getKeyPackages) rpm package gluster-swift-doc not found
>Thread-75::DEBUG::2014-01-28 00:14:41,536::caps::453::root::(_getKeyPackages) rpm package glusterfs-server not found
>Thread-75::DEBUG::2014-01-28 00:14:41,536::caps::453::root::(_getKeyPackages) rpm package gluster-swift-container not found
>Thread-75::DEBUG::2014-01-28 00:14:41,537::caps::453::root::(_getKeyPackages) rpm package glusterfs-geo-replication not found
>Thread-75::DEBUG::2014-01-28 00:14:41,557::BindingXMLRPC::976::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.3.1.el6.x86_64', 'buildtime': 1388745567.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '33.gitf029545.el6', 'buildtime': 1390094004L, 'version': '4.14.0'}, 'qemu-kvm': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.2', 'buildtime': 1387360004L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'mom': {'release': '20140117.gitfd877c5.el6', 'buildtime': 1389966957L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           L5320  @ 1.86GHz', 'hooks': {'before_vm_start': {'50_hostedengine': {'md5': '45dde62155b5412eafbfff5ef265acc2'}}}, 'cpuSockets': '2', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'gateway': '172.16.0.1', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.2', 'vnet0']}, 'os_net': {'iface': 'os_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.10']}, 'r2o_net': {'iface': 'r2o_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}, 'storage_network': {'iface': u'bond0.3', 'addr': '172.16.1.5', 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000', 'netmask': '255.255.255.255', 'bridged': False, 'interface': u'bond0.3', 'ipv6gateway': '::', 'gateway': ''}}, 'bridges': {'ovirtmgmt': {'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '172.16.0.1', 'ports': ['bond0.2', 'vnet0']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}, 'os_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.10']}, 'r2o_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}}, 'uuid': '44454C4C-3100-104D-8046-B9C04F583153', 'lastClientIface': 'ovirtmgmt', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'cb327caa-ca77-4efe-b7b8-63aba3cda84d', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:85', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth2', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:85', 'speed': 1000, 'mtu': '4000'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'e401a6a5-8773-4971-9213-b3c2bb52f16c', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:83', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:83', 'speed': 1000, 'mtu': '4000'}, 'eth0': {'netmask': '', 'addr': '', 'hwaddr': '00:e0:4c:80:13:02', 'cfg': {'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': '00:e0:4c:80:13:02', 'STP': 'no', 'DEVICE': 'eth0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'speed': 1000, 'mtu': '1500'}}, 'software_revision': '33', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,tm2,ssse3,cx16,xtpr,pdcm,dca,lahf_lm,dts,tpr_shadow,vnmi,flexpriority,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': ['eth1', 'eth2'], 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'MTU': '4000', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'BONDING_OPTS': 'miimon=100 mode=balance-rr', 'DEVICE': 'bond0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '32105', 'cpuSpeed': '1861.936', 'version_name': 'Snow Man', 'vlans': {'bond0.2': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'ovirtmgmt', 'VLAN': 'yes', 'MTU': '4000', 'BOOTPROTO': 'none', 'DEVICE': 'bond0.2', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 2, 'mtu': '4000'}, 'bond0.3': {'netmask': '255.255.255.255', 'iface': 'bond0', 'addr': '172.16.1.5', 'cfg': {'VLAN': 'yes', 'DEFROUTE': 'no', 'MTU': '4000', 'NM_CONTROLLED': 'no', 'DEVICE': 'bond0.3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 3, 'mtu': '4000'}, 'eth0.11': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'r2o_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.11', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 11, 'mtu': '1500'}, 'eth0.10': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'os_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.10', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 10, 'mtu': '1500'}}, 'cpuCores': '8', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '8', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '172.16.0.10'}}
>Thread-76::DEBUG::2014-01-28 00:14:41,590::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call getHardwareInfo with () {} flowID [430df2fc]
>Thread-76::DEBUG::2014-01-28 00:14:41,602::BindingXMLRPC::976::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'PowerEdge 1950', 'systemSerialNumber': '91MFX1S', 'systemFamily': 'Not Specified', 'systemVersion': 'Not Specified', 'systemUUID': '44454c4c-3100-104d-8046-b9c04f583153', 'systemManufacturer': 'Dell Inc.'}}
>Thread-77::DEBUG::2014-01-28 00:14:41,682::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call hostsList with () {} flowID [430df2fc]
>Thread-77::DEBUG::2014-01-28 00:14:41,768::BindingXMLRPC::976::vds::(wrapper) return hostsList with {'status': {'message': 'Done', 'code': 0}, 'hosts': [{'status': 'CONNECTED', 'hostname': '172.16.0.11', 'uuid': '1fba63f6-2109-40a1-bd33-e051fa939b81'}]}
>Thread-79::DEBUG::2014-01-28 00:14:41,833::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call vmGetStats with ('9c9befe0-9e17-462e-9bce-27ddd684a6d3',) {} flowID [430df2fc]
>Thread-79::DEBUG::2014-01-28 00:14:41,833::BindingXMLRPC::976::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Paused', 'hash': '0', 'acpiEnable': 'true', 'pid': '7673', 'displayIp': '0', 'displayPort': u'5907', 'displaySecurePort': u'5908', 'timeOffset': 39598L, 'kvmEnable': 'true', 'network': {}, 'vmId': '9c9befe0-9e17-462e-9bce-27ddd684a6d3', 'displayType': 'qxl', 'disks': {}, 'monitorResponse': '0', 'elapsedTime': '453114', 'vmType': 'kvm', 'clientIp': ''}]}
>Thread-81::DEBUG::2014-01-28 00:14:41,977::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10] flowID [143a11c8]
>Thread-81::INFO::2014-01-28 00:14:41,978::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=7, spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', conList=[{'port': '', 'connection': '172.16.1.5:/VM-DATA', 'iqn': '', 'portal': '', 'user': '', 'vfs_type': 'glusterfs', 'password': '******', 'id': '594fb154-104f-4248-b87b-4d01cf9b5325'}], options=None)
>Thread-81::DEBUG::2014-01-28 00:14:41,983::hsm::2285::Storage.HSM::(__prefetchDomains) glusterDomPath: glusterSD/*
>Thread-81::DEBUG::2014-01-28 00:14:41,987::hsm::2297::Storage.HSM::(__prefetchDomains) Found SD uuids: ()
>Thread-81::DEBUG::2014-01-28 00:14:41,988::hsm::2353::Storage.HSM::(connectStorageServer) knownSDs: {562f0160-7b80-42e2-b248-5754455c40fc: storage.nfsSD.findDomain}
>Thread-81::INFO::2014-01-28 00:14:41,988::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '594fb154-104f-4248-b87b-4d01cf9b5325'}]}
>Thread-12::DEBUG::2014-01-28 00:14:41,999::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:14:42,021::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-83::DEBUG::2014-01-28 00:14:42,047::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10] flowID [143a11c8]
>Thread-83::INFO::2014-01-28 00:14:42,048::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', conList=[{'connection': 'engine.melb.example.net:/var/lib/exports/iso', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'f414e026-440e-46ac-ac3f-213042ee2cce', 'port': ''}], options=None)
>Thread-83::DEBUG::2014-01-28 00:14:42,053::hsm::2273::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso
>Thread-83::DEBUG::2014-01-28 00:14:42,055::hsm::2297::Storage.HSM::(__prefetchDomains) Found SD uuids: ('436c344c-bc57-441b-b311-c9595c6039e1',)
>Thread-83::DEBUG::2014-01-28 00:14:42,055::hsm::2353::Storage.HSM::(connectStorageServer) knownSDs: {562f0160-7b80-42e2-b248-5754455c40fc: storage.nfsSD.findDomain, 436c344c-bc57-441b-b311-c9595c6039e1: storage.nfsSD.findDomain}
>Thread-83::INFO::2014-01-28 00:14:42,055::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'f414e026-440e-46ac-ac3f-213042ee2cce'}]}
>Thread-84::DEBUG::2014-01-28 00:14:42,121::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-84::INFO::2014-01-28 00:14:42,122::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', hostID=1, msdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', masterVersion=1, options=None)
>Thread-84::INFO::2014-01-28 00:14:42,125::sp::683::Storage.StoragePool::(connect) Connect host #1 to the storage pool 5849b030-626e-47cb-ad90-3ce782d831b3 with master domain: ff3a0c94-11dc-446d-b093-5f68ad81520d (ver = 1)
>Thread-84::DEBUG::2014-01-28 00:14:42,127::iscsi::393::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-84::DEBUG::2014-01-28 00:14:42,127::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-84::DEBUG::2014-01-28 00:14:42,157::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-84::DEBUG::2014-01-28 00:14:42,268::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-84::ERROR::2014-01-28 00:14:42,272::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain ff3a0c94-11dc-446d-b093-5f68ad81520d
>Thread-84::ERROR::2014-01-28 00:14:42,272::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain ff3a0c94-11dc-446d-b093-5f68ad81520d
>Thread-84::DEBUG::2014-01-28 00:14:42,274::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name ff3a0c94-11dc-446d-b093-5f68ad81520d' (cwd None)
>Thread-84::DEBUG::2014-01-28 00:14:42,510::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "ff3a0c94-11dc-446d-b093-5f68ad81520d" not found\n  Skipping volume group ff3a0c94-11dc-446d-b093-5f68ad81520d\n'; <rc> = 5
>Thread-84::WARNING::2014-01-28 00:14:42,512::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "ff3a0c94-11dc-446d-b093-5f68ad81520d" not found', '  Skipping volume group ff3a0c94-11dc-446d-b093-5f68ad81520d']
>Thread-84::ERROR::2014-01-28 00:14:42,535::sdc::143::Storage.StorageDomainCache::(_findDomain) domain ff3a0c94-11dc-446d-b093-5f68ad81520d not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('ff3a0c94-11dc-446d-b093-5f68ad81520d',)
>Thread-84::ERROR::2014-01-28 00:14:42,538::dispatcher::67::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=ff3a0c94-11dc-446d-b093-5f68ad81520d'", 'code': 304}}
>Thread-89::INFO::2014-01-28 00:14:42,609::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-89::INFO::2014-01-28 00:14:42,609::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00061236', 'lastCheck': '8.4', 'valid': True}}
>Thread-90::DEBUG::2014-01-28 00:14:42,855::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call setMOMPolicyParameters with ({'balloonEnabled': True},) {} flowID [143a11c8]
>Thread-90::DEBUG::2014-01-28 00:14:42,921::BindingXMLRPC::976::vds::(wrapper) return setMOMPolicyParameters with {'status': {'message': 'Done', 'code': 0}}
>Thread-91::DEBUG::2014-01-28 00:14:42,957::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call hostUUIDGet with () {} flowID [143a11c8]
>Thread-91::DEBUG::2014-01-28 00:14:42,995::BindingXMLRPC::976::vds::(wrapper) return hostUUIDGet with {'status': {'message': 'Done', 'code': 0}, 'uuid': '1fba63f6-2109-40a1-bd33-e051fa939b81'}
>Thread-92::DEBUG::2014-01-28 00:14:43,606::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {}
>Thread-92::DEBUG::2014-01-28 00:14:43,651::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'OFFLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-12::DEBUG::2014-01-28 00:14:44,038::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:14:44,063::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>clientIFinit::INFO::2014-01-28 00:14:44,130::logUtils::44::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList(options=None)
>clientIFinit::INFO::2014-01-28 00:14:44,131::logUtils::47::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList, Return response: {'poollist': []}
>Thread-17::DEBUG::2014-01-28 00:14:44,224::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc
>Thread-17::DEBUG::2014-01-28 00:14:44,227::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-17::DEBUG::2014-01-28 00:14:44,237::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=562f0160-7b80-42e2-b248-5754455c40fc:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=7e0cc075-eee7-4268-946e-33a79dd08748', 'REMOTE_PATH=172.16.1.5:/HOSTED-ENGINE', 'ROLE=Master', 'SDUUID=562f0160-7b80-42e2-b248-5754455c40fc', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=14825cb0b944c5c2ae9ed025c82e06730a2d91c4']
>Thread-17::DEBUG::2014-01-28 00:14:44,239::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-17::INFO::2014-01-28 00:14:44,239::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_imageNS already registered
>Thread-17::INFO::2014-01-28 00:14:44,239::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_volumeNS already registered
>Thread-17::DEBUG::2014-01-28 00:14:44,254::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-17::DEBUG::2014-01-28 00:14:44,283::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n478 bytes (478 B) copied, 0.000993214 s, 481 kB/s\n'; <rc> = 0
>Thread-96::INFO::2014-01-28 00:14:44,569::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-96::INFO::2014-01-28 00:14:44,569::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000993214', 'lastCheck': '0.3', 'valid': True}}
>Thread-97::DEBUG::2014-01-28 00:14:45,631::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call getCapabilities with () {}
>Thread-97::DEBUG::2014-01-28 00:14:45,633::utils::555::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-97::DEBUG::2014-01-28 00:14:45,652::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-97::DEBUG::2014-01-28 00:14:45,678::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-97::DEBUG::2014-01-28 00:14:45,700::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-97::DEBUG::2014-01-28 00:14:45,718::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.2' (cwd None)
>Thread-97::DEBUG::2014-01-28 00:14:45,737::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-97::DEBUG::2014-01-28 00:14:45,738::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.2' (cwd None)
>Thread-97::DEBUG::2014-01-28 00:14:45,757::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-97::DEBUG::2014-01-28 00:14:45,760::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.3' (cwd None)
>Thread-97::DEBUG::2014-01-28 00:14:45,779::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-97::DEBUG::2014-01-28 00:14:45,780::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.3' (cwd None)
>Thread-97::DEBUG::2014-01-28 00:14:45,800::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-97::DEBUG::2014-01-28 00:14:45,803::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.10' (cwd None)
>Thread-97::DEBUG::2014-01-28 00:14:45,822::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-97::DEBUG::2014-01-28 00:14:45,823::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.10' (cwd None)
>Thread-97::DEBUG::2014-01-28 00:14:45,843::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-97::DEBUG::2014-01-28 00:14:45,847::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.11' (cwd None)
>Thread-97::DEBUG::2014-01-28 00:14:45,866::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-97::DEBUG::2014-01-28 00:14:45,867::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.11' (cwd None)
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:45,894::vm::587::vm.Vm::(_getCpuStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::CPU stats not available: 'NoneType' object is unsubscriptable
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:45,898::vm::621::vm.Vm::(_getNetworkStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Network stats not available
>Thread-97::DEBUG::2014-01-28 00:14:45,902::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:45,902::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:45,907::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:45,912::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:45,915::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:45,917::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:45,920::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:45,929::vm::587::vm.Vm::(_getCpuStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::CPU stats not available: 'NoneType' object is unsubscriptable
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:45,932::vm::621::vm.Vm::(_getNetworkStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Network stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:45,936::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:45,940::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:45,944::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:45,947::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-97::DEBUG::2014-01-28 00:14:45,952::caps::453::root::(_getKeyPackages) rpm package glusterfs-rdma not found
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:45,954::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>Thread-97::DEBUG::2014-01-28 00:14:45,956::caps::453::root::(_getKeyPackages) rpm package glusterfs-fuse not found
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:45,957::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-97::DEBUG::2014-01-28 00:14:45,960::caps::453::root::(_getKeyPackages) rpm package gluster-swift not found
>Thread-97::DEBUG::2014-01-28 00:14:45,971::caps::453::root::(_getKeyPackages) rpm package gluster-swift-object not found
>Thread-97::DEBUG::2014-01-28 00:14:45,972::caps::453::root::(_getKeyPackages) rpm package glusterfs not found
>Thread-97::DEBUG::2014-01-28 00:14:45,972::caps::453::root::(_getKeyPackages) rpm package gluster-swift-plugin not found
>Thread-97::DEBUG::2014-01-28 00:14:45,979::caps::453::root::(_getKeyPackages) rpm package gluster-swift-account not found
>Thread-97::DEBUG::2014-01-28 00:14:45,979::caps::453::root::(_getKeyPackages) rpm package gluster-swift-proxy not found
>Thread-97::DEBUG::2014-01-28 00:14:45,980::caps::453::root::(_getKeyPackages) rpm package gluster-swift-doc not found
>Thread-97::DEBUG::2014-01-28 00:14:45,980::caps::453::root::(_getKeyPackages) rpm package glusterfs-server not found
>Thread-97::DEBUG::2014-01-28 00:14:45,980::caps::453::root::(_getKeyPackages) rpm package gluster-swift-container not found
>Thread-97::DEBUG::2014-01-28 00:14:45,981::caps::453::root::(_getKeyPackages) rpm package glusterfs-geo-replication not found
>Thread-97::DEBUG::2014-01-28 00:14:45,986::BindingXMLRPC::976::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.3.1.el6.x86_64', 'buildtime': 1388745567.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '33.gitf029545.el6', 'buildtime': 1390094004L, 'version': '4.14.0'}, 'qemu-kvm': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.2', 'buildtime': 1387360004L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'mom': {'release': '20140117.gitfd877c5.el6', 'buildtime': 1389966957L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           L5320  @ 1.86GHz', 'hooks': {'before_vm_start': {'50_hostedengine': {'md5': '45dde62155b5412eafbfff5ef265acc2'}}}, 'cpuSockets': '2', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'gateway': '172.16.0.1', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.2', 'vnet0']}, 'os_net': {'iface': 'os_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.10']}, 'r2o_net': {'iface': 'r2o_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}, 'storage_network': {'iface': u'bond0.3', 'addr': '172.16.1.5', 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000', 'netmask': '255.255.255.255', 'bridged': False, 'interface': u'bond0.3', 'ipv6gateway': '::', 'gateway': ''}}, 'bridges': {'ovirtmgmt': {'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '172.16.0.1', 'ports': ['bond0.2', 'vnet0']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}, 'os_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.10']}, 'r2o_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}}, 'uuid': '44454C4C-3100-104D-8046-B9C04F583153', 'lastClientIface': 'ovirtmgmt', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'cb327caa-ca77-4efe-b7b8-63aba3cda84d', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:85', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth2', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:85', 'speed': 1000, 'mtu': '4000'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'e401a6a5-8773-4971-9213-b3c2bb52f16c', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:83', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:83', 'speed': 1000, 'mtu': '4000'}, 'eth0': {'netmask': '', 'addr': '', 'hwaddr': '00:e0:4c:80:13:02', 'cfg': {'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': '00:e0:4c:80:13:02', 'STP': 'no', 'DEVICE': 'eth0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'speed': 1000, 'mtu': '1500'}}, 'software_revision': '33', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,tm2,ssse3,cx16,xtpr,pdcm,dca,lahf_lm,dts,tpr_shadow,vnmi,flexpriority,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': ['eth1', 'eth2'], 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'MTU': '4000', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'BONDING_OPTS': 'miimon=100 mode=balance-rr', 'DEVICE': 'bond0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '32105', 'cpuSpeed': '1861.936', 'version_name': 'Snow Man', 'vlans': {'bond0.2': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'ovirtmgmt', 'VLAN': 'yes', 'MTU': '4000', 'BOOTPROTO': 'none', 'DEVICE': 'bond0.2', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 2, 'mtu': '4000'}, 'bond0.3': {'netmask': '255.255.255.255', 'iface': 'bond0', 'addr': '172.16.1.5', 'cfg': {'VLAN': 'yes', 'DEFROUTE': 'no', 'MTU': '4000', 'NM_CONTROLLED': 'no', 'DEVICE': 'bond0.3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 3, 'mtu': '4000'}, 'eth0.11': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'r2o_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.11', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 11, 'mtu': '1500'}, 'eth0.10': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'os_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.10', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 10, 'mtu': '1500'}}, 'cpuCores': '8', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '8', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '172.16.0.11'}}
>Thread-12::DEBUG::2014-01-28 00:14:46,081::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:14:46,102::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-98::INFO::2014-01-28 00:14:46,220::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-98::INFO::2014-01-28 00:14:46,220::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000993214', 'lastCheck': '1.9', 'valid': True}}
>Thread-99::DEBUG::2014-01-28 00:14:46,255::vm::587::vm.Vm::(_getCpuStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::CPU stats not available: 'NoneType' object is unsubscriptable
>Thread-99::DEBUG::2014-01-28 00:14:46,255::vm::621::vm.Vm::(_getNetworkStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Network stats not available
>Thread-99::DEBUG::2014-01-28 00:14:46,256::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-99::DEBUG::2014-01-28 00:14:46,256::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>Thread-99::DEBUG::2014-01-28 00:14:46,256::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-99::DEBUG::2014-01-28 00:14:46,257::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-99::DEBUG::2014-01-28 00:14:46,257::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>Thread-99::DEBUG::2014-01-28 00:14:46,257::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-101::DEBUG::2014-01-28 00:14:47,915::BindingXMLRPC::159::vds::(wrapper) client [127.0.0.1]
>Thread-101::INFO::2014-01-28 00:14:47,916::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='7e0cc075-eee7-4268-946e-33a79dd08748', conList=[{'port': '', 'connection': '172.16.1.5:/HOSTED-ENGINE', 'iqn': '', 'portal': '', 'user': 'kvm', 'protocol_version': '3', 'password': '******', 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}], options=None)
>Thread-101::DEBUG::2014-01-28 00:14:47,923::hsm::2273::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE
>Thread-101::DEBUG::2014-01-28 00:14:47,925::hsm::2297::Storage.HSM::(__prefetchDomains) Found SD uuids: ('562f0160-7b80-42e2-b248-5754455c40fc',)
>Thread-101::DEBUG::2014-01-28 00:14:47,925::hsm::2353::Storage.HSM::(connectStorageServer) knownSDs: {562f0160-7b80-42e2-b248-5754455c40fc: storage.nfsSD.findDomain, 436c344c-bc57-441b-b311-c9595c6039e1: storage.nfsSD.findDomain}
>Thread-101::INFO::2014-01-28 00:14:47,926::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}]}
>Thread-12::DEBUG::2014-01-28 00:14:48,119::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:14:48,144::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-102::DEBUG::2014-01-28 00:14:48,207::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.11]
>Thread-102::INFO::2014-01-28 00:14:48,208::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-102::INFO::2014-01-28 00:14:48,208::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000993214', 'lastCheck': '3.9', 'valid': True}}
>Thread-103::DEBUG::2014-01-28 00:14:48,627::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call vmGetStats with ('2d425c4c-91b1-4564-b276-f27cc581e6a1',) {}
>Thread-103::DEBUG::2014-01-28 00:14:48,627::vm::587::vm.Vm::(_getCpuStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::CPU stats not available: 'NoneType' object is unsubscriptable
>Thread-103::DEBUG::2014-01-28 00:14:48,628::vm::621::vm.Vm::(_getNetworkStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Network stats not available
>Thread-103::DEBUG::2014-01-28 00:14:48,628::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-103::DEBUG::2014-01-28 00:14:48,628::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>Thread-103::DEBUG::2014-01-28 00:14:48,628::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-103::DEBUG::2014-01-28 00:14:48,629::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-103::DEBUG::2014-01-28 00:14:48,629::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>Thread-103::DEBUG::2014-01-28 00:14:48,629::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-103::DEBUG::2014-01-28 00:14:48,630::BindingXMLRPC::976::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'root', 'memUsage': '53', 'acpiEnable': 'true', 'guestFQDN': u'engine.melb.example.net', 'pid': '18113', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'/', u'total': '25323499520', u'used': '2592231424', u'fs': u'ext4'}, {u'path': u'/boot', u'total': '507744256', u'used': '55643136', u'fs': u'ext4'}], 'network': {u'vnet0': {'state': 'unknown', 'macAddr': '00:16:3e:0a:b7:55', 'speed': '1000', 'name': u'vnet0'}}, 'vmType': 'kvm', 'guestName': u'engine.melb.example.net', 'elapsedTime': '610873', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'mem_free': '1965672', u'swap_in': '0', u'pageflt': '407', u'mem_total': '3922728', u'mem_unused': '1431588'}, 'cpuSys': '0.00', 'appsList': [u'ovirt-guest-agent-1.0.8-1.el6', u'kernel-2.6.32-431.el6', u'kernel-2.6.32-431.3.1.el6'], 'guestOs': u'2.6.32-431.3.1.el6.x86_64', 'displayType': 'vnc', 'status': 'Up', 'hash': '2950633987041661405', 'vmId': '2d425c4c-91b1-4564-b276-f27cc581e6a1', 'displayIp': '0', 'displayPort': u'5900', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'truesize': '3920592896', 'apparentsize': '26843545600', 'imageID': 'daa7fa08-1354-4b19-a80f-8694ea63f914'}, 'hdc': {'truesize': '0', 'apparentsize': '0'}}, 'monitorResponse': '0', 'statsAge': '0.29', 'cpuUser': '0.00', 'lastLogin': 1390828479.279238, 'guestIPs': u'172.16.0.10', 'netIfaces': [{u'hw': u'00:16:3e:0a:b7:55', u'name': u'eth0', u'inet': [u'172.16.0.10'], u'inet6': [u'fe80::216:3eff:fe0a:b755']}]}]}
>Thread-104::DEBUG::2014-01-28 00:14:48,644::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10] flowID [66f2577c]
>Thread-104::INFO::2014-01-28 00:14:48,645::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-104::ERROR::2014-01-28 00:14:48,647::dispatcher::67::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('5849b030-626e-47cb-ad90-3ce782d831b3',)", 'code': 309}}
>Thread-105::DEBUG::2014-01-28 00:14:48,688::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10] flowID [66f2577c]
>Thread-105::INFO::2014-01-28 00:14:48,689::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', hostID=1, msdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', masterVersion=1, options=None)
>Thread-105::INFO::2014-01-28 00:14:48,692::sp::683::Storage.StoragePool::(connect) Connect host #1 to the storage pool 5849b030-626e-47cb-ad90-3ce782d831b3 with master domain: ff3a0c94-11dc-446d-b093-5f68ad81520d (ver = 1)
>Thread-105::DEBUG::2014-01-28 00:14:48,693::iscsi::393::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-105::DEBUG::2014-01-28 00:14:48,694::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-105::DEBUG::2014-01-28 00:14:48,753::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-106::DEBUG::2014-01-28 00:14:48,798::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {}
>Thread-106::DEBUG::2014-01-28 00:14:48,861::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'OFFLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-105::DEBUG::2014-01-28 00:14:48,868::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-105::ERROR::2014-01-28 00:14:48,871::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain ff3a0c94-11dc-446d-b093-5f68ad81520d
>Thread-105::ERROR::2014-01-28 00:14:48,871::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain ff3a0c94-11dc-446d-b093-5f68ad81520d
>Thread-105::DEBUG::2014-01-28 00:14:48,874::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name ff3a0c94-11dc-446d-b093-5f68ad81520d' (cwd None)
>Thread-105::DEBUG::2014-01-28 00:14:49,122::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "ff3a0c94-11dc-446d-b093-5f68ad81520d" not found\n  Skipping volume group ff3a0c94-11dc-446d-b093-5f68ad81520d\n'; <rc> = 5
>Thread-105::WARNING::2014-01-28 00:14:49,123::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "ff3a0c94-11dc-446d-b093-5f68ad81520d" not found', '  Skipping volume group ff3a0c94-11dc-446d-b093-5f68ad81520d']
>clientIFinit::INFO::2014-01-28 00:14:49,144::logUtils::44::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList(options=None)
>clientIFinit::INFO::2014-01-28 00:14:49,148::logUtils::47::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList, Return response: {'poollist': []}
>Thread-105::ERROR::2014-01-28 00:14:49,174::sdc::143::Storage.StorageDomainCache::(_findDomain) domain ff3a0c94-11dc-446d-b093-5f68ad81520d not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('ff3a0c94-11dc-446d-b093-5f68ad81520d',)
>Thread-105::ERROR::2014-01-28 00:14:49,178::dispatcher::67::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=ff3a0c94-11dc-446d-b093-5f68ad81520d'", 'code': 304}}
>Thread-112::DEBUG::2014-01-28 00:14:49,555::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10] flowID [156ff99d]
>Thread-112::INFO::2014-01-28 00:14:49,556::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None)
>Thread-112::ERROR::2014-01-28 00:14:49,558::dispatcher::67::Storage.Dispatcher.Protect::(run) {'status': {'message': 'Not SPM: ()', 'code': 654}}
>Thread-12::DEBUG::2014-01-28 00:14:50,163::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:14:50,186::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:50,977::vm::587::vm.Vm::(_getCpuStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::CPU stats not available: 'NoneType' object is unsubscriptable
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:50,977::vm::621::vm.Vm::(_getNetworkStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Network stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:50,978::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:50,978::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:50,978::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:50,979::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:50,979::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:50,979::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:50,980::vm::587::vm.Vm::(_getCpuStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::CPU stats not available: 'NoneType' object is unsubscriptable
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:50,980::vm::621::vm.Vm::(_getNetworkStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Network stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:50,980::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:50,981::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:50,981::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:50,981::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:50,982::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:50,982::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-12::DEBUG::2014-01-28 00:14:52,201::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:14:52,223::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-114::DEBUG::2014-01-28 00:14:53,971::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {}
>Thread-114::DEBUG::2014-01-28 00:14:54,017::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'OFFLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>clientIFinit::INFO::2014-01-28 00:14:54,179::logUtils::44::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList(options=None)
>clientIFinit::INFO::2014-01-28 00:14:54,180::logUtils::47::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList, Return response: {'poollist': []}
>Thread-12::DEBUG::2014-01-28 00:14:54,239::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:14:54,261::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-17::DEBUG::2014-01-28 00:14:54,310::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc
>Thread-17::DEBUG::2014-01-28 00:14:54,312::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-17::DEBUG::2014-01-28 00:14:54,323::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=562f0160-7b80-42e2-b248-5754455c40fc:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=7e0cc075-eee7-4268-946e-33a79dd08748', 'REMOTE_PATH=172.16.1.5:/HOSTED-ENGINE', 'ROLE=Master', 'SDUUID=562f0160-7b80-42e2-b248-5754455c40fc', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=14825cb0b944c5c2ae9ed025c82e06730a2d91c4']
>Thread-17::DEBUG::2014-01-28 00:14:54,327::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-17::INFO::2014-01-28 00:14:54,327::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_imageNS already registered
>Thread-17::INFO::2014-01-28 00:14:54,327::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_volumeNS already registered
>Thread-17::DEBUG::2014-01-28 00:14:54,341::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-17::DEBUG::2014-01-28 00:14:54,362::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n478 bytes (478 B) copied, 0.000582437 s, 821 kB/s\n'; <rc> = 0
>Thread-118::INFO::2014-01-28 00:14:54,578::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-118::INFO::2014-01-28 00:14:54,579::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000582437', 'lastCheck': '0.2', 'valid': True}}
>Thread-119::DEBUG::2014-01-28 00:14:55,642::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call getCapabilities with () {}
>Thread-119::DEBUG::2014-01-28 00:14:55,643::utils::555::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-119::DEBUG::2014-01-28 00:14:55,665::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-119::DEBUG::2014-01-28 00:14:55,692::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-119::DEBUG::2014-01-28 00:14:55,714::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-119::DEBUG::2014-01-28 00:14:55,733::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.2' (cwd None)
>Thread-119::DEBUG::2014-01-28 00:14:55,752::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-119::DEBUG::2014-01-28 00:14:55,753::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.2' (cwd None)
>Thread-119::DEBUG::2014-01-28 00:14:55,772::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-119::DEBUG::2014-01-28 00:14:55,775::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.3' (cwd None)
>Thread-119::DEBUG::2014-01-28 00:14:55,795::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-119::DEBUG::2014-01-28 00:14:55,796::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.3' (cwd None)
>Thread-119::DEBUG::2014-01-28 00:14:55,816::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-119::DEBUG::2014-01-28 00:14:55,819::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.10' (cwd None)
>Thread-119::DEBUG::2014-01-28 00:14:55,838::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-119::DEBUG::2014-01-28 00:14:55,839::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.10' (cwd None)
>Thread-119::DEBUG::2014-01-28 00:14:55,858::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-119::DEBUG::2014-01-28 00:14:55,862::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.11' (cwd None)
>Thread-119::DEBUG::2014-01-28 00:14:55,882::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-119::DEBUG::2014-01-28 00:14:55,883::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.11' (cwd None)
>Thread-119::DEBUG::2014-01-28 00:14:55,905::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-119::DEBUG::2014-01-28 00:14:55,914::caps::453::root::(_getKeyPackages) rpm package glusterfs-rdma not found
>Thread-119::DEBUG::2014-01-28 00:14:55,915::caps::453::root::(_getKeyPackages) rpm package glusterfs-fuse not found
>Thread-119::DEBUG::2014-01-28 00:14:55,916::caps::453::root::(_getKeyPackages) rpm package gluster-swift not found
>Thread-119::DEBUG::2014-01-28 00:14:55,922::caps::453::root::(_getKeyPackages) rpm package gluster-swift-object not found
>Thread-119::DEBUG::2014-01-28 00:14:55,922::caps::453::root::(_getKeyPackages) rpm package glusterfs not found
>Thread-119::DEBUG::2014-01-28 00:14:55,923::caps::453::root::(_getKeyPackages) rpm package gluster-swift-plugin not found
>Thread-119::DEBUG::2014-01-28 00:14:55,931::caps::453::root::(_getKeyPackages) rpm package gluster-swift-account not found
>Thread-119::DEBUG::2014-01-28 00:14:55,932::caps::453::root::(_getKeyPackages) rpm package gluster-swift-proxy not found
>Thread-119::DEBUG::2014-01-28 00:14:55,932::caps::453::root::(_getKeyPackages) rpm package gluster-swift-doc not found
>Thread-119::DEBUG::2014-01-28 00:14:55,932::caps::453::root::(_getKeyPackages) rpm package glusterfs-server not found
>Thread-119::DEBUG::2014-01-28 00:14:55,933::caps::453::root::(_getKeyPackages) rpm package gluster-swift-container not found
>Thread-119::DEBUG::2014-01-28 00:14:55,933::caps::453::root::(_getKeyPackages) rpm package glusterfs-geo-replication not found
>Thread-119::DEBUG::2014-01-28 00:14:55,938::BindingXMLRPC::976::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.3.1.el6.x86_64', 'buildtime': 1388745567.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '33.gitf029545.el6', 'buildtime': 1390094004L, 'version': '4.14.0'}, 'qemu-kvm': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.2', 'buildtime': 1387360004L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'mom': {'release': '20140117.gitfd877c5.el6', 'buildtime': 1389966957L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           L5320  @ 1.86GHz', 'hooks': {'before_vm_start': {'50_hostedengine': {'md5': '45dde62155b5412eafbfff5ef265acc2'}}}, 'cpuSockets': '2', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'gateway': '172.16.0.1', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.2', 'vnet0']}, 'os_net': {'iface': 'os_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.10']}, 'r2o_net': {'iface': 'r2o_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}, 'storage_network': {'iface': u'bond0.3', 'addr': '172.16.1.5', 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000', 'netmask': '255.255.255.255', 'bridged': False, 'interface': u'bond0.3', 'ipv6gateway': '::', 'gateway': ''}}, 'bridges': {'ovirtmgmt': {'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '172.16.0.1', 'ports': ['bond0.2', 'vnet0']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}, 'os_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.10']}, 'r2o_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}}, 'uuid': '44454C4C-3100-104D-8046-B9C04F583153', 'lastClientIface': 'ovirtmgmt', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'cb327caa-ca77-4efe-b7b8-63aba3cda84d', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:85', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth2', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:85', 'speed': 1000, 'mtu': '4000'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'e401a6a5-8773-4971-9213-b3c2bb52f16c', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:83', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:83', 'speed': 1000, 'mtu': '4000'}, 'eth0': {'netmask': '', 'addr': '', 'hwaddr': '00:e0:4c:80:13:02', 'cfg': {'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': '00:e0:4c:80:13:02', 'STP': 'no', 'DEVICE': 'eth0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'speed': 1000, 'mtu': '1500'}}, 'software_revision': '33', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,tm2,ssse3,cx16,xtpr,pdcm,dca,lahf_lm,dts,tpr_shadow,vnmi,flexpriority,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': ['eth1', 'eth2'], 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'MTU': '4000', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'BONDING_OPTS': 'miimon=100 mode=balance-rr', 'DEVICE': 'bond0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '32105', 'cpuSpeed': '1861.936', 'version_name': 'Snow Man', 'vlans': {'bond0.2': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'ovirtmgmt', 'VLAN': 'yes', 'MTU': '4000', 'BOOTPROTO': 'none', 'DEVICE': 'bond0.2', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 2, 'mtu': '4000'}, 'bond0.3': {'netmask': '255.255.255.255', 'iface': 'bond0', 'addr': '172.16.1.5', 'cfg': {'VLAN': 'yes', 'DEFROUTE': 'no', 'MTU': '4000', 'NM_CONTROLLED': 'no', 'DEVICE': 'bond0.3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 3, 'mtu': '4000'}, 'eth0.11': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'r2o_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.11', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 11, 'mtu': '1500'}, 'eth0.10': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'os_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.10', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 10, 'mtu': '1500'}}, 'cpuCores': '8', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '8', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '172.16.0.11'}}
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:55,984::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:55,984::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:55,984::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:55,985::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:55,985::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:55,985::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:55,986::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:55,986::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:55,987::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:55,987::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:55,987::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:14:55,987::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-12::DEBUG::2014-01-28 00:14:56,277::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:14:56,304::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-12::DEBUG::2014-01-28 00:14:58,320::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:14:58,351::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-121::DEBUG::2014-01-28 00:14:58,580::BindingXMLRPC::159::vds::(wrapper) client [127.0.0.1]
>Thread-121::INFO::2014-01-28 00:14:58,581::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='7e0cc075-eee7-4268-946e-33a79dd08748', conList=[{'port': '', 'connection': '172.16.1.5:/HOSTED-ENGINE', 'iqn': '', 'portal': '', 'user': 'kvm', 'protocol_version': '3', 'password': '******', 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}], options=None)
>Thread-121::DEBUG::2014-01-28 00:14:58,588::hsm::2273::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE
>Thread-122::DEBUG::2014-01-28 00:14:58,639::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call vmGetStats with ('2d425c4c-91b1-4564-b276-f27cc581e6a1',) {}
>Thread-122::DEBUG::2014-01-28 00:14:58,640::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-122::DEBUG::2014-01-28 00:14:58,640::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>Thread-122::DEBUG::2014-01-28 00:14:58,640::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-122::DEBUG::2014-01-28 00:14:58,641::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-122::DEBUG::2014-01-28 00:14:58,641::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>Thread-122::DEBUG::2014-01-28 00:14:58,641::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-122::DEBUG::2014-01-28 00:14:58,642::BindingXMLRPC::976::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'root', 'memUsage': '53', 'acpiEnable': 'true', 'guestFQDN': u'engine.melb.example.net', 'pid': '18113', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'/', u'total': '25323499520', u'used': '2592231424', u'fs': u'ext4'}, {u'path': u'/boot', u'total': '507744256', u'used': '55643136', u'fs': u'ext4'}], 'network': {u'vnet0': {'macAddr': '00:16:3e:0a:b7:55', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmType': 'kvm', 'guestName': u'engine.melb.example.net', 'elapsedTime': '610883', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'mem_free': '1962204', u'swap_in': '0', u'pageflt': '441', u'mem_total': '3922728', u'mem_unused': '1427872'}, 'cpuSys': '2.73', 'appsList': [u'ovirt-guest-agent-1.0.8-1.el6', u'kernel-2.6.32-431.el6', u'kernel-2.6.32-431.3.1.el6'], 'guestOs': u'2.6.32-431.3.1.el6.x86_64', 'displayType': 'vnc', 'status': 'Up', 'hash': '2950633987041661405', 'vmId': '2d425c4c-91b1-4564-b276-f27cc581e6a1', 'displayIp': '0', 'displayPort': u'5900', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'truesize': '3920592896', 'apparentsize': '26843545600', 'imageID': 'daa7fa08-1354-4b19-a80f-8694ea63f914'}, 'hdc': {'truesize': '0', 'apparentsize': '0'}}, 'monitorResponse': '0', 'statsAge': '0.26', 'cpuUser': '20.59', 'lastLogin': 1390828479.279238, 'guestIPs': u'172.16.0.10', 'netIfaces': [{u'hw': u'00:16:3e:0a:b7:55', u'name': u'eth0', u'inet': [u'172.16.0.10'], u'inet6': [u'fe80::216:3eff:fe0a:b755']}]}]}
>Thread-124::DEBUG::2014-01-28 00:14:59,127::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {} flowID [2697e1d7]
>clientIFinit::INFO::2014-01-28 00:14:59,186::logUtils::44::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList(options=None)
>clientIFinit::INFO::2014-01-28 00:14:59,187::logUtils::47::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList, Return response: {'poollist': []}
>Thread-124::DEBUG::2014-01-28 00:14:59,201::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-125::DEBUG::2014-01-28 00:14:59,305::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-125::INFO::2014-01-28 00:14:59,306::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-125::ERROR::2014-01-28 00:14:59,307::dispatcher::67::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('5849b030-626e-47cb-ad90-3ce782d831b3',)", 'code': 309}}
>Thread-126::DEBUG::2014-01-28 00:14:59,374::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-126::INFO::2014-01-28 00:14:59,375::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', hostID=1, msdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', masterVersion=1, options=None)
>Thread-126::INFO::2014-01-28 00:14:59,378::sp::683::Storage.StoragePool::(connect) Connect host #1 to the storage pool 5849b030-626e-47cb-ad90-3ce782d831b3 with master domain: ff3a0c94-11dc-446d-b093-5f68ad81520d (ver = 1)
>Thread-126::DEBUG::2014-01-28 00:14:59,380::iscsi::393::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-126::DEBUG::2014-01-28 00:14:59,381::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-126::DEBUG::2014-01-28 00:14:59,429::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-126::DEBUG::2014-01-28 00:14:59,540::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-126::ERROR::2014-01-28 00:14:59,542::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain ff3a0c94-11dc-446d-b093-5f68ad81520d
>Thread-126::ERROR::2014-01-28 00:14:59,542::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain ff3a0c94-11dc-446d-b093-5f68ad81520d
>Thread-126::DEBUG::2014-01-28 00:14:59,545::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name ff3a0c94-11dc-446d-b093-5f68ad81520d' (cwd None)
>Thread-126::DEBUG::2014-01-28 00:14:59,806::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "ff3a0c94-11dc-446d-b093-5f68ad81520d" not found\n  Skipping volume group ff3a0c94-11dc-446d-b093-5f68ad81520d\n'; <rc> = 5
>Thread-126::WARNING::2014-01-28 00:14:59,808::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "ff3a0c94-11dc-446d-b093-5f68ad81520d" not found', '  Skipping volume group ff3a0c94-11dc-446d-b093-5f68ad81520d']
>Thread-126::DEBUG::2014-01-28 00:14:59,833::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d
>Thread-126::DEBUG::2014-01-28 00:15:00,075::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-126::DEBUG::2014-01-28 00:15:00,087::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=VM-DATA', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=NextDC_M1', 'POOL_DOMAINS=ff3a0c94-11dc-446d-b093-5f68ad81520d:Active,436c344c-bc57-441b-b311-c9595c6039e1:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=1', 'POOL_UUID=5849b030-626e-47cb-ad90-3ce782d831b3', 'REMOTE_PATH=172.16.1.5:/VM-DATA', 'ROLE=Master', 'SDUUID=ff3a0c94-11dc-446d-b093-5f68ad81520d', 'TYPE=GLUSTERFS', 'VERSION=3', '_SHA_CKSUM=aed4059bcf1cca521e624929711bc334499d38cd']
>Thread-126::DEBUG::2014-01-28 00:15:00,096::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-126::DEBUG::2014-01-28 00:15:00,098::sp::1483::Storage.StoragePool::(setMasterDomain) Master domain ff3a0c94-11dc-446d-b093-5f68ad81520d verified, version 1
>Thread-126::DEBUG::2014-01-28 00:15:00,107::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=VM-DATA', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=NextDC_M1', 'POOL_DOMAINS=ff3a0c94-11dc-446d-b093-5f68ad81520d:Active,436c344c-bc57-441b-b311-c9595c6039e1:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=1', 'POOL_UUID=5849b030-626e-47cb-ad90-3ce782d831b3', 'REMOTE_PATH=172.16.1.5:/VM-DATA', 'ROLE=Master', 'SDUUID=ff3a0c94-11dc-446d-b093-5f68ad81520d', 'TYPE=GLUSTERFS', 'VERSION=3', '_SHA_CKSUM=aed4059bcf1cca521e624929711bc334499d38cd']
>Thread-126::INFO::2014-01-28 00:15:00,108::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring ff3a0c94-11dc-446d-b093-5f68ad81520d
>Thread-128::DEBUG::2014-01-28 00:15:00,109::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for ff3a0c94-11dc-446d-b093-5f68ad81520d
>Thread-126::DEBUG::2014-01-28 00:15:00,109::sp::1517::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `5849b030-626e-47cb-ad90-3ce782d831b3` started monitoring domain `ff3a0c94-11dc-446d-b093-5f68ad81520d`
>Thread-126::INFO::2014-01-28 00:15:00,113::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 436c344c-bc57-441b-b311-c9595c6039e1
>Thread-129::DEBUG::2014-01-28 00:15:00,118::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 436c344c-bc57-441b-b311-c9595c6039e1
>Thread-126::DEBUG::2014-01-28 00:15:00,118::sp::1517::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `5849b030-626e-47cb-ad90-3ce782d831b3` started monitoring domain `436c344c-bc57-441b-b311-c9595c6039e1`
>Thread-126::DEBUG::2014-01-28 00:15:00,205::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=VM-DATA', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=NextDC_M1', 'POOL_DOMAINS=ff3a0c94-11dc-446d-b093-5f68ad81520d:Active,436c344c-bc57-441b-b311-c9595c6039e1:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=1', 'POOL_UUID=5849b030-626e-47cb-ad90-3ce782d831b3', 'REMOTE_PATH=172.16.1.5:/VM-DATA', 'ROLE=Master', 'SDUUID=ff3a0c94-11dc-446d-b093-5f68ad81520d', 'TYPE=GLUSTERFS', 'VERSION=3', '_SHA_CKSUM=aed4059bcf1cca521e624929711bc334499d38cd']
>Thread-129::DEBUG::2014-01-28 00:15:00,264::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1
>Thread-133::DEBUG::2014-01-28 00:15:00,265::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10] flowID [2200647a]
>Thread-126::DEBUG::2014-01-28 00:15:00,308::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 436c344c-bc57-441b-b311-c9595c6039e1' (cwd None)
>Thread-133::INFO::2014-01-28 00:15:00,338::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=7, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'port': '', 'connection': '172.16.1.5:/VM-DATA', 'iqn': '', 'portal': '', 'user': '', 'vfs_type': 'glusterfs', 'password': '******', 'id': '594fb154-104f-4248-b87b-4d01cf9b5325'}], options=None)
>Thread-12::DEBUG::2014-01-28 00:15:00,375::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-133::DEBUG::2014-01-28 00:15:00,465::hsm::2285::Storage.HSM::(__prefetchDomains) glusterDomPath: glusterSD/*
>Thread-12::DEBUG::2014-01-28 00:15:00,498::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-133::DEBUG::2014-01-28 00:15:00,503::hsm::2297::Storage.HSM::(__prefetchDomains) Found SD uuids: ('ff3a0c94-11dc-446d-b093-5f68ad81520d',)
>Thread-133::DEBUG::2014-01-28 00:15:00,511::hsm::2353::Storage.HSM::(connectStorageServer) knownSDs: {562f0160-7b80-42e2-b248-5754455c40fc: storage.nfsSD.findDomain, 436c344c-bc57-441b-b311-c9595c6039e1: storage.nfsSD.findDomain, ff3a0c94-11dc-446d-b093-5f68ad81520d: storage.glusterSD.findDomain}
>Thread-133::INFO::2014-01-28 00:15:00,514::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '594fb154-104f-4248-b87b-4d01cf9b5325'}]}
>Thread-129::DEBUG::2014-01-28 00:15:00,561::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-128::DEBUG::2014-01-28 00:15:00,569::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-126::DEBUG::2014-01-28 00:15:00,625::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "436c344c-bc57-441b-b311-c9595c6039e1" not found\n  Skipping volume group 436c344c-bc57-441b-b311-c9595c6039e1\n'; <rc> = 5
>Thread-129::DEBUG::2014-01-28 00:15:00,638::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Iso', 'DESCRIPTION=ISOS', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=5849b030-626e-47cb-ad90-3ce782d831b3', 'REMOTE_PATH=engine.melb.example.net:/var/lib/exports/iso', 'ROLE=Regular', 'SDUUID=436c344c-bc57-441b-b311-c9595c6039e1', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=565367baaccc061ba62498d9ef0510a4acd42623']
>Thread-126::WARNING::2014-01-28 00:15:00,664::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "436c344c-bc57-441b-b311-c9595c6039e1" not found', '  Skipping volume group 436c344c-bc57-441b-b311-c9595c6039e1']
>Thread-129::DEBUG::2014-01-28 00:15:00,701::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-128::DEBUG::2014-01-28 00:15:00,760::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n504 bytes (504 B) copied, 0.00105706 s, 477 kB/s\n'; <rc> = 0
>Thread-128::DEBUG::2014-01-28 00:15:00,790::iscsi::393::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-128::DEBUG::2014-01-28 00:15:00,802::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-128::DEBUG::2014-01-28 00:15:00,857::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:15:00,890::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:15:00,926::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n364 bytes (364 B) copied, 0.00122219 s, 298 kB/s\n'; <rc> = 0
>Thread-129::DEBUG::2014-01-28 00:15:00,928::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 436c344c-bc57-441b-b311-c9595c6039e1 changed its status to Valid
>Thread-138::DEBUG::2014-01-28 00:15:00,930::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
>Thread-138::DEBUG::2014-01-28 00:15:00,931::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
>Thread-138::DEBUG::2014-01-28 00:15:00,933::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
>Thread-139::INFO::2014-01-28 00:15:00,946::clientIF::126::vds::(contEIOVms) vmContainerLock acquired
>Thread-128::DEBUG::2014-01-28 00:15:00,952::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-128::DEBUG::2014-01-28 00:15:00,965::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d
>Thread-128::DEBUG::2014-01-28 00:15:00,967::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-128::DEBUG::2014-01-28 00:15:00,977::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=VM-DATA', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=NextDC_M1', 'POOL_DOMAINS=ff3a0c94-11dc-446d-b093-5f68ad81520d:Active,436c344c-bc57-441b-b311-c9595c6039e1:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=1', 'POOL_UUID=5849b030-626e-47cb-ad90-3ce782d831b3', 'REMOTE_PATH=172.16.1.5:/VM-DATA', 'ROLE=Master', 'SDUUID=ff3a0c94-11dc-446d-b093-5f68ad81520d', 'TYPE=GLUSTERFS', 'VERSION=3', '_SHA_CKSUM=aed4059bcf1cca521e624929711bc334499d38cd']
>Thread-128::DEBUG::2014-01-28 00:15:00,983::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-128::INFO::2014-01-28 00:15:00,983::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace ff3a0c94-11dc-446d-b093-5f68ad81520d_imageNS already registered
>Thread-128::INFO::2014-01-28 00:15:00,983::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace ff3a0c94-11dc-446d-b093-5f68ad81520d_volumeNS already registered
>Thread-128::DEBUG::2014-01-28 00:15:00,991::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain ff3a0c94-11dc-446d-b093-5f68ad81520d changed its status to Valid
>Thread-141::DEBUG::2014-01-28 00:15:00,992::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
>Thread-141::DEBUG::2014-01-28 00:15:00,996::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:01,000::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-141::DEBUG::2014-01-28 00:15:01,000::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:01,002::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>Thread-142::INFO::2014-01-28 00:15:01,005::clientIF::126::vds::(contEIOVms) vmContainerLock acquired
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:01,007::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:01,008::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:01,009::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:01,009::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:01,011::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:01,011::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:01,011::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:01,012::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:01,012::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:01,012::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-143::INFO::2014-01-28 00:15:01,732::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-143::INFO::2014-01-28 00:15:01,732::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000582437', 'lastCheck': '7.4', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00122219', 'lastCheck': '0.8', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00105706', 'lastCheck': '0.7', 'valid': True}}
>Thread-12::DEBUG::2014-01-28 00:15:02,862::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:15:02,888::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-121::DEBUG::2014-01-28 00:15:03,904::hsm::2297::Storage.HSM::(__prefetchDomains) Found SD uuids: ('562f0160-7b80-42e2-b248-5754455c40fc',)
>Thread-121::DEBUG::2014-01-28 00:15:03,905::hsm::2353::Storage.HSM::(connectStorageServer) knownSDs: {562f0160-7b80-42e2-b248-5754455c40fc: storage.nfsSD.findDomain, 436c344c-bc57-441b-b311-c9595c6039e1: storage.nfsSD.findDomain, ff3a0c94-11dc-446d-b093-5f68ad81520d: storage.glusterSD.findDomain}
>Thread-121::INFO::2014-01-28 00:15:03,906::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}]}
>Thread-126::INFO::2014-01-28 00:15:03,908::sp::1160::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1 to /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/436c344c-bc57-441b-b311-c9595c6039e1
>Thread-126::INFO::2014-01-28 00:15:03,913::sp::1160::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d to /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/ff3a0c94-11dc-446d-b093-5f68ad81520d
>Thread-126::INFO::2014-01-28 00:15:03,914::sp::1160::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d to /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/mastersd
>Thread-126::INFO::2014-01-28 00:15:03,917::logUtils::47::dispatcher::(wrapper) Run and protect: connectStoragePool, Return response: True
>Thread-144::DEBUG::2014-01-28 00:15:03,954::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-144::INFO::2014-01-28 00:15:03,955::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-144::DEBUG::2014-01-28 00:15:03,977::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=VM-DATA', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=NextDC_M1', 'POOL_DOMAINS=ff3a0c94-11dc-446d-b093-5f68ad81520d:Active,436c344c-bc57-441b-b311-c9595c6039e1:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=1', 'POOL_UUID=5849b030-626e-47cb-ad90-3ce782d831b3', 'REMOTE_PATH=172.16.1.5:/VM-DATA', 'ROLE=Master', 'SDUUID=ff3a0c94-11dc-446d-b093-5f68ad81520d', 'TYPE=GLUSTERFS', 'VERSION=3', '_SHA_CKSUM=aed4059bcf1cca521e624929711bc334499d38cd']
>Thread-144::INFO::2014-01-28 00:15:03,977::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 1}}
>Thread-145::DEBUG::2014-01-28 00:15:03,985::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-145::INFO::2014-01-28 00:15:03,986::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-145::DEBUG::2014-01-28 00:15:03,997::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=VM-DATA', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=NextDC_M1', 'POOL_DOMAINS=ff3a0c94-11dc-446d-b093-5f68ad81520d:Active,436c344c-bc57-441b-b311-c9595c6039e1:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=1', 'POOL_UUID=5849b030-626e-47cb-ad90-3ce782d831b3', 'REMOTE_PATH=172.16.1.5:/VM-DATA', 'ROLE=Master', 'SDUUID=ff3a0c94-11dc-446d-b093-5f68ad81520d', 'TYPE=GLUSTERFS', 'VERSION=3', '_SHA_CKSUM=aed4059bcf1cca521e624929711bc334499d38cd']
>Thread-145::INFO::2014-01-28 00:15:03,998::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 1}}
>Thread-146::DEBUG::2014-01-28 00:15:04,126::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-146::DEBUG::2014-01-28 00:15:04,128::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>Thread-146::DEBUG::2014-01-28 00:15:04,129::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-147::DEBUG::2014-01-28 00:15:04,129::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-146::DEBUG::2014-01-28 00:15:04,130::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-146::DEBUG::2014-01-28 00:15:04,131::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>Thread-147::INFO::2014-01-28 00:15:04,132::logUtils::44::dispatcher::(wrapper) Run and protect: spmStart(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', prevID=1, prevLVER='1', maxHostID=250, domVersion='3', options=None)
>Thread-146::DEBUG::2014-01-28 00:15:04,132::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-147::INFO::2014-01-28 00:15:04,139::logUtils::47::dispatcher::(wrapper) Run and protect: spmStart, Return response: None
>4155532d-c8c0-4274-8551-25b0193797b3::DEBUG::2014-01-28 00:15:04,155::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=VM-DATA', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=NextDC_M1', 'POOL_DOMAINS=ff3a0c94-11dc-446d-b093-5f68ad81520d:Active,436c344c-bc57-441b-b311-c9595c6039e1:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=1', 'POOL_UUID=5849b030-626e-47cb-ad90-3ce782d831b3', 'REMOTE_PATH=172.16.1.5:/VM-DATA', 'ROLE=Master', 'SDUUID=ff3a0c94-11dc-446d-b093-5f68ad81520d', 'TYPE=GLUSTERFS', 'VERSION=3', '_SHA_CKSUM=aed4059bcf1cca521e624929711bc334499d38cd']
>4155532d-c8c0-4274-8551-25b0193797b3::DEBUG::2014-01-28 00:15:04,186::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=VM-DATA', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=NextDC_M1', 'POOL_DOMAINS=ff3a0c94-11dc-446d-b093-5f68ad81520d:Active,436c344c-bc57-441b-b311-c9595c6039e1:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=1', 'POOL_UUID=5849b030-626e-47cb-ad90-3ce782d831b3', 'REMOTE_PATH=172.16.1.5:/VM-DATA', 'ROLE=Master', 'SDUUID=ff3a0c94-11dc-446d-b093-5f68ad81520d', 'TYPE=GLUSTERFS', 'VERSION=3', '_SHA_CKSUM=aed4059bcf1cca521e624929711bc334499d38cd']
>clientIFinit::INFO::2014-01-28 00:15:04,193::logUtils::44::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList(options=None)
>clientIFinit::INFO::2014-01-28 00:15:04,194::logUtils::47::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList, Return response: {'poollist': ['5849b030-626e-47cb-ad90-3ce782d831b3']}
>clientIFinit::WARNING::2014-01-28 00:15:04,195::vm::2053::vm.Vm::(buildConfDevices) vmId=`48ad952b-4457-4ae1-bf85-93f869382298`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel0', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'1'}}' found
>clientIFinit::WARNING::2014-01-28 00:15:04,196::vm::2053::vm.Vm::(buildConfDevices) vmId=`48ad952b-4457-4ae1-bf85-93f869382298`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel1', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'2'}}' found
>clientIFinit::WARNING::2014-01-28 00:15:04,197::vm::2053::vm.Vm::(buildConfDevices) vmId=`48ad952b-4457-4ae1-bf85-93f869382298`::Unknown type found, device: '{'device': u'spicevmc', 'alias': u'channel2', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'3'}}' found
>clientIFinit::INFO::2014-01-28 00:15:04,198::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='51206945-475a-48b1-96e1-689ca96312a5', volUUID='1732793c-bf4f-4f8d-a5b9-4f41e99154ba', options=None)
>clientIFinit::INFO::2014-01-28 00:15:04,203::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '9283706880', 'apparentsize': '9283829760'}
>clientIFinit::INFO::2014-01-28 00:15:04,204::clientIF::320::vds::(prepareVolumePath) prepared volume path: 
>clientIFinit::INFO::2014-01-28 00:15:04,204::logUtils::44::dispatcher::(wrapper) Run and protect: prepareImage(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='51206945-475a-48b1-96e1-689ca96312a5', leafUUID='1732793c-bf4f-4f8d-a5b9-4f41e99154ba')
>clientIFinit::WARNING::2014-01-28 00:15:04,284::fileUtils::167::Storage.fileUtils::(createdir) Dir /var/run/vdsm/storage/ff3a0c94-11dc-446d-b093-5f68ad81520d already exists
>Thread-148::DEBUG::2014-01-28 00:15:04,285::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.11]
>Thread-148::INFO::2014-01-28 00:15:04,286::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-148::INFO::2014-01-28 00:15:04,286::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000582437', 'lastCheck': '9.9', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00122219', 'lastCheck': '3.4', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00105706', 'lastCheck': '3.3', 'valid': True}}
>clientIFinit::DEBUG::2014-01-28 00:15:04,302::fileSD::446::Storage.StorageDomain::(createImageLinks) img run dir already exists: /var/run/vdsm/storage/ff3a0c94-11dc-446d-b093-5f68ad81520d/51206945-475a-48b1-96e1-689ca96312a5
>clientIFinit::DEBUG::2014-01-28 00:15:04,304::fileVolume::529::Storage.Volume::(validateVolumePath) validate path for 1732793c-bf4f-4f8d-a5b9-4f41e99154ba
>clientIFinit::INFO::2014-01-28 00:15:04,305::logUtils::47::dispatcher::(wrapper) Run and protect: prepareImage, Return response: {'info': {'domainID': 'ff3a0c94-11dc-446d-b093-5f68ad81520d', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/51206945-475a-48b1-96e1-689ca96312a5/1732793c-bf4f-4f8d-a5b9-4f41e99154ba', 'volumeID': '1732793c-bf4f-4f8d-a5b9-4f41e99154ba', 'leasePath': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/51206945-475a-48b1-96e1-689ca96312a5/1732793c-bf4f-4f8d-a5b9-4f41e99154ba.lease', 'imageID': '51206945-475a-48b1-96e1-689ca96312a5'}, 'path': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/51206945-475a-48b1-96e1-689ca96312a5/1732793c-bf4f-4f8d-a5b9-4f41e99154ba', 'imgVolumesInfo': [{'domainID': 'ff3a0c94-11dc-446d-b093-5f68ad81520d', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/51206945-475a-48b1-96e1-689ca96312a5/1732793c-bf4f-4f8d-a5b9-4f41e99154ba', 'volumeID': '1732793c-bf4f-4f8d-a5b9-4f41e99154ba', 'leasePath': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/51206945-475a-48b1-96e1-689ca96312a5/1732793c-bf4f-4f8d-a5b9-4f41e99154ba.lease', 'imageID': '51206945-475a-48b1-96e1-689ca96312a5'}]}
>clientIFinit::INFO::2014-01-28 00:15:04,306::clientIF::320::vds::(prepareVolumePath) prepared volume path: /rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/51206945-475a-48b1-96e1-689ca96312a5/1732793c-bf4f-4f8d-a5b9-4f41e99154ba
>clientIFinit::WARNING::2014-01-28 00:15:04,307::vm::2053::vm.Vm::(buildConfDevices) vmId=`b3cde33c-f4c5-4b42-8e9d-820b80abc2fb`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel0', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'1'}}' found
>clientIFinit::WARNING::2014-01-28 00:15:04,308::vm::2053::vm.Vm::(buildConfDevices) vmId=`b3cde33c-f4c5-4b42-8e9d-820b80abc2fb`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel1', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'2'}}' found
>clientIFinit::WARNING::2014-01-28 00:15:04,309::vm::2053::vm.Vm::(buildConfDevices) vmId=`b3cde33c-f4c5-4b42-8e9d-820b80abc2fb`::Unknown type found, device: '{'device': u'spicevmc', 'alias': u'channel2', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'3'}}' found
>clientIFinit::INFO::2014-01-28 00:15:04,309::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='50cabd29-7058-4510-b61f-db21862d99df', volUUID='96d9eaab-9dff-45f6-8fe7-b4fd992ee23e', options=None)
>clientIFinit::INFO::2014-01-28 00:15:04,313::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2946445312', 'apparentsize': '2946564096'}
>clientIFinit::INFO::2014-01-28 00:15:04,314::clientIF::320::vds::(prepareVolumePath) prepared volume path: 
>clientIFinit::INFO::2014-01-28 00:15:04,314::logUtils::44::dispatcher::(wrapper) Run and protect: prepareImage(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='50cabd29-7058-4510-b61f-db21862d99df', leafUUID='96d9eaab-9dff-45f6-8fe7-b4fd992ee23e')
>clientIFinit::WARNING::2014-01-28 00:15:04,358::fileUtils::167::Storage.fileUtils::(createdir) Dir /var/run/vdsm/storage/ff3a0c94-11dc-446d-b093-5f68ad81520d already exists
>clientIFinit::DEBUG::2014-01-28 00:15:04,358::fileSD::446::Storage.StorageDomain::(createImageLinks) img run dir already exists: /var/run/vdsm/storage/ff3a0c94-11dc-446d-b093-5f68ad81520d/50cabd29-7058-4510-b61f-db21862d99df
>clientIFinit::DEBUG::2014-01-28 00:15:04,361::fileVolume::529::Storage.Volume::(validateVolumePath) validate path for 96d9eaab-9dff-45f6-8fe7-b4fd992ee23e
>Thread-149::DEBUG::2014-01-28 00:15:04,362::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {}
>clientIFinit::INFO::2014-01-28 00:15:04,364::logUtils::47::dispatcher::(wrapper) Run and protect: prepareImage, Return response: {'info': {'domainID': 'ff3a0c94-11dc-446d-b093-5f68ad81520d', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/50cabd29-7058-4510-b61f-db21862d99df/96d9eaab-9dff-45f6-8fe7-b4fd992ee23e', 'volumeID': '96d9eaab-9dff-45f6-8fe7-b4fd992ee23e', 'leasePath': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/50cabd29-7058-4510-b61f-db21862d99df/96d9eaab-9dff-45f6-8fe7-b4fd992ee23e.lease', 'imageID': '50cabd29-7058-4510-b61f-db21862d99df'}, 'path': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/50cabd29-7058-4510-b61f-db21862d99df/96d9eaab-9dff-45f6-8fe7-b4fd992ee23e', 'imgVolumesInfo': [{'domainID': 'ff3a0c94-11dc-446d-b093-5f68ad81520d', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/50cabd29-7058-4510-b61f-db21862d99df/96d9eaab-9dff-45f6-8fe7-b4fd992ee23e', 'volumeID': '96d9eaab-9dff-45f6-8fe7-b4fd992ee23e', 'leasePath': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/50cabd29-7058-4510-b61f-db21862d99df/96d9eaab-9dff-45f6-8fe7-b4fd992ee23e.lease', 'imageID': '50cabd29-7058-4510-b61f-db21862d99df'}]}
>clientIFinit::INFO::2014-01-28 00:15:04,365::clientIF::320::vds::(prepareVolumePath) prepared volume path: /rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/50cabd29-7058-4510-b61f-db21862d99df/96d9eaab-9dff-45f6-8fe7-b4fd992ee23e
>clientIFinit::WARNING::2014-01-28 00:15:04,365::vm::2053::vm.Vm::(buildConfDevices) vmId=`9c9befe0-9e17-462e-9bce-27ddd684a6d3`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel0', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'1'}}' found
>clientIFinit::WARNING::2014-01-28 00:15:04,366::vm::2053::vm.Vm::(buildConfDevices) vmId=`9c9befe0-9e17-462e-9bce-27ddd684a6d3`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel1', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'2'}}' found
>clientIFinit::WARNING::2014-01-28 00:15:04,366::vm::2053::vm.Vm::(buildConfDevices) vmId=`9c9befe0-9e17-462e-9bce-27ddd684a6d3`::Unknown type found, device: '{'device': u'spicevmc', 'alias': u'channel2', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'3'}}' found
>clientIFinit::INFO::2014-01-28 00:15:04,367::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='5af8ecbb-2530-48c9-8c74-704d51d34f70', volUUID='8d040005-63a7-43f8-9f67-fb8a4ac94193', options=None)
>clientIFinit::INFO::2014-01-28 00:15:04,369::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '7572697088', 'apparentsize': '7572815872'}
>clientIFinit::INFO::2014-01-28 00:15:04,369::clientIF::320::vds::(prepareVolumePath) prepared volume path: 
>clientIFinit::INFO::2014-01-28 00:15:04,370::logUtils::44::dispatcher::(wrapper) Run and protect: prepareImage(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='5af8ecbb-2530-48c9-8c74-704d51d34f70', leafUUID='8d040005-63a7-43f8-9f67-fb8a4ac94193')
>Thread-17::DEBUG::2014-01-28 00:15:04,372::iscsi::393::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-17::DEBUG::2014-01-28 00:15:04,373::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>clientIFinit::WARNING::2014-01-28 00:15:04,416::fileUtils::167::Storage.fileUtils::(createdir) Dir /var/run/vdsm/storage/ff3a0c94-11dc-446d-b093-5f68ad81520d already exists
>clientIFinit::DEBUG::2014-01-28 00:15:04,417::fileSD::446::Storage.StorageDomain::(createImageLinks) img run dir already exists: /var/run/vdsm/storage/ff3a0c94-11dc-446d-b093-5f68ad81520d/5af8ecbb-2530-48c9-8c74-704d51d34f70
>clientIFinit::DEBUG::2014-01-28 00:15:04,418::fileVolume::529::Storage.Volume::(validateVolumePath) validate path for 8d040005-63a7-43f8-9f67-fb8a4ac94193
>clientIFinit::INFO::2014-01-28 00:15:04,419::logUtils::47::dispatcher::(wrapper) Run and protect: prepareImage, Return response: {'info': {'domainID': 'ff3a0c94-11dc-446d-b093-5f68ad81520d', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/5af8ecbb-2530-48c9-8c74-704d51d34f70/8d040005-63a7-43f8-9f67-fb8a4ac94193', 'volumeID': '8d040005-63a7-43f8-9f67-fb8a4ac94193', 'leasePath': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/5af8ecbb-2530-48c9-8c74-704d51d34f70/8d040005-63a7-43f8-9f67-fb8a4ac94193.lease', 'imageID': '5af8ecbb-2530-48c9-8c74-704d51d34f70'}, 'path': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/5af8ecbb-2530-48c9-8c74-704d51d34f70/8d040005-63a7-43f8-9f67-fb8a4ac94193', 'imgVolumesInfo': [{'domainID': 'ff3a0c94-11dc-446d-b093-5f68ad81520d', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/5af8ecbb-2530-48c9-8c74-704d51d34f70/8d040005-63a7-43f8-9f67-fb8a4ac94193', 'volumeID': '8d040005-63a7-43f8-9f67-fb8a4ac94193', 'leasePath': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/5af8ecbb-2530-48c9-8c74-704d51d34f70/8d040005-63a7-43f8-9f67-fb8a4ac94193.lease', 'imageID': '5af8ecbb-2530-48c9-8c74-704d51d34f70'}]}
>clientIFinit::INFO::2014-01-28 00:15:04,421::clientIF::320::vds::(prepareVolumePath) prepared volume path: /rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/5af8ecbb-2530-48c9-8c74-704d51d34f70/8d040005-63a7-43f8-9f67-fb8a4ac94193
>clientIFinit::WARNING::2014-01-28 00:15:04,422::vm::2053::vm.Vm::(buildConfDevices) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel0', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'1'}}' found
>clientIFinit::WARNING::2014-01-28 00:15:04,422::vm::2053::vm.Vm::(buildConfDevices) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel1', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'2'}}' found
>Thread-17::DEBUG::2014-01-28 00:15:04,429::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>clientIFinit::INFO::2014-01-28 00:15:04,432::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='562f0160-7b80-42e2-b248-5754455c40fc', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='daa7fa08-1354-4b19-a80f-8694ea63f914', volUUID='8d9c27b1-8b26-4a44-8be2-79f024a56e0f', options=None)
>Thread-149::DEBUG::2014-01-28 00:15:04,455::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-17::DEBUG::2014-01-28 00:15:04,536::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-17::DEBUG::2014-01-28 00:15:04,561::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc
>Thread-17::DEBUG::2014-01-28 00:15:04,564::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-17::DEBUG::2014-01-28 00:15:04,575::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=562f0160-7b80-42e2-b248-5754455c40fc:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=7e0cc075-eee7-4268-946e-33a79dd08748', 'REMOTE_PATH=172.16.1.5:/HOSTED-ENGINE', 'ROLE=Master', 'SDUUID=562f0160-7b80-42e2-b248-5754455c40fc', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=14825cb0b944c5c2ae9ed025c82e06730a2d91c4']
>Thread-17::DEBUG::2014-01-28 00:15:04,577::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-17::INFO::2014-01-28 00:15:04,578::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_imageNS already registered
>Thread-17::INFO::2014-01-28 00:15:04,578::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_volumeNS already registered
>clientIFinit::INFO::2014-01-28 00:15:04,601::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '3920601088', 'apparentsize': '26843545600'}
>Thread-153::INFO::2014-01-28 00:15:04,613::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-153::INFO::2014-01-28 00:15:04,630::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000582437', 'lastCheck': '10.3', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00122219', 'lastCheck': '3.7', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00105706', 'lastCheck': '3.6', 'valid': True}}
>clientIFinit::INFO::2014-01-28 00:15:04,643::clientIF::320::vds::(prepareVolumePath) prepared volume path: /tmp/CentOS-6.5-x86_64-minimal.iso
>clientIFinit::INFO::2014-01-28 00:15:04,663::logUtils::44::dispatcher::(wrapper) Run and protect: prepareImage(sdUUID='562f0160-7b80-42e2-b248-5754455c40fc', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='daa7fa08-1354-4b19-a80f-8694ea63f914', leafUUID='8d9c27b1-8b26-4a44-8be2-79f024a56e0f')
>Thread-17::DEBUG::2014-01-28 00:15:04,729::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc/dom_md/metadata bs=4096 count=1' (cwd None)
>clientIFinit::WARNING::2014-01-28 00:15:04,774::fileUtils::167::Storage.fileUtils::(createdir) Dir /var/run/vdsm/storage/562f0160-7b80-42e2-b248-5754455c40fc already exists
>clientIFinit::DEBUG::2014-01-28 00:15:04,785::fileSD::446::Storage.StorageDomain::(createImageLinks) img run dir already exists: /var/run/vdsm/storage/562f0160-7b80-42e2-b248-5754455c40fc/daa7fa08-1354-4b19-a80f-8694ea63f914
>Thread-17::DEBUG::2014-01-28 00:15:04,820::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n478 bytes (478 B) copied, 0.000753818 s, 634 kB/s\n'; <rc> = 0
>clientIFinit::DEBUG::2014-01-28 00:15:04,822::fileVolume::529::Storage.Volume::(validateVolumePath) validate path for 8d9c27b1-8b26-4a44-8be2-79f024a56e0f
>clientIFinit::INFO::2014-01-28 00:15:04,860::logUtils::47::dispatcher::(wrapper) Run and protect: prepareImage, Return response: {'info': {'domainID': '562f0160-7b80-42e2-b248-5754455c40fc', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc/images/daa7fa08-1354-4b19-a80f-8694ea63f914/8d9c27b1-8b26-4a44-8be2-79f024a56e0f', 'volumeID': '8d9c27b1-8b26-4a44-8be2-79f024a56e0f', 'leasePath': '/rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc/images/daa7fa08-1354-4b19-a80f-8694ea63f914/8d9c27b1-8b26-4a44-8be2-79f024a56e0f.lease', 'imageID': 'daa7fa08-1354-4b19-a80f-8694ea63f914'}, 'path': '/var/run/vdsm/storage/562f0160-7b80-42e2-b248-5754455c40fc/daa7fa08-1354-4b19-a80f-8694ea63f914/8d9c27b1-8b26-4a44-8be2-79f024a56e0f', 'imgVolumesInfo': [{'domainID': '562f0160-7b80-42e2-b248-5754455c40fc', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc/images/daa7fa08-1354-4b19-a80f-8694ea63f914/8d9c27b1-8b26-4a44-8be2-79f024a56e0f', 'volumeID': '8d9c27b1-8b26-4a44-8be2-79f024a56e0f', 'leasePath': '/rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc/images/daa7fa08-1354-4b19-a80f-8694ea63f914/8d9c27b1-8b26-4a44-8be2-79f024a56e0f.lease', 'imageID': 'daa7fa08-1354-4b19-a80f-8694ea63f914'}]}
>clientIFinit::INFO::2014-01-28 00:15:04,872::clientIF::320::vds::(prepareVolumePath) prepared volume path: /var/run/vdsm/storage/562f0160-7b80-42e2-b248-5754455c40fc/daa7fa08-1354-4b19-a80f-8694ea63f914/8d9c27b1-8b26-4a44-8be2-79f024a56e0f
>clientIFinit::INFO::2014-01-28 00:15:04,873::clientIF::320::vds::(prepareVolumePath) prepared volume path: 
>clientIFinit::WARNING::2014-01-28 00:15:04,874::vm::2053::vm.Vm::(buildConfDevices) vmId=`bbbd4efa-022d-44d3-8274-a0506bda2d5e`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel0', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'1'}}' found
>clientIFinit::WARNING::2014-01-28 00:15:04,881::vm::2053::vm.Vm::(buildConfDevices) vmId=`bbbd4efa-022d-44d3-8274-a0506bda2d5e`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel1', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'2'}}' found
>clientIFinit::WARNING::2014-01-28 00:15:04,881::vm::2053::vm.Vm::(buildConfDevices) vmId=`bbbd4efa-022d-44d3-8274-a0506bda2d5e`::Unknown type found, device: '{'device': u'spicevmc', 'alias': u'channel2', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'3'}}' found
>clientIFinit::INFO::2014-01-28 00:15:04,882::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='91386b87-5fe0-4368-9e87-a28896024015', volUUID='b0db0737-8168-4502-a236-1c3162f98f6a', options=None)
>clientIFinit::INFO::2014-01-28 00:15:04,885::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '5274959872', 'apparentsize': '26843545600'}
>clientIFinit::INFO::2014-01-28 00:15:04,886::clientIF::320::vds::(prepareVolumePath) prepared volume path: 
>clientIFinit::INFO::2014-01-28 00:15:04,887::logUtils::44::dispatcher::(wrapper) Run and protect: prepareImage(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='91386b87-5fe0-4368-9e87-a28896024015', leafUUID='b0db0737-8168-4502-a236-1c3162f98f6a')
>Thread-12::DEBUG::2014-01-28 00:15:04,904::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:15:04,929::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>clientIFinit::WARNING::2014-01-28 00:15:04,949::fileUtils::167::Storage.fileUtils::(createdir) Dir /var/run/vdsm/storage/ff3a0c94-11dc-446d-b093-5f68ad81520d already exists
>clientIFinit::DEBUG::2014-01-28 00:15:04,949::fileSD::446::Storage.StorageDomain::(createImageLinks) img run dir already exists: /var/run/vdsm/storage/ff3a0c94-11dc-446d-b093-5f68ad81520d/91386b87-5fe0-4368-9e87-a28896024015
>clientIFinit::DEBUG::2014-01-28 00:15:04,951::fileVolume::529::Storage.Volume::(validateVolumePath) validate path for b0db0737-8168-4502-a236-1c3162f98f6a
>clientIFinit::INFO::2014-01-28 00:15:04,952::logUtils::47::dispatcher::(wrapper) Run and protect: prepareImage, Return response: {'info': {'domainID': 'ff3a0c94-11dc-446d-b093-5f68ad81520d', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/91386b87-5fe0-4368-9e87-a28896024015/b0db0737-8168-4502-a236-1c3162f98f6a', 'volumeID': 'b0db0737-8168-4502-a236-1c3162f98f6a', 'leasePath': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/91386b87-5fe0-4368-9e87-a28896024015/b0db0737-8168-4502-a236-1c3162f98f6a.lease', 'imageID': '91386b87-5fe0-4368-9e87-a28896024015'}, 'path': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/91386b87-5fe0-4368-9e87-a28896024015/b0db0737-8168-4502-a236-1c3162f98f6a', 'imgVolumesInfo': [{'domainID': 'ff3a0c94-11dc-446d-b093-5f68ad81520d', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/91386b87-5fe0-4368-9e87-a28896024015/b0db0737-8168-4502-a236-1c3162f98f6a', 'volumeID': 'b0db0737-8168-4502-a236-1c3162f98f6a', 'leasePath': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/91386b87-5fe0-4368-9e87-a28896024015/b0db0737-8168-4502-a236-1c3162f98f6a.lease', 'imageID': '91386b87-5fe0-4368-9e87-a28896024015'}]}
>clientIFinit::INFO::2014-01-28 00:15:04,954::clientIF::320::vds::(prepareVolumePath) prepared volume path: /rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/91386b87-5fe0-4368-9e87-a28896024015/b0db0737-8168-4502-a236-1c3162f98f6a
>clientIFinit::WARNING::2014-01-28 00:15:04,955::vm::2053::vm.Vm::(buildConfDevices) vmId=`546b9733-3453-4210-90fd-2b79cf08d53b`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel0', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'1'}}' found
>clientIFinit::WARNING::2014-01-28 00:15:04,955::vm::2053::vm.Vm::(buildConfDevices) vmId=`546b9733-3453-4210-90fd-2b79cf08d53b`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel1', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'2'}}' found
>clientIFinit::WARNING::2014-01-28 00:15:04,955::vm::2053::vm.Vm::(buildConfDevices) vmId=`546b9733-3453-4210-90fd-2b79cf08d53b`::Unknown type found, device: '{'device': u'spicevmc', 'alias': u'channel2', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'3'}}' found
>clientIFinit::INFO::2014-01-28 00:15:04,956::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='00d2f5e6-0263-4ac7-aa12-98ea66f02d00', volUUID='98d6fa79-aecc-429e-8624-4b6f706bdf4f', options=None)
>clientIFinit::INFO::2014-01-28 00:15:04,959::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2936643584', 'apparentsize': '26843545600'}
>clientIFinit::INFO::2014-01-28 00:15:04,960::clientIF::320::vds::(prepareVolumePath) prepared volume path: 
>clientIFinit::INFO::2014-01-28 00:15:04,961::logUtils::44::dispatcher::(wrapper) Run and protect: prepareImage(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='00d2f5e6-0263-4ac7-aa12-98ea66f02d00', leafUUID='98d6fa79-aecc-429e-8624-4b6f706bdf4f')
>clientIFinit::WARNING::2014-01-28 00:15:05,018::fileUtils::167::Storage.fileUtils::(createdir) Dir /var/run/vdsm/storage/ff3a0c94-11dc-446d-b093-5f68ad81520d already exists
>clientIFinit::DEBUG::2014-01-28 00:15:05,019::fileSD::446::Storage.StorageDomain::(createImageLinks) img run dir already exists: /var/run/vdsm/storage/ff3a0c94-11dc-446d-b093-5f68ad81520d/00d2f5e6-0263-4ac7-aa12-98ea66f02d00
>clientIFinit::DEBUG::2014-01-28 00:15:05,020::fileVolume::529::Storage.Volume::(validateVolumePath) validate path for 98d6fa79-aecc-429e-8624-4b6f706bdf4f
>clientIFinit::INFO::2014-01-28 00:15:05,021::logUtils::47::dispatcher::(wrapper) Run and protect: prepareImage, Return response: {'info': {'domainID': 'ff3a0c94-11dc-446d-b093-5f68ad81520d', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/00d2f5e6-0263-4ac7-aa12-98ea66f02d00/98d6fa79-aecc-429e-8624-4b6f706bdf4f', 'volumeID': '98d6fa79-aecc-429e-8624-4b6f706bdf4f', 'leasePath': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/00d2f5e6-0263-4ac7-aa12-98ea66f02d00/98d6fa79-aecc-429e-8624-4b6f706bdf4f.lease', 'imageID': '00d2f5e6-0263-4ac7-aa12-98ea66f02d00'}, 'path': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/00d2f5e6-0263-4ac7-aa12-98ea66f02d00/98d6fa79-aecc-429e-8624-4b6f706bdf4f', 'imgVolumesInfo': [{'domainID': 'ff3a0c94-11dc-446d-b093-5f68ad81520d', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/00d2f5e6-0263-4ac7-aa12-98ea66f02d00/98d6fa79-aecc-429e-8624-4b6f706bdf4f', 'volumeID': '98d6fa79-aecc-429e-8624-4b6f706bdf4f', 'leasePath': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/00d2f5e6-0263-4ac7-aa12-98ea66f02d00/98d6fa79-aecc-429e-8624-4b6f706bdf4f.lease', 'imageID': '00d2f5e6-0263-4ac7-aa12-98ea66f02d00'}]}
>clientIFinit::INFO::2014-01-28 00:15:05,022::clientIF::320::vds::(prepareVolumePath) prepared volume path: /rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/00d2f5e6-0263-4ac7-aa12-98ea66f02d00/98d6fa79-aecc-429e-8624-4b6f706bdf4f
>clientIFinit::WARNING::2014-01-28 00:15:05,023::vm::2053::vm.Vm::(buildConfDevices) vmId=`01d705e3-6b62-4796-b9e4-4de1c477401a`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel0', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'1'}}' found
>clientIFinit::WARNING::2014-01-28 00:15:05,023::vm::2053::vm.Vm::(buildConfDevices) vmId=`01d705e3-6b62-4796-b9e4-4de1c477401a`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel1', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'2'}}' found
>clientIFinit::WARNING::2014-01-28 00:15:05,023::vm::2053::vm.Vm::(buildConfDevices) vmId=`01d705e3-6b62-4796-b9e4-4de1c477401a`::Unknown type found, device: '{'device': u'spicevmc', 'alias': u'channel2', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'3'}}' found
>clientIFinit::INFO::2014-01-28 00:15:05,024::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='a746f514-51eb-4926-80d5-545108438f01', volUUID='ba8685ec-668a-4780-88ba-0b114e1a7e7d', options=None)
>clientIFinit::INFO::2014-01-28 00:15:05,027::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2744217600', 'apparentsize': '53687091200'}
>clientIFinit::INFO::2014-01-28 00:15:05,027::clientIF::320::vds::(prepareVolumePath) prepared volume path: 
>clientIFinit::INFO::2014-01-28 00:15:05,028::logUtils::44::dispatcher::(wrapper) Run and protect: prepareImage(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='a746f514-51eb-4926-80d5-545108438f01', leafUUID='ba8685ec-668a-4780-88ba-0b114e1a7e7d')
>clientIFinit::WARNING::2014-01-28 00:15:05,076::fileUtils::167::Storage.fileUtils::(createdir) Dir /var/run/vdsm/storage/ff3a0c94-11dc-446d-b093-5f68ad81520d already exists
>clientIFinit::DEBUG::2014-01-28 00:15:05,076::fileSD::446::Storage.StorageDomain::(createImageLinks) img run dir already exists: /var/run/vdsm/storage/ff3a0c94-11dc-446d-b093-5f68ad81520d/a746f514-51eb-4926-80d5-545108438f01
>clientIFinit::DEBUG::2014-01-28 00:15:05,077::fileVolume::529::Storage.Volume::(validateVolumePath) validate path for ba8685ec-668a-4780-88ba-0b114e1a7e7d
>clientIFinit::INFO::2014-01-28 00:15:05,078::logUtils::47::dispatcher::(wrapper) Run and protect: prepareImage, Return response: {'info': {'domainID': 'ff3a0c94-11dc-446d-b093-5f68ad81520d', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/a746f514-51eb-4926-80d5-545108438f01/ba8685ec-668a-4780-88ba-0b114e1a7e7d', 'volumeID': 'ba8685ec-668a-4780-88ba-0b114e1a7e7d', 'leasePath': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/a746f514-51eb-4926-80d5-545108438f01/ba8685ec-668a-4780-88ba-0b114e1a7e7d.lease', 'imageID': 'a746f514-51eb-4926-80d5-545108438f01'}, 'path': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/a746f514-51eb-4926-80d5-545108438f01/ba8685ec-668a-4780-88ba-0b114e1a7e7d', 'imgVolumesInfo': [{'domainID': 'ff3a0c94-11dc-446d-b093-5f68ad81520d', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/a746f514-51eb-4926-80d5-545108438f01/ba8685ec-668a-4780-88ba-0b114e1a7e7d', 'volumeID': 'ba8685ec-668a-4780-88ba-0b114e1a7e7d', 'leasePath': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/a746f514-51eb-4926-80d5-545108438f01/ba8685ec-668a-4780-88ba-0b114e1a7e7d.lease', 'imageID': 'a746f514-51eb-4926-80d5-545108438f01'}]}
>clientIFinit::INFO::2014-01-28 00:15:05,079::clientIF::320::vds::(prepareVolumePath) prepared volume path: /rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/a746f514-51eb-4926-80d5-545108438f01/ba8685ec-668a-4780-88ba-0b114e1a7e7d
>clientIFinit::WARNING::2014-01-28 00:15:05,080::vm::2053::vm.Vm::(buildConfDevices) vmId=`e85be056-7715-485c-9cb8-69f9980ed3a3`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel0', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'1'}}' found
>clientIFinit::WARNING::2014-01-28 00:15:05,080::vm::2053::vm.Vm::(buildConfDevices) vmId=`e85be056-7715-485c-9cb8-69f9980ed3a3`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel1', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'2'}}' found
>clientIFinit::WARNING::2014-01-28 00:15:05,080::vm::2053::vm.Vm::(buildConfDevices) vmId=`e85be056-7715-485c-9cb8-69f9980ed3a3`::Unknown type found, device: '{'device': u'spicevmc', 'alias': u'channel2', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'3'}}' found
>clientIFinit::INFO::2014-01-28 00:15:05,081::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='6b90cf80-28ea-48a6-9a11-bc9dcc835138', volUUID='487cfec8-3f1a-4c04-9b54-8aa7b4a5ae68', options=None)
>clientIFinit::INFO::2014-01-28 00:15:05,084::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2007703552', 'apparentsize': '2007826432'}
>clientIFinit::INFO::2014-01-28 00:15:05,084::clientIF::320::vds::(prepareVolumePath) prepared volume path: 
>clientIFinit::INFO::2014-01-28 00:15:05,085::logUtils::44::dispatcher::(wrapper) Run and protect: prepareImage(sdUUID='ff3a0c94-11dc-446d-b093-5f68ad81520d', spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', imgUUID='6b90cf80-28ea-48a6-9a11-bc9dcc835138', leafUUID='487cfec8-3f1a-4c04-9b54-8aa7b4a5ae68')
>clientIFinit::WARNING::2014-01-28 00:15:05,148::fileUtils::167::Storage.fileUtils::(createdir) Dir /var/run/vdsm/storage/ff3a0c94-11dc-446d-b093-5f68ad81520d already exists
>clientIFinit::DEBUG::2014-01-28 00:15:05,148::fileSD::446::Storage.StorageDomain::(createImageLinks) img run dir already exists: /var/run/vdsm/storage/ff3a0c94-11dc-446d-b093-5f68ad81520d/6b90cf80-28ea-48a6-9a11-bc9dcc835138
>clientIFinit::DEBUG::2014-01-28 00:15:05,150::fileVolume::529::Storage.Volume::(validateVolumePath) validate path for 487cfec8-3f1a-4c04-9b54-8aa7b4a5ae68
>clientIFinit::INFO::2014-01-28 00:15:05,151::logUtils::47::dispatcher::(wrapper) Run and protect: prepareImage, Return response: {'info': {'domainID': 'ff3a0c94-11dc-446d-b093-5f68ad81520d', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/6b90cf80-28ea-48a6-9a11-bc9dcc835138/487cfec8-3f1a-4c04-9b54-8aa7b4a5ae68', 'volumeID': '487cfec8-3f1a-4c04-9b54-8aa7b4a5ae68', 'leasePath': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/6b90cf80-28ea-48a6-9a11-bc9dcc835138/487cfec8-3f1a-4c04-9b54-8aa7b4a5ae68.lease', 'imageID': '6b90cf80-28ea-48a6-9a11-bc9dcc835138'}, 'path': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/6b90cf80-28ea-48a6-9a11-bc9dcc835138/487cfec8-3f1a-4c04-9b54-8aa7b4a5ae68', 'imgVolumesInfo': [{'domainID': 'ff3a0c94-11dc-446d-b093-5f68ad81520d', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/6b90cf80-28ea-48a6-9a11-bc9dcc835138/487cfec8-3f1a-4c04-9b54-8aa7b4a5ae68', 'volumeID': '487cfec8-3f1a-4c04-9b54-8aa7b4a5ae68', 'leasePath': '/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/6b90cf80-28ea-48a6-9a11-bc9dcc835138/487cfec8-3f1a-4c04-9b54-8aa7b4a5ae68.lease', 'imageID': '6b90cf80-28ea-48a6-9a11-bc9dcc835138'}]}
>clientIFinit::INFO::2014-01-28 00:15:05,152::clientIF::320::vds::(prepareVolumePath) prepared volume path: /rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/images/6b90cf80-28ea-48a6-9a11-bc9dcc835138/487cfec8-3f1a-4c04-9b54-8aa7b4a5ae68
>Thread-154::DEBUG::2014-01-28 00:15:05,163::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-154::INFO::2014-01-28 00:15:05,164::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='4155532d-c8c0-4274-8551-25b0193797b3', spUUID=None, options=None)
>Thread-154::INFO::2014-01-28 00:15:05,164::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '4155532d-c8c0-4274-8551-25b0193797b3'}}
>Thread-155::DEBUG::2014-01-28 00:15:05,652::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call getCapabilities with () {}
>Thread-155::DEBUG::2014-01-28 00:15:05,654::utils::555::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-155::DEBUG::2014-01-28 00:15:05,674::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-155::DEBUG::2014-01-28 00:15:05,697::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-155::DEBUG::2014-01-28 00:15:05,720::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-155::DEBUG::2014-01-28 00:15:05,742::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.2' (cwd None)
>Thread-155::DEBUG::2014-01-28 00:15:05,762::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-155::DEBUG::2014-01-28 00:15:05,763::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.2' (cwd None)
>Thread-155::DEBUG::2014-01-28 00:15:05,783::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-155::DEBUG::2014-01-28 00:15:05,786::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.3' (cwd None)
>Thread-155::DEBUG::2014-01-28 00:15:05,806::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-155::DEBUG::2014-01-28 00:15:05,807::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.3' (cwd None)
>Thread-155::DEBUG::2014-01-28 00:15:05,827::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-155::DEBUG::2014-01-28 00:15:05,830::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.10' (cwd None)
>Thread-155::DEBUG::2014-01-28 00:15:05,850::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-155::DEBUG::2014-01-28 00:15:05,851::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.10' (cwd None)
>Thread-155::DEBUG::2014-01-28 00:15:05,870::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-155::DEBUG::2014-01-28 00:15:05,874::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.11' (cwd None)
>Thread-155::DEBUG::2014-01-28 00:15:05,894::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-155::DEBUG::2014-01-28 00:15:05,895::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.11' (cwd None)
>Thread-155::DEBUG::2014-01-28 00:15:05,915::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-155::DEBUG::2014-01-28 00:15:05,928::caps::453::root::(_getKeyPackages) rpm package glusterfs-rdma not found
>Thread-155::DEBUG::2014-01-28 00:15:05,928::caps::453::root::(_getKeyPackages) rpm package glusterfs-fuse not found
>Thread-155::DEBUG::2014-01-28 00:15:05,930::caps::453::root::(_getKeyPackages) rpm package gluster-swift not found
>Thread-155::DEBUG::2014-01-28 00:15:05,936::caps::453::root::(_getKeyPackages) rpm package gluster-swift-object not found
>Thread-155::DEBUG::2014-01-28 00:15:05,937::caps::453::root::(_getKeyPackages) rpm package glusterfs not found
>Thread-155::DEBUG::2014-01-28 00:15:05,937::caps::453::root::(_getKeyPackages) rpm package gluster-swift-plugin not found
>Thread-155::DEBUG::2014-01-28 00:15:05,944::caps::453::root::(_getKeyPackages) rpm package gluster-swift-account not found
>Thread-155::DEBUG::2014-01-28 00:15:05,945::caps::453::root::(_getKeyPackages) rpm package gluster-swift-proxy not found
>Thread-155::DEBUG::2014-01-28 00:15:05,945::caps::453::root::(_getKeyPackages) rpm package gluster-swift-doc not found
>Thread-155::DEBUG::2014-01-28 00:15:05,946::caps::453::root::(_getKeyPackages) rpm package glusterfs-server not found
>Thread-155::DEBUG::2014-01-28 00:15:05,946::caps::453::root::(_getKeyPackages) rpm package gluster-swift-container not found
>Thread-155::DEBUG::2014-01-28 00:15:05,947::caps::453::root::(_getKeyPackages) rpm package glusterfs-geo-replication not found
>Thread-155::DEBUG::2014-01-28 00:15:05,952::BindingXMLRPC::976::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.3.1.el6.x86_64', 'buildtime': 1388745567.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '33.gitf029545.el6', 'buildtime': 1390094004L, 'version': '4.14.0'}, 'qemu-kvm': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.2', 'buildtime': 1387360004L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'mom': {'release': '20140117.gitfd877c5.el6', 'buildtime': 1389966957L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           L5320  @ 1.86GHz', 'hooks': {'before_vm_start': {'50_hostedengine': {'md5': '45dde62155b5412eafbfff5ef265acc2'}}}, 'cpuSockets': '2', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'gateway': '172.16.0.1', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.2', 'vnet0']}, 'os_net': {'iface': 'os_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.10']}, 'r2o_net': {'iface': 'r2o_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}, 'storage_network': {'iface': u'bond0.3', 'addr': '172.16.1.5', 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000', 'netmask': '255.255.255.255', 'bridged': False, 'interface': u'bond0.3', 'ipv6gateway': '::', 'gateway': ''}}, 'bridges': {'ovirtmgmt': {'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '172.16.0.1', 'ports': ['bond0.2', 'vnet0']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}, 'os_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.10']}, 'r2o_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}}, 'uuid': '44454C4C-3100-104D-8046-B9C04F583153', 'lastClientIface': 'ovirtmgmt', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'cb327caa-ca77-4efe-b7b8-63aba3cda84d', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:85', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth2', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:85', 'speed': 1000, 'mtu': '4000'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'e401a6a5-8773-4971-9213-b3c2bb52f16c', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:83', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:83', 'speed': 1000, 'mtu': '4000'}, 'eth0': {'netmask': '', 'addr': '', 'hwaddr': '00:e0:4c:80:13:02', 'cfg': {'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': '00:e0:4c:80:13:02', 'STP': 'no', 'DEVICE': 'eth0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'speed': 1000, 'mtu': '1500'}}, 'software_revision': '33', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,tm2,ssse3,cx16,xtpr,pdcm,dca,lahf_lm,dts,tpr_shadow,vnmi,flexpriority,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': ['eth1', 'eth2'], 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'MTU': '4000', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'BONDING_OPTS': 'miimon=100 mode=balance-rr', 'DEVICE': 'bond0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '32105', 'cpuSpeed': '1861.936', 'version_name': 'Snow Man', 'vlans': {'bond0.2': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'ovirtmgmt', 'VLAN': 'yes', 'MTU': '4000', 'BOOTPROTO': 'none', 'DEVICE': 'bond0.2', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 2, 'mtu': '4000'}, 'bond0.3': {'netmask': '255.255.255.255', 'iface': 'bond0', 'addr': '172.16.1.5', 'cfg': {'VLAN': 'yes', 'DEFROUTE': 'no', 'MTU': '4000', 'NM_CONTROLLED': 'no', 'DEVICE': 'bond0.3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 3, 'mtu': '4000'}, 'eth0.11': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'r2o_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.11', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 11, 'mtu': '1500'}, 'eth0.10': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'os_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.10', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 10, 'mtu': '1500'}}, 'cpuCores': '8', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '8', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '172.16.0.11'}}
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:06,019::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:06,019::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:06,020::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:06,020::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:06,020::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:06,021::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:06,021::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:06,022::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:06,022::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:06,022::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:06,022::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:06,023::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-156::DEBUG::2014-01-28 00:15:06,194::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-156::INFO::2014-01-28 00:15:06,195::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='4155532d-c8c0-4274-8551-25b0193797b3', spUUID=None, options=None)
>Thread-156::INFO::2014-01-28 00:15:06,196::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '4155532d-c8c0-4274-8551-25b0193797b3'}}
>Thread-12::DEBUG::2014-01-28 00:15:06,947::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:15:06,980::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-157::DEBUG::2014-01-28 00:15:07,204::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-157::INFO::2014-01-28 00:15:07,205::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='4155532d-c8c0-4274-8551-25b0193797b3', spUUID=None, options=None)
>Thread-157::INFO::2014-01-28 00:15:07,205::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '4155532d-c8c0-4274-8551-25b0193797b3'}}
>Thread-159::DEBUG::2014-01-28 00:15:07,773::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call tasksList with () {}
>Thread-159::ERROR::2014-01-28 00:15:07,812::BindingXMLRPC::985::vds::(wrapper) vdsm exception occured
>Traceback (most recent call last):
>  File "/usr/share/vdsm/BindingXMLRPC.py", line 972, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/gluster/api.py", line 54, in wrapper
>    rv = func(*args, **kwargs)
>  File "/usr/share/vdsm/gluster/api.py", line 306, in tasksList
>    status = self.svdsmProxy.glusterTasksList(taskIds)
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in glusterTasksList
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 740, in _callmethod
>    raise convert_to_error(kind, result)
>GlusterCmdExecFailedException: Command execution failed
>error: tasks is not a valid status option
>Usage: volume status [all | <VOLNAME> [nfs|shd|<BRICK>]] [detail|clients|mem|inode|fd|callpool]
>return code: 1
>Thread-160::DEBUG::2014-01-28 00:15:08,212::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-160::INFO::2014-01-28 00:15:08,213::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='4155532d-c8c0-4274-8551-25b0193797b3', spUUID=None, options=None)
>Thread-160::INFO::2014-01-28 00:15:08,214::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '4155532d-c8c0-4274-8551-25b0193797b3'}}
>Thread-161::DEBUG::2014-01-28 00:15:08,648::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call vmGetStats with ('2d425c4c-91b1-4564-b276-f27cc581e6a1',) {}
>Thread-161::DEBUG::2014-01-28 00:15:08,649::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-161::DEBUG::2014-01-28 00:15:08,649::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>Thread-161::DEBUG::2014-01-28 00:15:08,649::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-161::DEBUG::2014-01-28 00:15:08,649::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-161::DEBUG::2014-01-28 00:15:08,650::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>Thread-161::DEBUG::2014-01-28 00:15:08,650::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-161::DEBUG::2014-01-28 00:15:08,651::BindingXMLRPC::976::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'root', 'memUsage': '53', 'acpiEnable': 'true', 'guestFQDN': u'engine.melb.example.net', 'pid': '18113', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'/', u'total': '25323499520', u'used': '2592231424', u'fs': u'ext4'}, {u'path': u'/boot', u'total': '507744256', u'used': '55643136', u'fs': u'ext4'}], 'network': {u'vnet0': {'macAddr': '00:16:3e:0a:b7:55', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmType': 'kvm', 'guestName': u'engine.melb.example.net', 'elapsedTime': '610893', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'mem_free': '1959596', u'swap_in': '0', u'pageflt': '626', u'mem_total': '3922728', u'mem_unused': '1425260'}, 'cpuSys': '1.46', 'appsList': [u'ovirt-guest-agent-1.0.8-1.el6', u'kernel-2.6.32-431.el6', u'kernel-2.6.32-431.3.1.el6'], 'guestOs': u'2.6.32-431.3.1.el6.x86_64', 'displayType': 'vnc', 'status': 'Up', 'hash': '2950633987041661405', 'vmId': '2d425c4c-91b1-4564-b276-f27cc581e6a1', 'displayIp': '0', 'displayPort': u'5900', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'truesize': '3920592896', 'apparentsize': '26843545600', 'imageID': 'daa7fa08-1354-4b19-a80f-8694ea63f914'}, 'hdc': {'truesize': '0', 'apparentsize': '0'}}, 'monitorResponse': '0', 'statsAge': '0.25', 'cpuUser': '15.48', 'lastLogin': 1390828479.279238, 'guestIPs': u'172.16.0.10', 'netIfaces': [{u'hw': u'00:16:3e:0a:b7:55', u'name': u'eth0', u'inet': [u'172.16.0.10'], u'inet6': [u'fe80::216:3eff:fe0a:b755']}]}]}
>Thread-12::DEBUG::2014-01-28 00:15:08,999::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:15:09,021::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-162::DEBUG::2014-01-28 00:15:09,222::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-162::INFO::2014-01-28 00:15:09,223::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='4155532d-c8c0-4274-8551-25b0193797b3', spUUID=None, options=None)
>Thread-162::INFO::2014-01-28 00:15:09,224::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '4155532d-c8c0-4274-8551-25b0193797b3'}}
>Thread-163::DEBUG::2014-01-28 00:15:09,615::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {}
>Thread-163::DEBUG::2014-01-28 00:15:09,651::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-164::DEBUG::2014-01-28 00:15:10,230::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-164::INFO::2014-01-28 00:15:10,231::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='4155532d-c8c0-4274-8551-25b0193797b3', spUUID=None, options=None)
>Thread-164::INFO::2014-01-28 00:15:10,232::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '4155532d-c8c0-4274-8551-25b0193797b3'}}
>Thread-129::DEBUG::2014-01-28 00:15:10,942::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:15:10,963::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n364 bytes (364 B) copied, 0.000336253 s, 1.1 MB/s\n'; <rc> = 0
>Thread-128::DEBUG::2014-01-28 00:15:11,013::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/dom_md/metadata bs=4096 count=1' (cwd None)
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:11,035::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:11,038::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:11,041::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-12::DEBUG::2014-01-28 00:15:11,046::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:11,047::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:11,073::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:11,079::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:11,091::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:11,096::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:11,101::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:11,111::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-128::DEBUG::2014-01-28 00:15:11,112::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n504 bytes (504 B) copied, 0.0010249 s, 492 kB/s\n'; <rc> = 0
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:11,119::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:11,129::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-12::DEBUG::2014-01-28 00:15:11,146::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-166::DEBUG::2014-01-28 00:15:11,239::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-166::INFO::2014-01-28 00:15:11,241::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='4155532d-c8c0-4274-8551-25b0193797b3', spUUID=None, options=None)
>Thread-166::INFO::2014-01-28 00:15:11,241::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '4155532d-c8c0-4274-8551-25b0193797b3'}}
>Thread-167::DEBUG::2014-01-28 00:15:12,249::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-167::INFO::2014-01-28 00:15:12,250::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='4155532d-c8c0-4274-8551-25b0193797b3', spUUID=None, options=None)
>Thread-167::INFO::2014-01-28 00:15:12,251::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '4155532d-c8c0-4274-8551-25b0193797b3'}}
>Thread-12::DEBUG::2014-01-28 00:15:13,170::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:15:13,192::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-168::DEBUG::2014-01-28 00:15:13,258::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-168::INFO::2014-01-28 00:15:13,260::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='4155532d-c8c0-4274-8551-25b0193797b3', spUUID=None, options=None)
>Thread-168::INFO::2014-01-28 00:15:13,260::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '4155532d-c8c0-4274-8551-25b0193797b3'}}
>Thread-170::DEBUG::2014-01-28 00:15:14,269::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-170::INFO::2014-01-28 00:15:14,270::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='4155532d-c8c0-4274-8551-25b0193797b3', spUUID=None, options=None)
>Thread-170::INFO::2014-01-28 00:15:14,271::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '4155532d-c8c0-4274-8551-25b0193797b3'}}
>Thread-171::INFO::2014-01-28 00:15:14,607::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-171::INFO::2014-01-28 00:15:14,608::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000753818', 'lastCheck': '9.5', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000336253', 'lastCheck': '3.6', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00105706', 'lastCheck': '13.6', 'valid': True}}
>Thread-172::DEBUG::2014-01-28 00:15:14,644::BindingXMLRPC::159::vds::(wrapper) client [127.0.0.1]
>Thread-172::INFO::2014-01-28 00:15:14,645::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='7e0cc075-eee7-4268-946e-33a79dd08748', conList=[{'port': '', 'connection': '172.16.1.5:/HOSTED-ENGINE', 'iqn': '', 'portal': '', 'user': 'kvm', 'protocol_version': '3', 'password': '******', 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}], options=None)
>Thread-172::DEBUG::2014-01-28 00:15:14,650::hsm::2273::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE
>Thread-172::DEBUG::2014-01-28 00:15:14,651::hsm::2297::Storage.HSM::(__prefetchDomains) Found SD uuids: ('562f0160-7b80-42e2-b248-5754455c40fc',)
>Thread-172::DEBUG::2014-01-28 00:15:14,652::hsm::2353::Storage.HSM::(connectStorageServer) knownSDs: {562f0160-7b80-42e2-b248-5754455c40fc: storage.nfsSD.findDomain, 436c344c-bc57-441b-b311-c9595c6039e1: storage.nfsSD.findDomain, ff3a0c94-11dc-446d-b093-5f68ad81520d: storage.glusterSD.findDomain}
>Thread-172::INFO::2014-01-28 00:15:14,652::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}]}
>Thread-173::DEBUG::2014-01-28 00:15:14,811::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {}
>Thread-173::DEBUG::2014-01-28 00:15:14,864::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-174::DEBUG::2014-01-28 00:15:14,906::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.11]
>Thread-174::INFO::2014-01-28 00:15:14,907::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-174::INFO::2014-01-28 00:15:14,907::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000753818', 'lastCheck': '9.8', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000336253', 'lastCheck': '3.9', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00105706', 'lastCheck': '13.9', 'valid': True}}
>Thread-17::DEBUG::2014-01-28 00:15:15,105::iscsi::393::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-17::DEBUG::2014-01-28 00:15:15,105::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-17::DEBUG::2014-01-28 00:15:15,140::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:15:15,207::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:15:15,237::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-17::DEBUG::2014-01-28 00:15:15,255::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-176::DEBUG::2014-01-28 00:15:15,319::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-176::INFO::2014-01-28 00:15:15,335::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='4155532d-c8c0-4274-8551-25b0193797b3', spUUID=None, options=None)
>Thread-176::INFO::2014-01-28 00:15:15,337::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '4155532d-c8c0-4274-8551-25b0193797b3'}}
>Thread-17::DEBUG::2014-01-28 00:15:15,348::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc
>Thread-17::DEBUG::2014-01-28 00:15:15,350::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-17::DEBUG::2014-01-28 00:15:15,360::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=562f0160-7b80-42e2-b248-5754455c40fc:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=7e0cc075-eee7-4268-946e-33a79dd08748', 'REMOTE_PATH=172.16.1.5:/HOSTED-ENGINE', 'ROLE=Master', 'SDUUID=562f0160-7b80-42e2-b248-5754455c40fc', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=14825cb0b944c5c2ae9ed025c82e06730a2d91c4']
>Thread-17::DEBUG::2014-01-28 00:15:15,362::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-17::INFO::2014-01-28 00:15:15,363::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_imageNS already registered
>Thread-17::INFO::2014-01-28 00:15:15,363::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_volumeNS already registered
>Thread-17::DEBUG::2014-01-28 00:15:15,376::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-17::DEBUG::2014-01-28 00:15:15,399::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n478 bytes (478 B) copied, 0.000533952 s, 895 kB/s\n'; <rc> = 0
>Thread-179::DEBUG::2014-01-28 00:15:15,660::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call getCapabilities with () {}
>Thread-179::DEBUG::2014-01-28 00:15:15,662::utils::555::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-179::DEBUG::2014-01-28 00:15:15,682::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-179::DEBUG::2014-01-28 00:15:15,711::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-179::DEBUG::2014-01-28 00:15:15,744::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-179::DEBUG::2014-01-28 00:15:15,763::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.2' (cwd None)
>Thread-179::DEBUG::2014-01-28 00:15:15,782::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-179::DEBUG::2014-01-28 00:15:15,783::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.2' (cwd None)
>Thread-179::DEBUG::2014-01-28 00:15:15,803::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-179::DEBUG::2014-01-28 00:15:15,806::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.3' (cwd None)
>Thread-179::DEBUG::2014-01-28 00:15:15,826::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-179::DEBUG::2014-01-28 00:15:15,827::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.3' (cwd None)
>Thread-179::DEBUG::2014-01-28 00:15:15,846::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-179::DEBUG::2014-01-28 00:15:15,849::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.10' (cwd None)
>Thread-179::DEBUG::2014-01-28 00:15:15,868::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-179::DEBUG::2014-01-28 00:15:15,869::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.10' (cwd None)
>Thread-179::DEBUG::2014-01-28 00:15:15,888::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-179::DEBUG::2014-01-28 00:15:15,893::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.11' (cwd None)
>Thread-179::DEBUG::2014-01-28 00:15:15,912::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-179::DEBUG::2014-01-28 00:15:15,913::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.11' (cwd None)
>Thread-179::DEBUG::2014-01-28 00:15:15,934::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-179::DEBUG::2014-01-28 00:15:15,943::caps::453::root::(_getKeyPackages) rpm package glusterfs-rdma not found
>Thread-179::DEBUG::2014-01-28 00:15:15,944::caps::453::root::(_getKeyPackages) rpm package glusterfs-fuse not found
>Thread-179::DEBUG::2014-01-28 00:15:15,945::caps::453::root::(_getKeyPackages) rpm package gluster-swift not found
>Thread-179::DEBUG::2014-01-28 00:15:15,951::caps::453::root::(_getKeyPackages) rpm package gluster-swift-object not found
>Thread-179::DEBUG::2014-01-28 00:15:15,951::caps::453::root::(_getKeyPackages) rpm package glusterfs not found
>Thread-179::DEBUG::2014-01-28 00:15:15,952::caps::453::root::(_getKeyPackages) rpm package gluster-swift-plugin not found
>Thread-179::DEBUG::2014-01-28 00:15:15,959::caps::453::root::(_getKeyPackages) rpm package gluster-swift-account not found
>Thread-179::DEBUG::2014-01-28 00:15:15,959::caps::453::root::(_getKeyPackages) rpm package gluster-swift-proxy not found
>Thread-179::DEBUG::2014-01-28 00:15:15,960::caps::453::root::(_getKeyPackages) rpm package gluster-swift-doc not found
>Thread-179::DEBUG::2014-01-28 00:15:15,960::caps::453::root::(_getKeyPackages) rpm package glusterfs-server not found
>Thread-179::DEBUG::2014-01-28 00:15:15,961::caps::453::root::(_getKeyPackages) rpm package gluster-swift-container not found
>Thread-179::DEBUG::2014-01-28 00:15:15,961::caps::453::root::(_getKeyPackages) rpm package glusterfs-geo-replication not found
>Thread-179::DEBUG::2014-01-28 00:15:15,966::BindingXMLRPC::976::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.3.1.el6.x86_64', 'buildtime': 1388745567.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '33.gitf029545.el6', 'buildtime': 1390094004L, 'version': '4.14.0'}, 'qemu-kvm': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.2', 'buildtime': 1387360004L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'mom': {'release': '20140117.gitfd877c5.el6', 'buildtime': 1389966957L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           L5320  @ 1.86GHz', 'hooks': {'before_vm_start': {'50_hostedengine': {'md5': '45dde62155b5412eafbfff5ef265acc2'}}}, 'cpuSockets': '2', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'gateway': '172.16.0.1', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.2', 'vnet0']}, 'os_net': {'iface': 'os_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.10']}, 'r2o_net': {'iface': 'r2o_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}, 'storage_network': {'iface': u'bond0.3', 'addr': '172.16.1.5', 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000', 'netmask': '255.255.255.255', 'bridged': False, 'interface': u'bond0.3', 'ipv6gateway': '::', 'gateway': ''}}, 'bridges': {'ovirtmgmt': {'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '172.16.0.1', 'ports': ['bond0.2', 'vnet0']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}, 'os_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.10']}, 'r2o_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}}, 'uuid': '44454C4C-3100-104D-8046-B9C04F583153', 'lastClientIface': 'ovirtmgmt', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'cb327caa-ca77-4efe-b7b8-63aba3cda84d', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:85', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth2', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:85', 'speed': 1000, 'mtu': '4000'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'e401a6a5-8773-4971-9213-b3c2bb52f16c', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:83', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:83', 'speed': 1000, 'mtu': '4000'}, 'eth0': {'netmask': '', 'addr': '', 'hwaddr': '00:e0:4c:80:13:02', 'cfg': {'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': '00:e0:4c:80:13:02', 'STP': 'no', 'DEVICE': 'eth0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'speed': 1000, 'mtu': '1500'}}, 'software_revision': '33', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,tm2,ssse3,cx16,xtpr,pdcm,dca,lahf_lm,dts,tpr_shadow,vnmi,flexpriority,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': ['eth1', 'eth2'], 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'MTU': '4000', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'BONDING_OPTS': 'miimon=100 mode=balance-rr', 'DEVICE': 'bond0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '32105', 'cpuSpeed': '1861.936', 'version_name': 'Snow Man', 'vlans': {'bond0.2': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'ovirtmgmt', 'VLAN': 'yes', 'MTU': '4000', 'BOOTPROTO': 'none', 'DEVICE': 'bond0.2', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 2, 'mtu': '4000'}, 'bond0.3': {'netmask': '255.255.255.255', 'iface': 'bond0', 'addr': '172.16.1.5', 'cfg': {'VLAN': 'yes', 'DEFROUTE': 'no', 'MTU': '4000', 'NM_CONTROLLED': 'no', 'DEVICE': 'bond0.3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 3, 'mtu': '4000'}, 'eth0.11': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'r2o_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.11', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 11, 'mtu': '1500'}, 'eth0.10': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'os_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.10', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 10, 'mtu': '1500'}}, 'cpuCores': '8', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '8', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '172.16.0.11'}}
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:16,149::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:16,149::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:16,149::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:16,150::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:16,150::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:16,150::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:16,151::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:16,151::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:16,151::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:16,152::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:16,152::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:16,152::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-180::DEBUG::2014-01-28 00:15:16,351::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-180::INFO::2014-01-28 00:15:16,352::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='4155532d-c8c0-4274-8551-25b0193797b3', spUUID=None, options=None)
>Thread-180::INFO::2014-01-28 00:15:16,353::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '4155532d-c8c0-4274-8551-25b0193797b3'}}
>Thread-12::DEBUG::2014-01-28 00:15:17,322::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:15:17,344::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-182::DEBUG::2014-01-28 00:15:17,371::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-182::INFO::2014-01-28 00:15:17,372::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='4155532d-c8c0-4274-8551-25b0193797b3', spUUID=None, options=None)
>Thread-182::INFO::2014-01-28 00:15:17,372::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '4155532d-c8c0-4274-8551-25b0193797b3'}}
>Thread-183::DEBUG::2014-01-28 00:15:18,381::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-183::INFO::2014-01-28 00:15:18,382::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='4155532d-c8c0-4274-8551-25b0193797b3', spUUID=None, options=None)
>Thread-183::INFO::2014-01-28 00:15:18,382::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '4155532d-c8c0-4274-8551-25b0193797b3'}}
>Thread-184::DEBUG::2014-01-28 00:15:18,652::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call vmGetStats with ('2d425c4c-91b1-4564-b276-f27cc581e6a1',) {}
>Thread-184::DEBUG::2014-01-28 00:15:18,652::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-184::DEBUG::2014-01-28 00:15:18,652::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>Thread-184::DEBUG::2014-01-28 00:15:18,653::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-184::DEBUG::2014-01-28 00:15:18,653::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-184::DEBUG::2014-01-28 00:15:18,653::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>Thread-184::DEBUG::2014-01-28 00:15:18,654::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-184::DEBUG::2014-01-28 00:15:18,654::BindingXMLRPC::976::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'root', 'memUsage': '53', 'acpiEnable': 'true', 'guestFQDN': u'engine.melb.example.net', 'pid': '18113', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'/', u'total': '25323499520', u'used': '2592231424', u'fs': u'ext4'}, {u'path': u'/boot', u'total': '507744256', u'used': '55643136', u'fs': u'ext4'}], 'network': {u'vnet0': {'macAddr': '00:16:3e:0a:b7:55', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmType': 'kvm', 'guestName': u'engine.melb.example.net', 'elapsedTime': '610903', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'mem_free': '1947644', u'swap_in': '0', u'pageflt': '144', u'mem_total': '3922728', u'mem_unused': '1413228'}, 'cpuSys': '1.46', 'appsList': [u'ovirt-guest-agent-1.0.8-1.el6', u'kernel-2.6.32-431.el6', u'kernel-2.6.32-431.3.1.el6'], 'guestOs': u'2.6.32-431.3.1.el6.x86_64', 'displayType': 'vnc', 'status': 'Up', 'hash': '2950633987041661405', 'vmId': '2d425c4c-91b1-4564-b276-f27cc581e6a1', 'displayIp': '0', 'displayPort': u'5900', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'truesize': '3920592896', 'apparentsize': '26843545600', 'imageID': 'daa7fa08-1354-4b19-a80f-8694ea63f914'}, 'hdc': {'truesize': '0', 'apparentsize': '0'}}, 'monitorResponse': '0', 'statsAge': '0.22', 'cpuUser': '15.48', 'lastLogin': 1390828479.279238, 'guestIPs': u'172.16.0.10', 'netIfaces': [{u'hw': u'00:16:3e:0a:b7:55', u'name': u'eth0', u'inet': [u'172.16.0.10'], u'inet6': [u'fe80::216:3eff:fe0a:b755']}]}]}
>Thread-12::DEBUG::2014-01-28 00:15:19,360::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:15:19,382::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-185::DEBUG::2014-01-28 00:15:19,588::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-185::INFO::2014-01-28 00:15:19,593::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='4155532d-c8c0-4274-8551-25b0193797b3', spUUID=None, options=None)
>Thread-185::INFO::2014-01-28 00:15:19,596::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '4155532d-c8c0-4274-8551-25b0193797b3'}}
>Thread-186::INFO::2014-01-28 00:15:19,597::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-186::INFO::2014-01-28 00:15:19,600::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000533952', 'lastCheck': '4.2', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000336253', 'lastCheck': '8.6', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00105706', 'lastCheck': '18.6', 'valid': True}}
>Thread-187::DEBUG::2014-01-28 00:15:19,660::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-187::DEBUG::2014-01-28 00:15:19,661::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>Thread-187::DEBUG::2014-01-28 00:15:19,661::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-187::DEBUG::2014-01-28 00:15:19,661::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-187::DEBUG::2014-01-28 00:15:19,662::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>Thread-187::DEBUG::2014-01-28 00:15:19,662::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-188::DEBUG::2014-01-28 00:15:20,024::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {} flowID [44f165b7]
>Thread-188::DEBUG::2014-01-28 00:15:20,070::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-189::DEBUG::2014-01-28 00:15:20,612::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-189::INFO::2014-01-28 00:15:20,613::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='4155532d-c8c0-4274-8551-25b0193797b3', spUUID=None, options=None)
>Thread-189::INFO::2014-01-28 00:15:20,613::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '4155532d-c8c0-4274-8551-25b0193797b3'}}
>Thread-129::DEBUG::2014-01-28 00:15:20,981::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:15:21,003::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n364 bytes (364 B) copied, 0.000390179 s, 933 kB/s\n'; <rc> = 0
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:21,159::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:21,159::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:21,160::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:21,160::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:21,160::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:21,161::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:21,161::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:21,162::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:21,162::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:21,162::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:21,162::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:21,163::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-12::DEBUG::2014-01-28 00:15:21,406::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:15:21,429::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-190::DEBUG::2014-01-28 00:15:21,621::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-190::INFO::2014-01-28 00:15:21,622::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='4155532d-c8c0-4274-8551-25b0193797b3', spUUID=None, options=None)
>Thread-190::INFO::2014-01-28 00:15:21,623::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '4155532d-c8c0-4274-8551-25b0193797b3'}}
>4155532d-c8c0-4274-8551-25b0193797b3::DEBUG::2014-01-28 00:15:22,272::sp::309::Storage.StoragePool::(startSpm) spm lock acquired successfully
>4155532d-c8c0-4274-8551-25b0193797b3::DEBUG::2014-01-28 00:15:22,292::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=VM-DATA', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=NextDC_M1', 'POOL_DOMAINS=ff3a0c94-11dc-446d-b093-5f68ad81520d:Active,436c344c-bc57-441b-b311-c9595c6039e1:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=1', 'POOL_UUID=5849b030-626e-47cb-ad90-3ce782d831b3', 'REMOTE_PATH=172.16.1.5:/VM-DATA', 'ROLE=Master', 'SDUUID=ff3a0c94-11dc-446d-b093-5f68ad81520d', 'TYPE=GLUSTERFS', 'VERSION=3', '_SHA_CKSUM=aed4059bcf1cca521e624929711bc334499d38cd']
>4155532d-c8c0-4274-8551-25b0193797b3::DEBUG::2014-01-28 00:15:22,292::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
>4155532d-c8c0-4274-8551-25b0193797b3::DEBUG::2014-01-28 00:15:22,293::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
>4155532d-c8c0-4274-8551-25b0193797b3::DEBUG::2014-01-28 00:15:22,293::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=VM-DATA', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=NextDC_M1', 'POOL_DOMAINS=ff3a0c94-11dc-446d-b093-5f68ad81520d:Active,436c344c-bc57-441b-b311-c9595c6039e1:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=5849b030-626e-47cb-ad90-3ce782d831b3', 'REMOTE_PATH=172.16.1.5:/VM-DATA', 'ROLE=Master', 'SDUUID=ff3a0c94-11dc-446d-b093-5f68ad81520d', 'TYPE=GLUSTERFS', 'VERSION=3', '_SHA_CKSUM=96f1a2aead4d5dd9b067189eb2a4a0e7359ad1ce']
>4155532d-c8c0-4274-8551-25b0193797b3::DEBUG::2014-01-28 00:15:22,299::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
>4155532d-c8c0-4274-8551-25b0193797b3::INFO::2014-01-28 00:15:22,300::sp::473::Storage.StoragePool::(_upgradePool) Trying to upgrade master domain `ff3a0c94-11dc-446d-b093-5f68ad81520d`
>4155532d-c8c0-4274-8551-25b0193797b3::DEBUG::2014-01-28 00:15:22,308::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d
>4155532d-c8c0-4274-8551-25b0193797b3::DEBUG::2014-01-28 00:15:22,310::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>4155532d-c8c0-4274-8551-25b0193797b3::DEBUG::2014-01-28 00:15:22,318::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=VM-DATA', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=NextDC_M1', 'POOL_DOMAINS=ff3a0c94-11dc-446d-b093-5f68ad81520d:Active,436c344c-bc57-441b-b311-c9595c6039e1:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=5849b030-626e-47cb-ad90-3ce782d831b3', 'REMOTE_PATH=172.16.1.5:/VM-DATA', 'ROLE=Master', 'SDUUID=ff3a0c94-11dc-446d-b093-5f68ad81520d', 'TYPE=GLUSTERFS', 'VERSION=3', '_SHA_CKSUM=96f1a2aead4d5dd9b067189eb2a4a0e7359ad1ce']
>4155532d-c8c0-4274-8551-25b0193797b3::DEBUG::2014-01-28 00:15:22,333::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>4155532d-c8c0-4274-8551-25b0193797b3::INFO::2014-01-28 00:15:22,335::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace ff3a0c94-11dc-446d-b093-5f68ad81520d_imageNS already registered
>4155532d-c8c0-4274-8551-25b0193797b3::INFO::2014-01-28 00:15:22,335::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace ff3a0c94-11dc-446d-b093-5f68ad81520d_volumeNS already registered
>4155532d-c8c0-4274-8551-25b0193797b3::DEBUG::2014-01-28 00:15:22,337::sp::478::Storage.StoragePool::(_upgradePool) Marking all domains for upgrade
>4155532d-c8c0-4274-8551-25b0193797b3::DEBUG::2014-01-28 00:15:22,346::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=VM-DATA', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=NextDC_M1', 'POOL_DOMAINS=ff3a0c94-11dc-446d-b093-5f68ad81520d:Active,436c344c-bc57-441b-b311-c9595c6039e1:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=5849b030-626e-47cb-ad90-3ce782d831b3', 'REMOTE_PATH=172.16.1.5:/VM-DATA', 'ROLE=Master', 'SDUUID=ff3a0c94-11dc-446d-b093-5f68ad81520d', 'TYPE=GLUSTERFS', 'VERSION=3', '_SHA_CKSUM=96f1a2aead4d5dd9b067189eb2a4a0e7359ad1ce']
>4155532d-c8c0-4274-8551-25b0193797b3::DEBUG::2014-01-28 00:15:22,347::sp::485::Storage.StoragePool::(_upgradePool) Registering with state change event
>4155532d-c8c0-4274-8551-25b0193797b3::DEBUG::2014-01-28 00:15:22,347::sp::488::Storage.StoragePool::(_upgradePool) Running initial domain upgrade threads
>Thread-192::DEBUG::2014-01-28 00:15:22,348::sp::195::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain 436c344c-bc57-441b-b311-c9595c6039e1
>4155532d-c8c0-4274-8551-25b0193797b3::DEBUG::2014-01-28 00:15:22,357::sd::620::Storage.StorageDomain::(createMasterTree) creating vms dir: /rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/master/vms
>Thread-192::DEBUG::2014-01-28 00:15:22,357::sp::218::Storage.StoragePool::(_upgradePoolDomain) Domain `436c344c-bc57-441b-b311-c9595c6039e1` is not a data domain it is an 2 domain, not upgrading
>Thread-192::DEBUG::2014-01-28 00:15:22,358::sp::235::Storage.StoragePool::(_upgradePoolDomain) All domains are upgraded, unregistering from state change event
>4155532d-c8c0-4274-8551-25b0193797b3::DEBUG::2014-01-28 00:15:22,359::sd::623::Storage.StorageDomain::(createMasterTree) creating task dir: /rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/master/tasks
>4155532d-c8c0-4274-8551-25b0193797b3::DEBUG::2014-01-28 00:15:22,370::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=VM-DATA', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=NextDC_M1', 'POOL_DOMAINS=ff3a0c94-11dc-446d-b093-5f68ad81520d:Active,436c344c-bc57-441b-b311-c9595c6039e1:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=5849b030-626e-47cb-ad90-3ce782d831b3', 'REMOTE_PATH=172.16.1.5:/VM-DATA', 'ROLE=Master', 'SDUUID=ff3a0c94-11dc-446d-b093-5f68ad81520d', 'TYPE=GLUSTERFS', 'VERSION=3', '_SHA_CKSUM=96f1a2aead4d5dd9b067189eb2a4a0e7359ad1ce']
>4155532d-c8c0-4274-8551-25b0193797b3::DEBUG::2014-01-28 00:15:22,385::sp::368::Storage.StoragePool::(startSpm) ended.
>Thread-193::DEBUG::2014-01-28 00:15:22,630::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-193::INFO::2014-01-28 00:15:22,632::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='4155532d-c8c0-4274-8551-25b0193797b3', spUUID=None, options=None)
>Thread-193::INFO::2014-01-28 00:15:22,632::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': '4155532d-c8c0-4274-8551-25b0193797b3'}}
>Thread-194::DEBUG::2014-01-28 00:15:22,641::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-194::INFO::2014-01-28 00:15:22,641::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-194::INFO::2014-01-28 00:15:22,642::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 2}}
>Thread-195::DEBUG::2014-01-28 00:15:22,695::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-195::INFO::2014-01-28 00:15:22,701::logUtils::44::dispatcher::(wrapper) Run and protect: clearTask(taskID='4155532d-c8c0-4274-8551-25b0193797b3', spUUID=None, options=None)
>Thread-195::INFO::2014-01-28 00:15:22,705::logUtils::47::dispatcher::(wrapper) Run and protect: clearTask, Return response: None
>Thread-197::DEBUG::2014-01-28 00:15:22,919::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-197::INFO::2014-01-28 00:15:22,920::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-197::INFO::2014-01-28 00:15:22,926::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': 'ff3a0c94-11dc-446d-b093-5f68ad81520d', 'name': 'NextDC_M1', 'version': '3', 'domains': 'ff3a0c94-11dc-446d-b093-5f68ad81520d:Active,436c344c-bc57-441b-b311-c9595c6039e1:Active', 'pool_status': 'connected', 'isoprefix': '/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/images/11111111-1111-1111-1111-111111111111', 'type': 'GLUSTERFS', 'master_ver': 1, 'lver': 2}, 'dominfo': {'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'status': 'Active', 'diskfree': '917084569600', 'isoprefix': '', 'alerts': [], 'disktotal': '982907879424', 'version': 3}, '436c344c-bc57-441b-b311-c9595c6039e1': {'status': 'Active', 'diskfree': '21444952064', 'isoprefix': '/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '25323634688', 'version': 0}}}
>Thread-12::DEBUG::2014-01-28 00:15:23,444::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:15:23,470::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-198::INFO::2014-01-28 00:15:24,608::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-198::INFO::2014-01-28 00:15:24,609::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000533952', 'lastCheck': '9.2', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000390179', 'lastCheck': '3.6', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0010249', 'lastCheck': '2.4', 'valid': True}}
>Thread-200::DEBUG::2014-01-28 00:15:25,217::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {} flowID [6be25d2]
>Thread-199::DEBUG::2014-01-28 00:15:25,248::BindingXMLRPC::159::vds::(wrapper) client [127.0.0.1]
>Thread-199::INFO::2014-01-28 00:15:25,249::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='7e0cc075-eee7-4268-946e-33a79dd08748', conList=[{'port': '', 'connection': '172.16.1.5:/HOSTED-ENGINE', 'iqn': '', 'portal': '', 'user': 'kvm', 'protocol_version': '3', 'password': '******', 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}], options=None)
>Thread-199::DEBUG::2014-01-28 00:15:25,253::hsm::2273::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE
>Thread-199::DEBUG::2014-01-28 00:15:25,254::hsm::2297::Storage.HSM::(__prefetchDomains) Found SD uuids: ('562f0160-7b80-42e2-b248-5754455c40fc',)
>Thread-199::DEBUG::2014-01-28 00:15:25,255::hsm::2353::Storage.HSM::(connectStorageServer) knownSDs: {562f0160-7b80-42e2-b248-5754455c40fc: storage.nfsSD.findDomain, 436c344c-bc57-441b-b311-c9595c6039e1: storage.nfsSD.findDomain, ff3a0c94-11dc-446d-b093-5f68ad81520d: storage.glusterSD.findDomain}
>Thread-199::INFO::2014-01-28 00:15:25,255::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}]}
>Thread-200::DEBUG::2014-01-28 00:15:25,263::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-17::DEBUG::2014-01-28 00:15:25,406::iscsi::393::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-17::DEBUG::2014-01-28 00:15:25,407::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-17::DEBUG::2014-01-28 00:15:25,477::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:15:25,503::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:15:25,530::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-201::DEBUG::2014-01-28 00:15:25,572::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.11]
>Thread-201::INFO::2014-01-28 00:15:25,574::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-201::INFO::2014-01-28 00:15:25,574::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000533952', 'lastCheck': '10.2', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000390179', 'lastCheck': '4.6', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0010249', 'lastCheck': '3.4', 'valid': True}}
>Thread-17::DEBUG::2014-01-28 00:15:25,631::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-17::DEBUG::2014-01-28 00:15:25,662::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc
>Thread-17::DEBUG::2014-01-28 00:15:25,664::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-17::DEBUG::2014-01-28 00:15:25,674::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=562f0160-7b80-42e2-b248-5754455c40fc:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=7e0cc075-eee7-4268-946e-33a79dd08748', 'REMOTE_PATH=172.16.1.5:/HOSTED-ENGINE', 'ROLE=Master', 'SDUUID=562f0160-7b80-42e2-b248-5754455c40fc', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=14825cb0b944c5c2ae9ed025c82e06730a2d91c4']
>Thread-17::DEBUG::2014-01-28 00:15:25,677::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-17::INFO::2014-01-28 00:15:25,677::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_imageNS already registered
>Thread-17::INFO::2014-01-28 00:15:25,678::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_volumeNS already registered
>Thread-205::DEBUG::2014-01-28 00:15:25,684::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call getCapabilities with () {}
>Thread-205::DEBUG::2014-01-28 00:15:25,689::utils::555::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-205::DEBUG::2014-01-28 00:15:25,722::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-17::DEBUG::2014-01-28 00:15:25,743::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-17::DEBUG::2014-01-28 00:15:25,775::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n478 bytes (478 B) copied, 0.00066343 s, 720 kB/s\n'; <rc> = 0
>Thread-205::DEBUG::2014-01-28 00:15:25,800::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-205::DEBUG::2014-01-28 00:15:25,825::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-205::DEBUG::2014-01-28 00:15:25,844::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.2' (cwd None)
>Thread-205::DEBUG::2014-01-28 00:15:25,877::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-205::DEBUG::2014-01-28 00:15:25,883::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.2' (cwd None)
>Thread-205::DEBUG::2014-01-28 00:15:25,911::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-205::DEBUG::2014-01-28 00:15:25,914::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.3' (cwd None)
>Thread-205::DEBUG::2014-01-28 00:15:25,933::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-205::DEBUG::2014-01-28 00:15:25,934::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.3' (cwd None)
>Thread-205::DEBUG::2014-01-28 00:15:25,958::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-205::DEBUG::2014-01-28 00:15:25,960::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.10' (cwd None)
>Thread-205::DEBUG::2014-01-28 00:15:25,980::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-205::DEBUG::2014-01-28 00:15:25,981::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.10' (cwd None)
>Thread-205::DEBUG::2014-01-28 00:15:26,000::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-205::DEBUG::2014-01-28 00:15:26,005::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.11' (cwd None)
>Thread-205::DEBUG::2014-01-28 00:15:26,024::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-205::DEBUG::2014-01-28 00:15:26,025::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.11' (cwd None)
>Thread-205::DEBUG::2014-01-28 00:15:26,045::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-205::DEBUG::2014-01-28 00:15:26,055::caps::453::root::(_getKeyPackages) rpm package glusterfs-rdma not found
>Thread-205::DEBUG::2014-01-28 00:15:26,055::caps::453::root::(_getKeyPackages) rpm package glusterfs-fuse not found
>Thread-205::DEBUG::2014-01-28 00:15:26,056::caps::453::root::(_getKeyPackages) rpm package gluster-swift not found
>Thread-205::DEBUG::2014-01-28 00:15:26,062::caps::453::root::(_getKeyPackages) rpm package gluster-swift-object not found
>Thread-205::DEBUG::2014-01-28 00:15:26,063::caps::453::root::(_getKeyPackages) rpm package glusterfs not found
>Thread-205::DEBUG::2014-01-28 00:15:26,063::caps::453::root::(_getKeyPackages) rpm package gluster-swift-plugin not found
>Thread-205::DEBUG::2014-01-28 00:15:26,070::caps::453::root::(_getKeyPackages) rpm package gluster-swift-account not found
>Thread-205::DEBUG::2014-01-28 00:15:26,071::caps::453::root::(_getKeyPackages) rpm package gluster-swift-proxy not found
>Thread-205::DEBUG::2014-01-28 00:15:26,071::caps::453::root::(_getKeyPackages) rpm package gluster-swift-doc not found
>Thread-205::DEBUG::2014-01-28 00:15:26,072::caps::453::root::(_getKeyPackages) rpm package glusterfs-server not found
>Thread-205::DEBUG::2014-01-28 00:15:26,072::caps::453::root::(_getKeyPackages) rpm package gluster-swift-container not found
>Thread-205::DEBUG::2014-01-28 00:15:26,072::caps::453::root::(_getKeyPackages) rpm package glusterfs-geo-replication not found
>Thread-205::DEBUG::2014-01-28 00:15:26,078::BindingXMLRPC::976::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.3.1.el6.x86_64', 'buildtime': 1388745567.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '33.gitf029545.el6', 'buildtime': 1390094004L, 'version': '4.14.0'}, 'qemu-kvm': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.2', 'buildtime': 1387360004L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'mom': {'release': '20140117.gitfd877c5.el6', 'buildtime': 1389966957L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           L5320  @ 1.86GHz', 'hooks': {'before_vm_start': {'50_hostedengine': {'md5': '45dde62155b5412eafbfff5ef265acc2'}}}, 'cpuSockets': '2', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'gateway': '172.16.0.1', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.2', 'vnet0']}, 'os_net': {'iface': 'os_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.10']}, 'r2o_net': {'iface': 'r2o_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}, 'storage_network': {'iface': u'bond0.3', 'addr': '172.16.1.5', 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000', 'netmask': '255.255.255.255', 'bridged': False, 'interface': u'bond0.3', 'ipv6gateway': '::', 'gateway': ''}}, 'bridges': {'ovirtmgmt': {'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '172.16.0.1', 'ports': ['bond0.2', 'vnet0']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}, 'os_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.10']}, 'r2o_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}}, 'uuid': '44454C4C-3100-104D-8046-B9C04F583153', 'lastClientIface': 'ovirtmgmt', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'cb327caa-ca77-4efe-b7b8-63aba3cda84d', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:85', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth2', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:85', 'speed': 1000, 'mtu': '4000'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'e401a6a5-8773-4971-9213-b3c2bb52f16c', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:83', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:83', 'speed': 1000, 'mtu': '4000'}, 'eth0': {'netmask': '', 'addr': '', 'hwaddr': '00:e0:4c:80:13:02', 'cfg': {'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': '00:e0:4c:80:13:02', 'STP': 'no', 'DEVICE': 'eth0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'speed': 1000, 'mtu': '1500'}}, 'software_revision': '33', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,tm2,ssse3,cx16,xtpr,pdcm,dca,lahf_lm,dts,tpr_shadow,vnmi,flexpriority,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': ['eth1', 'eth2'], 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'MTU': '4000', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'BONDING_OPTS': 'miimon=100 mode=balance-rr', 'DEVICE': 'bond0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '32105', 'cpuSpeed': '1861.936', 'version_name': 'Snow Man', 'vlans': {'bond0.2': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'ovirtmgmt', 'VLAN': 'yes', 'MTU': '4000', 'BOOTPROTO': 'none', 'DEVICE': 'bond0.2', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 2, 'mtu': '4000'}, 'bond0.3': {'netmask': '255.255.255.255', 'iface': 'bond0', 'addr': '172.16.1.5', 'cfg': {'VLAN': 'yes', 'DEFROUTE': 'no', 'MTU': '4000', 'NM_CONTROLLED': 'no', 'DEVICE': 'bond0.3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 3, 'mtu': '4000'}, 'eth0.11': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'r2o_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.11', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 11, 'mtu': '1500'}, 'eth0.10': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'os_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.10', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 10, 'mtu': '1500'}}, 'cpuCores': '8', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '8', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '172.16.0.11'}}
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:26,169::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:26,169::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:26,170::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:26,170::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:26,170::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:26,171::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:26,171::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:26,172::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:26,172::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:26,172::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:26,172::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:26,173::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-12::DEBUG::2014-01-28 00:15:27,563::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:15:27,595::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-207::DEBUG::2014-01-28 00:15:28,662::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call vmGetStats with ('2d425c4c-91b1-4564-b276-f27cc581e6a1',) {}
>Thread-207::DEBUG::2014-01-28 00:15:28,662::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-207::DEBUG::2014-01-28 00:15:28,663::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>Thread-207::DEBUG::2014-01-28 00:15:28,663::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-207::DEBUG::2014-01-28 00:15:28,663::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-207::DEBUG::2014-01-28 00:15:28,663::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>Thread-207::DEBUG::2014-01-28 00:15:28,664::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-207::DEBUG::2014-01-28 00:15:28,664::BindingXMLRPC::976::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'root', 'memUsage': '53', 'acpiEnable': 'true', 'guestFQDN': u'engine.melb.example.net', 'pid': '18113', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'/', u'total': '25323499520', u'used': '2592231424', u'fs': u'ext4'}, {u'path': u'/boot', u'total': '507744256', u'used': '55643136', u'fs': u'ext4'}], 'network': {u'vnet0': {'macAddr': '00:16:3e:0a:b7:55', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmType': 'kvm', 'guestName': u'engine.melb.example.net', 'elapsedTime': '610913', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'mem_free': '1935676', u'swap_in': '0', u'pageflt': '1522', u'mem_total': '3922728', u'mem_unused': '1401076'}, 'cpuSys': '1.73', 'appsList': [u'ovirt-guest-agent-1.0.8-1.el6', u'kernel-2.6.32-431.el6', u'kernel-2.6.32-431.3.1.el6'], 'guestOs': u'2.6.32-431.3.1.el6.x86_64', 'displayType': 'vnc', 'status': 'Up', 'hash': '2950633987041661405', 'vmId': '2d425c4c-91b1-4564-b276-f27cc581e6a1', 'displayIp': '0', 'displayPort': u'5900', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'truesize': '3920592896', 'apparentsize': '26843545600', 'imageID': 'daa7fa08-1354-4b19-a80f-8694ea63f914'}, 'hdc': {'truesize': '0', 'apparentsize': '0'}}, 'monitorResponse': '0', 'statsAge': '0.19', 'cpuUser': '14.21', 'lastLogin': 1390828479.279238, 'guestIPs': u'172.16.0.10', 'netIfaces': [{u'hw': u'00:16:3e:0a:b7:55', u'name': u'eth0', u'inet': [u'172.16.0.10'], u'inet6': [u'fe80::216:3eff:fe0a:b755']}]}]}
>Thread-12::DEBUG::2014-01-28 00:15:29,616::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:15:29,638::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-209::DEBUG::2014-01-28 00:15:30,383::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {} flowID [195dc414]
>Thread-209::DEBUG::2014-01-28 00:15:30,450::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-129::DEBUG::2014-01-28 00:15:31,020::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:15:31,041::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n364 bytes (364 B) copied, 0.000365527 s, 996 kB/s\n'; <rc> = 0
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:31,179::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:31,179::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:31,180::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:31,180::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:31,180::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:31,181::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:31,181::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:31,182::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:31,182::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:31,182::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:31,183::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:31,183::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-12::DEBUG::2014-01-28 00:15:31,655::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:15:31,677::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-128::DEBUG::2014-01-28 00:15:32,220::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-128::DEBUG::2014-01-28 00:15:32,242::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n504 bytes (504 B) copied, 0.000925856 s, 544 kB/s\n'; <rc> = 0
>Thread-211::DEBUG::2014-01-28 00:15:33,200::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-211::INFO::2014-01-28 00:15:33,201::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-211::INFO::2014-01-28 00:15:33,202::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 2}}
>Thread-212::DEBUG::2014-01-28 00:15:33,211::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-212::INFO::2014-01-28 00:15:33,212::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-212::INFO::2014-01-28 00:15:33,218::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': 'ff3a0c94-11dc-446d-b093-5f68ad81520d', 'name': 'NextDC_M1', 'version': '3', 'domains': 'ff3a0c94-11dc-446d-b093-5f68ad81520d:Active,436c344c-bc57-441b-b311-c9595c6039e1:Active', 'pool_status': 'connected', 'isoprefix': '/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/images/11111111-1111-1111-1111-111111111111', 'type': 'GLUSTERFS', 'master_ver': 1, 'lver': 2}, 'dominfo': {'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'status': 'Active', 'diskfree': '917084569600', 'isoprefix': '', 'alerts': [], 'disktotal': '982907879424', 'version': 3}, '436c344c-bc57-441b-b311-c9595c6039e1': {'status': 'Active', 'diskfree': '21444952064', 'isoprefix': '/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '25323634688', 'version': 0}}}
>Thread-12::DEBUG::2014-01-28 00:15:33,693::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:15:33,724::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-213::INFO::2014-01-28 00:15:34,619::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-213::INFO::2014-01-28 00:15:34,620::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00066343', 'lastCheck': '8.8', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000365527', 'lastCheck': '3.6', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000925856', 'lastCheck': '2.4', 'valid': True}}
>Thread-214::INFO::2014-01-28 00:15:35,034::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-214::INFO::2014-01-28 00:15:35,034::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00066343', 'lastCheck': '9.2', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000365527', 'lastCheck': '4.0', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000925856', 'lastCheck': '2.8', 'valid': True}}
>Thread-215::DEBUG::2014-01-28 00:15:35,064::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-215::DEBUG::2014-01-28 00:15:35,064::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>Thread-215::DEBUG::2014-01-28 00:15:35,065::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-215::DEBUG::2014-01-28 00:15:35,065::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-215::DEBUG::2014-01-28 00:15:35,065::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>Thread-215::DEBUG::2014-01-28 00:15:35,065::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-216::DEBUG::2014-01-28 00:15:35,597::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {}
>Thread-216::DEBUG::2014-01-28 00:15:35,638::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-217::DEBUG::2014-01-28 00:15:35,683::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call getCapabilities with () {}
>Thread-217::DEBUG::2014-01-28 00:15:35,687::utils::555::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-217::DEBUG::2014-01-28 00:15:35,710::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-217::DEBUG::2014-01-28 00:15:35,731::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:15:35,751::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-217::DEBUG::2014-01-28 00:15:35,838::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-12::DEBUG::2014-01-28 00:15:35,861::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-218::DEBUG::2014-01-28 00:15:36,006::BindingXMLRPC::159::vds::(wrapper) client [127.0.0.1]
>Thread-218::INFO::2014-01-28 00:15:36,040::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='7e0cc075-eee7-4268-946e-33a79dd08748', conList=[{'port': '', 'connection': '172.16.1.5:/HOSTED-ENGINE', 'iqn': '', 'portal': '', 'user': 'kvm', 'protocol_version': '3', 'password': '******', 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}], options=None)
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:36,199::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-218::DEBUG::2014-01-28 00:15:36,207::hsm::2273::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:36,212::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:36,226::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-218::DEBUG::2014-01-28 00:15:36,235::hsm::2297::Storage.HSM::(__prefetchDomains) Found SD uuids: ('562f0160-7b80-42e2-b248-5754455c40fc',)
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:36,238::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-218::DEBUG::2014-01-28 00:15:36,246::hsm::2353::Storage.HSM::(connectStorageServer) knownSDs: {562f0160-7b80-42e2-b248-5754455c40fc: storage.nfsSD.findDomain, 436c344c-bc57-441b-b311-c9595c6039e1: storage.nfsSD.findDomain, ff3a0c94-11dc-446d-b093-5f68ad81520d: storage.glusterSD.findDomain}
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:36,249::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>Thread-218::INFO::2014-01-28 00:15:36,256::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}]}
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:36,259::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:36,285::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:36,299::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:36,312::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:36,323::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:36,326::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:36,328::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-17::DEBUG::2014-01-28 00:15:36,343::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-217::DEBUG::2014-01-28 00:15:36,371::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.2' (cwd None)
>Thread-217::DEBUG::2014-01-28 00:15:36,409::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-217::DEBUG::2014-01-28 00:15:36,416::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.2' (cwd None)
>Thread-17::DEBUG::2014-01-28 00:15:36,440::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n478 bytes (478 B) copied, 0.000677501 s, 706 kB/s\n'; <rc> = 0
>Thread-17::DEBUG::2014-01-28 00:15:36,449::iscsi::393::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-217::DEBUG::2014-01-28 00:15:36,452::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-17::DEBUG::2014-01-28 00:15:36,454::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-217::DEBUG::2014-01-28 00:15:36,484::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.3' (cwd None)
>Thread-17::DEBUG::2014-01-28 00:15:36,522::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-217::DEBUG::2014-01-28 00:15:36,551::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-217::DEBUG::2014-01-28 00:15:36,552::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.3' (cwd None)
>Thread-217::DEBUG::2014-01-28 00:15:36,574::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-217::DEBUG::2014-01-28 00:15:36,583::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.10' (cwd None)
>Thread-217::DEBUG::2014-01-28 00:15:36,605::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-217::DEBUG::2014-01-28 00:15:36,606::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.10' (cwd None)
>Thread-219::DEBUG::2014-01-28 00:15:36,627::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.11]
>Thread-219::INFO::2014-01-28 00:15:36,637::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-217::DEBUG::2014-01-28 00:15:36,640::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-219::INFO::2014-01-28 00:15:36,642::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00066343', 'lastCheck': '10.8', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000365527', 'lastCheck': '5.6', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000925856', 'lastCheck': '4.4', 'valid': True}}
>Thread-17::DEBUG::2014-01-28 00:15:36,674::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-217::DEBUG::2014-01-28 00:15:36,695::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.11' (cwd None)
>Thread-217::DEBUG::2014-01-28 00:15:36,746::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-217::DEBUG::2014-01-28 00:15:36,754::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.11' (cwd None)
>Thread-217::DEBUG::2014-01-28 00:15:36,785::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-17::DEBUG::2014-01-28 00:15:36,813::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc
>Thread-17::DEBUG::2014-01-28 00:15:36,822::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-217::DEBUG::2014-01-28 00:15:36,832::caps::453::root::(_getKeyPackages) rpm package glusterfs-rdma not found
>Thread-217::DEBUG::2014-01-28 00:15:36,833::caps::453::root::(_getKeyPackages) rpm package glusterfs-fuse not found
>Thread-217::DEBUG::2014-01-28 00:15:36,834::caps::453::root::(_getKeyPackages) rpm package gluster-swift not found
>Thread-217::DEBUG::2014-01-28 00:15:36,841::caps::453::root::(_getKeyPackages) rpm package gluster-swift-object not found
>Thread-217::DEBUG::2014-01-28 00:15:36,844::caps::453::root::(_getKeyPackages) rpm package glusterfs not found
>Thread-17::DEBUG::2014-01-28 00:15:36,847::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=562f0160-7b80-42e2-b248-5754455c40fc:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=7e0cc075-eee7-4268-946e-33a79dd08748', 'REMOTE_PATH=172.16.1.5:/HOSTED-ENGINE', 'ROLE=Master', 'SDUUID=562f0160-7b80-42e2-b248-5754455c40fc', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=14825cb0b944c5c2ae9ed025c82e06730a2d91c4']
>Thread-217::DEBUG::2014-01-28 00:15:36,848::caps::453::root::(_getKeyPackages) rpm package gluster-swift-plugin not found
>Thread-217::DEBUG::2014-01-28 00:15:36,859::caps::453::root::(_getKeyPackages) rpm package gluster-swift-account not found
>Thread-217::DEBUG::2014-01-28 00:15:36,863::caps::453::root::(_getKeyPackages) rpm package gluster-swift-proxy not found
>Thread-217::DEBUG::2014-01-28 00:15:36,865::caps::453::root::(_getKeyPackages) rpm package gluster-swift-doc not found
>Thread-217::DEBUG::2014-01-28 00:15:36,868::caps::453::root::(_getKeyPackages) rpm package glusterfs-server not found
>Thread-217::DEBUG::2014-01-28 00:15:36,869::caps::453::root::(_getKeyPackages) rpm package gluster-swift-container not found
>Thread-217::DEBUG::2014-01-28 00:15:36,871::caps::453::root::(_getKeyPackages) rpm package glusterfs-geo-replication not found
>Thread-17::DEBUG::2014-01-28 00:15:36,871::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-17::INFO::2014-01-28 00:15:36,873::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_imageNS already registered
>Thread-17::INFO::2014-01-28 00:15:36,874::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_volumeNS already registered
>Thread-217::DEBUG::2014-01-28 00:15:36,882::BindingXMLRPC::976::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.3.1.el6.x86_64', 'buildtime': 1388745567.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '33.gitf029545.el6', 'buildtime': 1390094004L, 'version': '4.14.0'}, 'qemu-kvm': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.2', 'buildtime': 1387360004L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'mom': {'release': '20140117.gitfd877c5.el6', 'buildtime': 1389966957L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           L5320  @ 1.86GHz', 'hooks': {'before_vm_start': {'50_hostedengine': {'md5': '45dde62155b5412eafbfff5ef265acc2'}}}, 'cpuSockets': '2', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'gateway': '172.16.0.1', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.2', 'vnet0']}, 'os_net': {'iface': 'os_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.10']}, 'r2o_net': {'iface': 'r2o_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}, 'storage_network': {'iface': u'bond0.3', 'addr': '172.16.1.5', 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000', 'netmask': '255.255.255.255', 'bridged': False, 'interface': u'bond0.3', 'ipv6gateway': '::', 'gateway': ''}}, 'bridges': {'ovirtmgmt': {'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '172.16.0.1', 'ports': ['bond0.2', 'vnet0']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}, 'os_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.10']}, 'r2o_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}}, 'uuid': '44454C4C-3100-104D-8046-B9C04F583153', 'lastClientIface': 'ovirtmgmt', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'cb327caa-ca77-4efe-b7b8-63aba3cda84d', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:85', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth2', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:85', 'speed': 1000, 'mtu': '4000'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'e401a6a5-8773-4971-9213-b3c2bb52f16c', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:83', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:83', 'speed': 1000, 'mtu': '4000'}, 'eth0': {'netmask': '', 'addr': '', 'hwaddr': '00:e0:4c:80:13:02', 'cfg': {'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': '00:e0:4c:80:13:02', 'STP': 'no', 'DEVICE': 'eth0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'speed': 1000, 'mtu': '1500'}}, 'software_revision': '33', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,tm2,ssse3,cx16,xtpr,pdcm,dca,lahf_lm,dts,tpr_shadow,vnmi,flexpriority,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': ['eth1', 'eth2'], 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'MTU': '4000', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'BONDING_OPTS': 'miimon=100 mode=balance-rr', 'DEVICE': 'bond0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '32105', 'cpuSpeed': '1861.936', 'version_name': 'Snow Man', 'vlans': {'bond0.2': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'ovirtmgmt', 'VLAN': 'yes', 'MTU': '4000', 'BOOTPROTO': 'none', 'DEVICE': 'bond0.2', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 2, 'mtu': '4000'}, 'bond0.3': {'netmask': '255.255.255.255', 'iface': 'bond0', 'addr': '172.16.1.5', 'cfg': {'VLAN': 'yes', 'DEFROUTE': 'no', 'MTU': '4000', 'NM_CONTROLLED': 'no', 'DEVICE': 'bond0.3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 3, 'mtu': '4000'}, 'eth0.11': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'r2o_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.11', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 11, 'mtu': '1500'}, 'eth0.10': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'os_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.10', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 10, 'mtu': '1500'}}, 'cpuCores': '8', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '8', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '172.16.0.11'}}
>Thread-12::DEBUG::2014-01-28 00:15:38,136::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:15:38,159::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-71::INFO::2014-01-28 00:15:38,483::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='562f0160-7b80-42e2-b248-5754455c40fc', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='daa7fa08-1354-4b19-a80f-8694ea63f914', volUUID='8d9c27b1-8b26-4a44-8be2-79f024a56e0f', options=None)
>Thread-71::INFO::2014-01-28 00:15:38,486::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '3920601088', 'apparentsize': '26843545600'}
>Thread-224::DEBUG::2014-01-28 00:15:38,679::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call vmGetStats with ('2d425c4c-91b1-4564-b276-f27cc581e6a1',) {}
>Thread-224::DEBUG::2014-01-28 00:15:38,679::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-224::DEBUG::2014-01-28 00:15:38,680::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>Thread-224::DEBUG::2014-01-28 00:15:38,680::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-224::DEBUG::2014-01-28 00:15:38,680::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-224::DEBUG::2014-01-28 00:15:38,681::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>Thread-224::DEBUG::2014-01-28 00:15:38,681::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-224::DEBUG::2014-01-28 00:15:38,681::BindingXMLRPC::976::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'root', 'memUsage': '53', 'acpiEnable': 'true', 'guestFQDN': u'engine.melb.example.net', 'pid': '18113', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'/', u'total': '25323499520', u'used': '2592231424', u'fs': u'ext4'}, {u'path': u'/boot', u'total': '507744256', u'used': '55643136', u'fs': u'ext4'}], 'network': {u'vnet0': {'macAddr': '00:16:3e:0a:b7:55', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmType': 'kvm', 'guestName': u'engine.melb.example.net', 'elapsedTime': '610923', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'mem_free': '1932888', u'swap_in': '0', u'pageflt': '144', u'mem_total': '3922728', u'mem_unused': '1398284'}, 'cpuSys': '0.86', 'appsList': [u'ovirt-guest-agent-1.0.8-1.el6', u'kernel-2.6.32-431.el6', u'kernel-2.6.32-431.3.1.el6'], 'guestOs': u'2.6.32-431.3.1.el6.x86_64', 'displayType': 'vnc', 'status': 'Up', 'hash': '2950633987041661405', 'vmId': '2d425c4c-91b1-4564-b276-f27cc581e6a1', 'displayIp': '0', 'displayPort': u'5900', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'truesize': '3920601088', 'apparentsize': '26843545600', 'imageID': 'daa7fa08-1354-4b19-a80f-8694ea63f914'}, 'hdc': {'truesize': '0', 'apparentsize': '0'}}, 'monitorResponse': '0', 'statsAge': '0.20', 'cpuUser': '10.07', 'lastLogin': 1390828479.279238, 'guestIPs': u'172.16.0.10', 'netIfaces': [{u'hw': u'00:16:3e:0a:b7:55', u'name': u'eth0', u'inet': [u'172.16.0.10'], u'inet6': [u'fe80::216:3eff:fe0a:b755']}]}]}
>Thread-12::DEBUG::2014-01-28 00:15:40,181::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:15:40,205::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-225::DEBUG::2014-01-28 00:15:40,756::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {} flowID [4c514a31]
>Thread-225::DEBUG::2014-01-28 00:15:40,810::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-129::DEBUG::2014-01-28 00:15:41,059::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:15:41,080::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n364 bytes (364 B) copied, 0.000338565 s, 1.1 MB/s\n'; <rc> = 0
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:41,339::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:41,340::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:41,340::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:41,340::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:41,341::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:41,341::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:41,342::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:41,342::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:41,342::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:41,343::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:41,343::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:41,343::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-12::DEBUG::2014-01-28 00:15:42,221::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:15:42,245::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-128::DEBUG::2014-01-28 00:15:42,307::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-128::DEBUG::2014-01-28 00:15:42,328::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n504 bytes (504 B) copied, 0.000786485 s, 641 kB/s\n'; <rc> = 0
>Thread-227::DEBUG::2014-01-28 00:15:43,258::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10] flowID [5108ae22]
>Thread-227::INFO::2014-01-28 00:15:43,260::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-227::INFO::2014-01-28 00:15:43,260::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 2}}
>Thread-228::DEBUG::2014-01-28 00:15:43,267::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10] flowID [5108ae22]
>Thread-228::INFO::2014-01-28 00:15:43,268::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-228::INFO::2014-01-28 00:15:43,275::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': 'ff3a0c94-11dc-446d-b093-5f68ad81520d', 'name': 'NextDC_M1', 'version': '3', 'domains': 'ff3a0c94-11dc-446d-b093-5f68ad81520d:Active,436c344c-bc57-441b-b311-c9595c6039e1:Active', 'pool_status': 'connected', 'isoprefix': '/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/images/11111111-1111-1111-1111-111111111111', 'type': 'GLUSTERFS', 'master_ver': 1, 'lver': 2}, 'dominfo': {'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'status': 'Active', 'diskfree': '917084569600', 'isoprefix': '', 'alerts': [], 'disktotal': '982907879424', 'version': 3}, '436c344c-bc57-441b-b311-c9595c6039e1': {'status': 'Active', 'diskfree': '21444952064', 'isoprefix': '/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '25323634688', 'version': 0}}}
>Thread-12::DEBUG::2014-01-28 00:15:44,290::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:15:44,346::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-230::INFO::2014-01-28 00:15:44,627::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-230::INFO::2014-01-28 00:15:44,628::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000677501', 'lastCheck': '7.7', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000338565', 'lastCheck': '3.5', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000786485', 'lastCheck': '2.3', 'valid': True}}
>Thread-231::DEBUG::2014-01-28 00:15:45,683::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call getCapabilities with () {}
>Thread-231::DEBUG::2014-01-28 00:15:45,685::utils::555::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-231::DEBUG::2014-01-28 00:15:45,704::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-231::DEBUG::2014-01-28 00:15:45,733::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-231::DEBUG::2014-01-28 00:15:45,756::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-231::DEBUG::2014-01-28 00:15:45,775::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.2' (cwd None)
>Thread-231::DEBUG::2014-01-28 00:15:45,795::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-231::DEBUG::2014-01-28 00:15:45,796::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.2' (cwd None)
>Thread-231::DEBUG::2014-01-28 00:15:45,816::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-231::DEBUG::2014-01-28 00:15:45,819::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.3' (cwd None)
>Thread-231::DEBUG::2014-01-28 00:15:45,839::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-231::DEBUG::2014-01-28 00:15:45,840::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.3' (cwd None)
>Thread-231::DEBUG::2014-01-28 00:15:45,859::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-231::DEBUG::2014-01-28 00:15:45,862::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.10' (cwd None)
>Thread-231::DEBUG::2014-01-28 00:15:45,882::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-231::DEBUG::2014-01-28 00:15:45,883::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.10' (cwd None)
>Thread-231::DEBUG::2014-01-28 00:15:45,902::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-231::DEBUG::2014-01-28 00:15:45,906::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.11' (cwd None)
>Thread-231::DEBUG::2014-01-28 00:15:45,927::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-231::DEBUG::2014-01-28 00:15:45,928::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.11' (cwd None)
>Thread-231::DEBUG::2014-01-28 00:15:45,947::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-231::DEBUG::2014-01-28 00:15:45,958::caps::453::root::(_getKeyPackages) rpm package glusterfs-rdma not found
>Thread-231::DEBUG::2014-01-28 00:15:45,959::caps::453::root::(_getKeyPackages) rpm package glusterfs-fuse not found
>Thread-231::DEBUG::2014-01-28 00:15:45,960::caps::453::root::(_getKeyPackages) rpm package gluster-swift not found
>Thread-231::DEBUG::2014-01-28 00:15:45,966::caps::453::root::(_getKeyPackages) rpm package gluster-swift-object not found
>Thread-231::DEBUG::2014-01-28 00:15:45,967::caps::453::root::(_getKeyPackages) rpm package glusterfs not found
>Thread-231::DEBUG::2014-01-28 00:15:45,967::caps::453::root::(_getKeyPackages) rpm package gluster-swift-plugin not found
>Thread-231::DEBUG::2014-01-28 00:15:45,974::caps::453::root::(_getKeyPackages) rpm package gluster-swift-account not found
>Thread-231::DEBUG::2014-01-28 00:15:45,975::caps::453::root::(_getKeyPackages) rpm package gluster-swift-proxy not found
>Thread-231::DEBUG::2014-01-28 00:15:45,975::caps::453::root::(_getKeyPackages) rpm package gluster-swift-doc not found
>Thread-231::DEBUG::2014-01-28 00:15:45,976::caps::453::root::(_getKeyPackages) rpm package glusterfs-server not found
>Thread-231::DEBUG::2014-01-28 00:15:45,976::caps::453::root::(_getKeyPackages) rpm package gluster-swift-container not found
>Thread-231::DEBUG::2014-01-28 00:15:45,977::caps::453::root::(_getKeyPackages) rpm package glusterfs-geo-replication not found
>Thread-231::DEBUG::2014-01-28 00:15:45,982::BindingXMLRPC::976::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.3.1.el6.x86_64', 'buildtime': 1388745567.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '33.gitf029545.el6', 'buildtime': 1390094004L, 'version': '4.14.0'}, 'qemu-kvm': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.2', 'buildtime': 1387360004L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'mom': {'release': '20140117.gitfd877c5.el6', 'buildtime': 1389966957L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           L5320  @ 1.86GHz', 'hooks': {'before_vm_start': {'50_hostedengine': {'md5': '45dde62155b5412eafbfff5ef265acc2'}}}, 'cpuSockets': '2', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'gateway': '172.16.0.1', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.2', 'vnet0']}, 'os_net': {'iface': 'os_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.10']}, 'r2o_net': {'iface': 'r2o_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}, 'storage_network': {'iface': u'bond0.3', 'addr': '172.16.1.5', 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000', 'netmask': '255.255.255.255', 'bridged': False, 'interface': u'bond0.3', 'ipv6gateway': '::', 'gateway': ''}}, 'bridges': {'ovirtmgmt': {'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '172.16.0.1', 'ports': ['bond0.2', 'vnet0']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}, 'os_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.10']}, 'r2o_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}}, 'uuid': '44454C4C-3100-104D-8046-B9C04F583153', 'lastClientIface': 'ovirtmgmt', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'cb327caa-ca77-4efe-b7b8-63aba3cda84d', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:85', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth2', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:85', 'speed': 1000, 'mtu': '4000'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'e401a6a5-8773-4971-9213-b3c2bb52f16c', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:83', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:83', 'speed': 1000, 'mtu': '4000'}, 'eth0': {'netmask': '', 'addr': '', 'hwaddr': '00:e0:4c:80:13:02', 'cfg': {'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': '00:e0:4c:80:13:02', 'STP': 'no', 'DEVICE': 'eth0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'speed': 1000, 'mtu': '1500'}}, 'software_revision': '33', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,tm2,ssse3,cx16,xtpr,pdcm,dca,lahf_lm,dts,tpr_shadow,vnmi,flexpriority,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': ['eth1', 'eth2'], 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'MTU': '4000', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'BONDING_OPTS': 'miimon=100 mode=balance-rr', 'DEVICE': 'bond0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '32105', 'cpuSpeed': '1861.936', 'version_name': 'Snow Man', 'vlans': {'bond0.2': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'ovirtmgmt', 'VLAN': 'yes', 'MTU': '4000', 'BOOTPROTO': 'none', 'DEVICE': 'bond0.2', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 2, 'mtu': '4000'}, 'bond0.3': {'netmask': '255.255.255.255', 'iface': 'bond0', 'addr': '172.16.1.5', 'cfg': {'VLAN': 'yes', 'DEFROUTE': 'no', 'MTU': '4000', 'NM_CONTROLLED': 'no', 'DEVICE': 'bond0.3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 3, 'mtu': '4000'}, 'eth0.11': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'r2o_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.11', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 11, 'mtu': '1500'}, 'eth0.10': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'os_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.10', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 10, 'mtu': '1500'}}, 'cpuCores': '8', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '8', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '172.16.0.11'}}
>Thread-232::DEBUG::2014-01-28 00:15:45,997::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {}
>Thread-232::DEBUG::2014-01-28 00:15:46,036::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:46,349::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:46,349::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:46,349::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:46,350::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:46,350::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:46,350::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:46,351::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:46,351::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:46,352::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:46,352::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:46,352::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:46,352::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-12::DEBUG::2014-01-28 00:15:46,361::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:15:46,383::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-17::DEBUG::2014-01-28 00:15:46,931::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-17::DEBUG::2014-01-28 00:15:46,959::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n478 bytes (478 B) copied, 0.000544155 s, 878 kB/s\n'; <rc> = 0
>Thread-233::DEBUG::2014-01-28 00:15:47,047::BindingXMLRPC::159::vds::(wrapper) client [127.0.0.1]
>Thread-233::INFO::2014-01-28 00:15:47,048::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='7e0cc075-eee7-4268-946e-33a79dd08748', conList=[{'port': '', 'connection': '172.16.1.5:/HOSTED-ENGINE', 'iqn': '', 'portal': '', 'user': 'kvm', 'protocol_version': '3', 'password': '******', 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}], options=None)
>Thread-233::DEBUG::2014-01-28 00:15:47,053::hsm::2273::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE
>Thread-233::DEBUG::2014-01-28 00:15:47,055::hsm::2297::Storage.HSM::(__prefetchDomains) Found SD uuids: ('562f0160-7b80-42e2-b248-5754455c40fc',)
>Thread-233::DEBUG::2014-01-28 00:15:47,055::hsm::2353::Storage.HSM::(connectStorageServer) knownSDs: {562f0160-7b80-42e2-b248-5754455c40fc: storage.nfsSD.findDomain, 436c344c-bc57-441b-b311-c9595c6039e1: storage.nfsSD.findDomain, ff3a0c94-11dc-446d-b093-5f68ad81520d: storage.glusterSD.findDomain}
>Thread-233::INFO::2014-01-28 00:15:47,055::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}]}
>Thread-234::DEBUG::2014-01-28 00:15:47,303::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.11]
>Thread-234::INFO::2014-01-28 00:15:47,304::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-234::INFO::2014-01-28 00:15:47,304::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000544155', 'lastCheck': '0.3', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000338565', 'lastCheck': '6.2', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000786485', 'lastCheck': '5.0', 'valid': True}}
>Thread-12::DEBUG::2014-01-28 00:15:48,399::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:15:48,424::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-236::DEBUG::2014-01-28 00:15:48,681::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call vmGetStats with ('2d425c4c-91b1-4564-b276-f27cc581e6a1',) {}
>Thread-236::DEBUG::2014-01-28 00:15:48,682::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-236::DEBUG::2014-01-28 00:15:48,682::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>Thread-236::DEBUG::2014-01-28 00:15:48,682::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-236::DEBUG::2014-01-28 00:15:48,683::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-236::DEBUG::2014-01-28 00:15:48,683::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>Thread-236::DEBUG::2014-01-28 00:15:48,683::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-236::DEBUG::2014-01-28 00:15:48,684::BindingXMLRPC::976::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'root', 'memUsage': '53', 'acpiEnable': 'true', 'guestFQDN': u'engine.melb.example.net', 'pid': '18113', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'/', u'total': '25323499520', u'used': '2592231424', u'fs': u'ext4'}, {u'path': u'/boot', u'total': '507744256', u'used': '55643136', u'fs': u'ext4'}], 'network': {u'vnet0': {'macAddr': '00:16:3e:0a:b7:55', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmType': 'kvm', 'guestName': u'engine.melb.example.net', 'elapsedTime': '610933', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'mem_free': '1932900', u'swap_in': '0', u'pageflt': '151', u'mem_total': '3922728', u'mem_unused': '1398284'}, 'cpuSys': '0.86', 'appsList': [u'ovirt-guest-agent-1.0.8-1.el6', u'kernel-2.6.32-431.el6', u'kernel-2.6.32-431.3.1.el6'], 'guestOs': u'2.6.32-431.3.1.el6.x86_64', 'displayType': 'vnc', 'status': 'Up', 'hash': '2950633987041661405', 'vmId': '2d425c4c-91b1-4564-b276-f27cc581e6a1', 'displayIp': '0', 'displayPort': u'5900', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'truesize': '3920601088', 'apparentsize': '26843545600', 'imageID': 'daa7fa08-1354-4b19-a80f-8694ea63f914'}, 'hdc': {'truesize': '0', 'apparentsize': '0'}}, 'monitorResponse': '0', 'statsAge': '0.08', 'cpuUser': '10.07', 'lastLogin': 1390828479.279238, 'guestIPs': u'172.16.0.10', 'netIfaces': [{u'hw': u'00:16:3e:0a:b7:55', u'name': u'eth0', u'inet': [u'172.16.0.10'], u'inet6': [u'fe80::216:3eff:fe0a:b755']}]}]}
>Thread-237::INFO::2014-01-28 00:15:50,427::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-237::INFO::2014-01-28 00:15:50,429::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000544155', 'lastCheck': '3.5', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000338565', 'lastCheck': '9.3', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000786485', 'lastCheck': '8.1', 'valid': True}}
>Thread-12::DEBUG::2014-01-28 00:15:50,441::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:15:50,479::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-238::DEBUG::2014-01-28 00:15:50,550::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-238::DEBUG::2014-01-28 00:15:50,550::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>Thread-238::DEBUG::2014-01-28 00:15:50,550::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-238::DEBUG::2014-01-28 00:15:50,551::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-238::DEBUG::2014-01-28 00:15:50,551::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>Thread-238::DEBUG::2014-01-28 00:15:50,551::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-129::DEBUG::2014-01-28 00:15:51,099::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:15:51,124::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n364 bytes (364 B) copied, 0.00133274 s, 273 kB/s\n'; <rc> = 0
>Thread-239::DEBUG::2014-01-28 00:15:51,140::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {}
>Thread-239::DEBUG::2014-01-28 00:15:51,181::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:51,359::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:51,359::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:51,360::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:51,360::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:51,360::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:51,361::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:51,361::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:51,362::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:51,362::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:51,362::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:51,362::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:51,363::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-128::DEBUG::2014-01-28 00:15:52,360::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-128::DEBUG::2014-01-28 00:15:52,383::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n504 bytes (504 B) copied, 0.000880437 s, 572 kB/s\n'; <rc> = 0
>Thread-12::DEBUG::2014-01-28 00:15:52,540::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:15:52,563::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-240::DEBUG::2014-01-28 00:15:53,320::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-240::INFO::2014-01-28 00:15:53,322::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-240::INFO::2014-01-28 00:15:53,325::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 2}}
>Thread-241::DEBUG::2014-01-28 00:15:53,331::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-241::INFO::2014-01-28 00:15:53,332::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-241::INFO::2014-01-28 00:15:53,338::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': 'ff3a0c94-11dc-446d-b093-5f68ad81520d', 'name': 'NextDC_M1', 'version': '3', 'domains': 'ff3a0c94-11dc-446d-b093-5f68ad81520d:Active,436c344c-bc57-441b-b311-c9595c6039e1:Active', 'pool_status': 'connected', 'isoprefix': '/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/images/11111111-1111-1111-1111-111111111111', 'type': 'GLUSTERFS', 'master_ver': 1, 'lver': 2}, 'dominfo': {'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'status': 'Active', 'diskfree': '917084569600', 'isoprefix': '', 'alerts': [], 'disktotal': '982907879424', 'version': 3}, '436c344c-bc57-441b-b311-c9595c6039e1': {'status': 'Active', 'diskfree': '21444952064', 'isoprefix': '/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '25323634688', 'version': 0}}}
>Thread-12::DEBUG::2014-01-28 00:15:54,578::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:15:54,603::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-243::INFO::2014-01-28 00:15:54,656::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-243::INFO::2014-01-28 00:15:54,656::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000544155', 'lastCheck': '7.7', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00133274', 'lastCheck': '3.5', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000880437', 'lastCheck': '2.3', 'valid': True}}
>Thread-244::DEBUG::2014-01-28 00:15:55,692::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call getCapabilities with () {}
>Thread-244::DEBUG::2014-01-28 00:15:55,693::utils::555::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-244::DEBUG::2014-01-28 00:15:55,714::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-244::DEBUG::2014-01-28 00:15:55,733::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-244::DEBUG::2014-01-28 00:15:55,755::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-244::DEBUG::2014-01-28 00:15:55,774::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.2' (cwd None)
>Thread-244::DEBUG::2014-01-28 00:15:55,794::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-244::DEBUG::2014-01-28 00:15:55,795::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.2' (cwd None)
>Thread-244::DEBUG::2014-01-28 00:15:55,815::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-244::DEBUG::2014-01-28 00:15:55,818::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.3' (cwd None)
>Thread-244::DEBUG::2014-01-28 00:15:55,837::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-244::DEBUG::2014-01-28 00:15:55,838::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.3' (cwd None)
>Thread-244::DEBUG::2014-01-28 00:15:55,857::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-244::DEBUG::2014-01-28 00:15:55,860::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.10' (cwd None)
>Thread-244::DEBUG::2014-01-28 00:15:55,880::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-244::DEBUG::2014-01-28 00:15:55,880::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.10' (cwd None)
>Thread-244::DEBUG::2014-01-28 00:15:55,900::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-244::DEBUG::2014-01-28 00:15:55,904::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.11' (cwd None)
>Thread-244::DEBUG::2014-01-28 00:15:55,925::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-244::DEBUG::2014-01-28 00:15:55,926::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.11' (cwd None)
>Thread-244::DEBUG::2014-01-28 00:15:55,946::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-244::DEBUG::2014-01-28 00:15:55,955::caps::453::root::(_getKeyPackages) rpm package glusterfs-rdma not found
>Thread-244::DEBUG::2014-01-28 00:15:55,956::caps::453::root::(_getKeyPackages) rpm package glusterfs-fuse not found
>Thread-244::DEBUG::2014-01-28 00:15:55,957::caps::453::root::(_getKeyPackages) rpm package gluster-swift not found
>Thread-244::DEBUG::2014-01-28 00:15:55,963::caps::453::root::(_getKeyPackages) rpm package gluster-swift-object not found
>Thread-244::DEBUG::2014-01-28 00:15:55,963::caps::453::root::(_getKeyPackages) rpm package glusterfs not found
>Thread-244::DEBUG::2014-01-28 00:15:55,964::caps::453::root::(_getKeyPackages) rpm package gluster-swift-plugin not found
>Thread-244::DEBUG::2014-01-28 00:15:55,971::caps::453::root::(_getKeyPackages) rpm package gluster-swift-account not found
>Thread-244::DEBUG::2014-01-28 00:15:55,971::caps::453::root::(_getKeyPackages) rpm package gluster-swift-proxy not found
>Thread-244::DEBUG::2014-01-28 00:15:55,972::caps::453::root::(_getKeyPackages) rpm package gluster-swift-doc not found
>Thread-244::DEBUG::2014-01-28 00:15:55,972::caps::453::root::(_getKeyPackages) rpm package glusterfs-server not found
>Thread-244::DEBUG::2014-01-28 00:15:55,973::caps::453::root::(_getKeyPackages) rpm package gluster-swift-container not found
>Thread-244::DEBUG::2014-01-28 00:15:55,973::caps::453::root::(_getKeyPackages) rpm package glusterfs-geo-replication not found
>Thread-244::DEBUG::2014-01-28 00:15:55,978::BindingXMLRPC::976::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.3.1.el6.x86_64', 'buildtime': 1388745567.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '33.gitf029545.el6', 'buildtime': 1390094004L, 'version': '4.14.0'}, 'qemu-kvm': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.2', 'buildtime': 1387360004L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'mom': {'release': '20140117.gitfd877c5.el6', 'buildtime': 1389966957L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           L5320  @ 1.86GHz', 'hooks': {'before_vm_start': {'50_hostedengine': {'md5': '45dde62155b5412eafbfff5ef265acc2'}}}, 'cpuSockets': '2', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'gateway': '172.16.0.1', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.2', 'vnet0']}, 'os_net': {'iface': 'os_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.10']}, 'r2o_net': {'iface': 'r2o_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}, 'storage_network': {'iface': u'bond0.3', 'addr': '172.16.1.5', 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000', 'netmask': '255.255.255.255', 'bridged': False, 'interface': u'bond0.3', 'ipv6gateway': '::', 'gateway': ''}}, 'bridges': {'ovirtmgmt': {'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '172.16.0.1', 'ports': ['bond0.2', 'vnet0']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}, 'os_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.10']}, 'r2o_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}}, 'uuid': '44454C4C-3100-104D-8046-B9C04F583153', 'lastClientIface': 'ovirtmgmt', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'cb327caa-ca77-4efe-b7b8-63aba3cda84d', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:85', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth2', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:85', 'speed': 1000, 'mtu': '4000'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'e401a6a5-8773-4971-9213-b3c2bb52f16c', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:83', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:83', 'speed': 1000, 'mtu': '4000'}, 'eth0': {'netmask': '', 'addr': '', 'hwaddr': '00:e0:4c:80:13:02', 'cfg': {'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': '00:e0:4c:80:13:02', 'STP': 'no', 'DEVICE': 'eth0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'speed': 1000, 'mtu': '1500'}}, 'software_revision': '33', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,tm2,ssse3,cx16,xtpr,pdcm,dca,lahf_lm,dts,tpr_shadow,vnmi,flexpriority,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': ['eth1', 'eth2'], 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'MTU': '4000', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'BONDING_OPTS': 'miimon=100 mode=balance-rr', 'DEVICE': 'bond0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '32105', 'cpuSpeed': '1861.936', 'version_name': 'Snow Man', 'vlans': {'bond0.2': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'ovirtmgmt', 'VLAN': 'yes', 'MTU': '4000', 'BOOTPROTO': 'none', 'DEVICE': 'bond0.2', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 2, 'mtu': '4000'}, 'bond0.3': {'netmask': '255.255.255.255', 'iface': 'bond0', 'addr': '172.16.1.5', 'cfg': {'VLAN': 'yes', 'DEFROUTE': 'no', 'MTU': '4000', 'NM_CONTROLLED': 'no', 'DEVICE': 'bond0.3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 3, 'mtu': '4000'}, 'eth0.11': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'r2o_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.11', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 11, 'mtu': '1500'}, 'eth0.10': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'os_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.10', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 10, 'mtu': '1500'}}, 'cpuCores': '8', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '8', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '172.16.0.11'}}
>Thread-245::DEBUG::2014-01-28 00:15:56,285::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {}
>Thread-245::DEBUG::2014-01-28 00:15:56,325::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:56,366::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:56,366::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:56,367::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:56,367::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:56,367::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:56,368::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:56,368::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:56,369::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:56,369::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:56,369::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:56,369::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:15:56,370::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-12::DEBUG::2014-01-28 00:15:56,627::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:15:56,649::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-17::DEBUG::2014-01-28 00:15:56,967::iscsi::393::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-17::DEBUG::2014-01-28 00:15:56,967::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-17::DEBUG::2014-01-28 00:15:57,010::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-17::DEBUG::2014-01-28 00:15:57,150::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-17::DEBUG::2014-01-28 00:15:57,175::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc
>Thread-17::DEBUG::2014-01-28 00:15:57,177::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-17::DEBUG::2014-01-28 00:15:57,187::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=562f0160-7b80-42e2-b248-5754455c40fc:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=7e0cc075-eee7-4268-946e-33a79dd08748', 'REMOTE_PATH=172.16.1.5:/HOSTED-ENGINE', 'ROLE=Master', 'SDUUID=562f0160-7b80-42e2-b248-5754455c40fc', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=14825cb0b944c5c2ae9ed025c82e06730a2d91c4']
>Thread-17::DEBUG::2014-01-28 00:15:57,189::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-17::INFO::2014-01-28 00:15:57,190::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_imageNS already registered
>Thread-17::INFO::2014-01-28 00:15:57,190::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_volumeNS already registered
>Thread-17::DEBUG::2014-01-28 00:15:57,203::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-17::DEBUG::2014-01-28 00:15:57,225::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n478 bytes (478 B) copied, 0.000678246 s, 705 kB/s\n'; <rc> = 0
>Thread-250::DEBUG::2014-01-28 00:15:57,642::BindingXMLRPC::159::vds::(wrapper) client [127.0.0.1]
>Thread-250::INFO::2014-01-28 00:15:57,652::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='7e0cc075-eee7-4268-946e-33a79dd08748', conList=[{'port': '', 'connection': '172.16.1.5:/HOSTED-ENGINE', 'iqn': '', 'portal': '', 'user': 'kvm', 'protocol_version': '3', 'password': '******', 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}], options=None)
>Thread-250::DEBUG::2014-01-28 00:15:57,656::hsm::2273::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE
>Thread-250::DEBUG::2014-01-28 00:15:57,659::hsm::2297::Storage.HSM::(__prefetchDomains) Found SD uuids: ('562f0160-7b80-42e2-b248-5754455c40fc',)
>Thread-250::DEBUG::2014-01-28 00:15:57,659::hsm::2353::Storage.HSM::(connectStorageServer) knownSDs: {562f0160-7b80-42e2-b248-5754455c40fc: storage.nfsSD.findDomain, 436c344c-bc57-441b-b311-c9595c6039e1: storage.nfsSD.findDomain, ff3a0c94-11dc-446d-b093-5f68ad81520d: storage.glusterSD.findDomain}
>Thread-250::INFO::2014-01-28 00:15:57,659::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}]}
>Thread-251::DEBUG::2014-01-28 00:15:57,910::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.11]
>Thread-251::INFO::2014-01-28 00:15:57,911::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-251::INFO::2014-01-28 00:15:57,912::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000678246', 'lastCheck': '0.7', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00133274', 'lastCheck': '6.8', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000880437', 'lastCheck': '5.5', 'valid': True}}
>Thread-252::DEBUG::2014-01-28 00:15:58,213::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call vmCont with ('9c9befe0-9e17-462e-9bce-27ddd684a6d3',) {} flowID [6aad9c7]
>Thread-252::ERROR::2014-01-28 00:15:58,215::BindingXMLRPC::988::vds::(wrapper) unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/BindingXMLRPC.py", line 972, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/BindingXMLRPC.py", line 209, in vmCont
>    return vm.cont()
>  File "/usr/share/vdsm/API.py", line 154, in cont
>    return v.cont()
>  File "/usr/share/vdsm/vm.py", line 2587, in cont
>    self._underlyingCont()
>  File "/usr/share/vdsm/vm.py", line 3726, in _underlyingCont
>    hooks.before_vm_cont(self._dom.XMLDesc(0), self.conf)
>AttributeError: 'NoneType' object has no attribute 'XMLDesc'
>Thread-12::DEBUG::2014-01-28 00:15:58,664::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:15:58,687::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-253::DEBUG::2014-01-28 00:15:58,692::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call vmGetStats with ('2d425c4c-91b1-4564-b276-f27cc581e6a1',) {}
>Thread-253::DEBUG::2014-01-28 00:15:58,693::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-253::DEBUG::2014-01-28 00:15:58,694::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>Thread-253::DEBUG::2014-01-28 00:15:58,694::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-253::DEBUG::2014-01-28 00:15:58,695::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-253::DEBUG::2014-01-28 00:15:58,698::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>Thread-253::DEBUG::2014-01-28 00:15:58,701::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-253::DEBUG::2014-01-28 00:15:58,706::BindingXMLRPC::976::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'root', 'memUsage': '53', 'acpiEnable': 'true', 'guestFQDN': u'engine.melb.example.net', 'pid': '18113', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'/', u'total': '25323499520', u'used': '2592231424', u'fs': u'ext4'}, {u'path': u'/boot', u'total': '507744256', u'used': '55643136', u'fs': u'ext4'}], 'network': {u'vnet0': {'macAddr': '00:16:3e:0a:b7:55', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmType': 'kvm', 'guestName': u'engine.melb.example.net', 'elapsedTime': '610943', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'mem_free': '1932736', u'swap_in': '0', u'pageflt': '459', u'mem_total': '3922728', u'mem_unused': '1398088'}, 'cpuSys': '1.26', 'appsList': [u'ovirt-guest-agent-1.0.8-1.el6', u'kernel-2.6.32-431.el6', u'kernel-2.6.32-431.3.1.el6'], 'guestOs': u'2.6.32-431.3.1.el6.x86_64', 'displayType': 'vnc', 'status': 'Up', 'hash': '2950633987041661405', 'vmId': '2d425c4c-91b1-4564-b276-f27cc581e6a1', 'displayIp': '0', 'displayPort': u'5900', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'truesize': '3920601088', 'apparentsize': '26843545600', 'imageID': 'daa7fa08-1354-4b19-a80f-8694ea63f914'}, 'hdc': {'truesize': '0', 'apparentsize': '0'}}, 'monitorResponse': '0', 'statsAge': '0.07', 'cpuUser': '10.71', 'lastLogin': 1390828479.279238, 'guestIPs': u'172.16.0.10', 'netIfaces': [{u'hw': u'00:16:3e:0a:b7:55', u'name': u'eth0', u'inet': [u'172.16.0.10'], u'inet6': [u'fe80::216:3eff:fe0a:b755']}]}]}
>Thread-12::DEBUG::2014-01-28 00:16:00,744::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:16:00,766::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-129::DEBUG::2014-01-28 00:16:01,140::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:16:01,162::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n364 bytes (364 B) copied, 0.000330166 s, 1.1 MB/s\n'; <rc> = 0
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:01,376::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:01,377::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:01,377::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:01,378::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:01,378::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:01,378::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:01,379::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:01,379::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:01,380::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:01,380::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:01,380::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:01,381::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-255::DEBUG::2014-01-28 00:16:01,437::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {}
>Thread-255::DEBUG::2014-01-28 00:16:01,491::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-128::DEBUG::2014-01-28 00:16:02,408::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-128::DEBUG::2014-01-28 00:16:02,431::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n504 bytes (504 B) copied, 0.000990767 s, 509 kB/s\n'; <rc> = 0
>Thread-12::DEBUG::2014-01-28 00:16:02,781::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:16:02,804::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-257::DEBUG::2014-01-28 00:16:03,385::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-257::INFO::2014-01-28 00:16:03,387::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-257::INFO::2014-01-28 00:16:03,387::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 2}}
>Thread-258::DEBUG::2014-01-28 00:16:03,395::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-258::INFO::2014-01-28 00:16:03,396::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-258::INFO::2014-01-28 00:16:03,403::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': 'ff3a0c94-11dc-446d-b093-5f68ad81520d', 'name': 'NextDC_M1', 'version': '3', 'domains': 'ff3a0c94-11dc-446d-b093-5f68ad81520d:Active,436c344c-bc57-441b-b311-c9595c6039e1:Active', 'pool_status': 'connected', 'isoprefix': '/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/images/11111111-1111-1111-1111-111111111111', 'type': 'GLUSTERFS', 'master_ver': 1, 'lver': 2}, 'dominfo': {'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'status': 'Active', 'diskfree': '917084569600', 'isoprefix': '', 'alerts': [], 'disktotal': '982907879424', 'version': 3}, '436c344c-bc57-441b-b311-c9595c6039e1': {'status': 'Active', 'diskfree': '21444952064', 'isoprefix': '/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '25323634688', 'version': 0}}}
>Thread-259::INFO::2014-01-28 00:16:04,647::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-259::INFO::2014-01-28 00:16:04,648::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000678246', 'lastCheck': '7.4', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000330166', 'lastCheck': '3.5', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000990767', 'lastCheck': '2.2', 'valid': True}}
>Thread-12::DEBUG::2014-01-28 00:16:04,819::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:16:04,842::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-260::DEBUG::2014-01-28 00:16:05,708::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call getCapabilities with () {}
>Thread-260::DEBUG::2014-01-28 00:16:05,710::utils::555::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-260::DEBUG::2014-01-28 00:16:05,730::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-260::DEBUG::2014-01-28 00:16:05,759::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-260::DEBUG::2014-01-28 00:16:05,781::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-260::DEBUG::2014-01-28 00:16:05,800::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.2' (cwd None)
>Thread-260::DEBUG::2014-01-28 00:16:05,820::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-260::DEBUG::2014-01-28 00:16:05,821::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.2' (cwd None)
>Thread-260::DEBUG::2014-01-28 00:16:05,840::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-260::DEBUG::2014-01-28 00:16:05,843::utils::555::root::(execCmd) '/sbin/ip -d link show dev bond0.3' (cwd None)
>Thread-260::DEBUG::2014-01-28 00:16:05,863::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-260::DEBUG::2014-01-28 00:16:05,863::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev bond0.3' (cwd None)
>Thread-260::DEBUG::2014-01-28 00:16:05,891::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-260::DEBUG::2014-01-28 00:16:05,908::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.10' (cwd None)
>Thread-260::DEBUG::2014-01-28 00:16:05,931::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-260::DEBUG::2014-01-28 00:16:05,932::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.10' (cwd None)
>Thread-261::INFO::2014-01-28 00:16:05,949::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-261::INFO::2014-01-28 00:16:05,956::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000678246', 'lastCheck': '8.7', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000330166', 'lastCheck': '4.8', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000990767', 'lastCheck': '3.5', 'valid': True}}
>Thread-260::DEBUG::2014-01-28 00:16:05,962::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-260::DEBUG::2014-01-28 00:16:05,983::utils::555::root::(execCmd) '/sbin/ip -d link show dev eth0.11' (cwd None)
>Thread-260::DEBUG::2014-01-28 00:16:06,008::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-260::DEBUG::2014-01-28 00:16:06,008::utils::555::root::(execCmd) '/sbin/ip -d -o link show dev eth0.11' (cwd None)
>Thread-260::DEBUG::2014-01-28 00:16:06,047::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-260::DEBUG::2014-01-28 00:16:06,098::caps::453::root::(_getKeyPackages) rpm package glusterfs-rdma not found
>Thread-260::DEBUG::2014-01-28 00:16:06,101::caps::453::root::(_getKeyPackages) rpm package glusterfs-fuse not found
>Thread-260::DEBUG::2014-01-28 00:16:06,105::caps::453::root::(_getKeyPackages) rpm package gluster-swift not found
>Thread-260::DEBUG::2014-01-28 00:16:06,115::caps::453::root::(_getKeyPackages) rpm package gluster-swift-object not found
>Thread-260::DEBUG::2014-01-28 00:16:06,118::caps::453::root::(_getKeyPackages) rpm package glusterfs not found
>Thread-260::DEBUG::2014-01-28 00:16:06,121::caps::453::root::(_getKeyPackages) rpm package gluster-swift-plugin not found
>Thread-260::DEBUG::2014-01-28 00:16:06,131::caps::453::root::(_getKeyPackages) rpm package gluster-swift-account not found
>Thread-260::DEBUG::2014-01-28 00:16:06,134::caps::453::root::(_getKeyPackages) rpm package gluster-swift-proxy not found
>Thread-260::DEBUG::2014-01-28 00:16:06,136::caps::453::root::(_getKeyPackages) rpm package gluster-swift-doc not found
>Thread-260::DEBUG::2014-01-28 00:16:06,138::caps::453::root::(_getKeyPackages) rpm package glusterfs-server not found
>Thread-260::DEBUG::2014-01-28 00:16:06,139::caps::453::root::(_getKeyPackages) rpm package gluster-swift-container not found
>Thread-260::DEBUG::2014-01-28 00:16:06,139::caps::453::root::(_getKeyPackages) rpm package glusterfs-geo-replication not found
>Thread-260::DEBUG::2014-01-28 00:16:06,147::BindingXMLRPC::976::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.3.1.el6.x86_64', 'buildtime': 1388745567.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '33.gitf029545.el6', 'buildtime': 1390094004L, 'version': '4.14.0'}, 'qemu-kvm': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.2', 'buildtime': 1387360004L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.3', 'buildtime': 1386101870L, 'version': '0.12.1.2'}, 'mom': {'release': '20140117.gitfd877c5.el6', 'buildtime': 1389966957L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           L5320  @ 1.86GHz', 'hooks': {'before_vm_start': {'50_hostedengine': {'md5': '45dde62155b5412eafbfff5ef265acc2'}}}, 'cpuSockets': '2', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'gateway': '172.16.0.1', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.2', 'vnet0']}, 'os_net': {'iface': 'os_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.10']}, 'r2o_net': {'iface': 'r2o_net', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}, 'storage_network': {'iface': u'bond0.3', 'addr': '172.16.1.5', 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000', 'netmask': '255.255.255.255', 'bridged': False, 'interface': u'bond0.3', 'ipv6gateway': '::', 'gateway': ''}}, 'bridges': {'ovirtmgmt': {'addr': '172.16.0.11', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '172.16.0.1', 'ports': ['bond0.2', 'vnet0']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}, 'os_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'os_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.10']}, 'r2o_net': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'r2o_net', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['eth0.11', 'vnet1', 'vnet2', 'vnet3', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'vnet8', 'vnet9', 'vnet10']}}, 'uuid': '44454C4C-3100-104D-8046-B9C04F583153', 'lastClientIface': 'ovirtmgmt', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'cb327caa-ca77-4efe-b7b8-63aba3cda84d', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:85', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth2', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:85', 'speed': 1000, 'mtu': '4000'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'SLAVE': 'yes', 'UUID': 'e401a6a5-8773-4971-9213-b3c2bb52f16c', 'NM_CONTROLLED': 'no', 'HWADDR': '00:1E:C9:B6:D8:83', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': '00:1E:C9:B6:D8:83', 'speed': 1000, 'mtu': '4000'}, 'eth0': {'netmask': '', 'addr': '', 'hwaddr': '00:e0:4c:80:13:02', 'cfg': {'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': '00:e0:4c:80:13:02', 'STP': 'no', 'DEVICE': 'eth0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'speed': 1000, 'mtu': '1500'}}, 'software_revision': '33', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,tm2,ssse3,cx16,xtpr,pdcm,dca,lahf_lm,dts,tpr_shadow,vnmi,flexpriority,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:f5260b8e95e', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': ['eth1', 'eth2'], 'hwaddr': '00:1e:c9:b6:d8:83', 'cfg': {'MTU': '4000', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'BONDING_OPTS': 'miimon=100 mode=balance-rr', 'DEVICE': 'bond0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'mtu': '4000'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '32105', 'cpuSpeed': '1861.936', 'version_name': 'Snow Man', 'vlans': {'bond0.2': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'ovirtmgmt', 'VLAN': 'yes', 'MTU': '4000', 'BOOTPROTO': 'none', 'DEVICE': 'bond0.2', 'TYPE': 'Ethernet', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 2, 'mtu': '4000'}, 'bond0.3': {'netmask': '255.255.255.255', 'iface': 'bond0', 'addr': '172.16.1.5', 'cfg': {'VLAN': 'yes', 'DEFROUTE': 'no', 'MTU': '4000', 'NM_CONTROLLED': 'no', 'DEVICE': 'bond0.3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::21e:c9ff:feb6:d883/64'], 'vlanid': 3, 'mtu': '4000'}, 'eth0.11': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'r2o_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.11', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 11, 'mtu': '1500'}, 'eth0.10': {'netmask': '', 'iface': 'eth0', 'addr': '', 'cfg': {'BRIDGE': 'os_net', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.10', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::2e0:4cff:fe80:1302/64'], 'vlanid': 10, 'mtu': '1500'}}, 'cpuCores': '8', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '8', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '172.16.0.11'}}
>Thread-262::DEBUG::2014-01-28 00:16:06,159::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-262::DEBUG::2014-01-28 00:16:06,159::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>Thread-262::DEBUG::2014-01-28 00:16:06,160::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-262::DEBUG::2014-01-28 00:16:06,160::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-262::DEBUG::2014-01-28 00:16:06,160::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>Thread-262::DEBUG::2014-01-28 00:16:06,160::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:06,387::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:06,387::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:06,388::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:06,388::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:06,388::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:06,389::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:06,389::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:06,390::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:06,390::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:06,390::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:06,390::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:06,391::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-263::DEBUG::2014-01-28 00:16:06,600::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {}
>Thread-263::DEBUG::2014-01-28 00:16:06,638::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-12::DEBUG::2014-01-28 00:16:06,858::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:16:06,880::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-17::DEBUG::2014-01-28 00:16:07,234::iscsi::393::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-17::DEBUG::2014-01-28 00:16:07,236::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-17::DEBUG::2014-01-28 00:16:07,270::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-17::DEBUG::2014-01-28 00:16:07,389::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-17::DEBUG::2014-01-28 00:16:07,421::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc
>Thread-17::DEBUG::2014-01-28 00:16:07,423::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-17::DEBUG::2014-01-28 00:16:07,433::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=562f0160-7b80-42e2-b248-5754455c40fc:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=7e0cc075-eee7-4268-946e-33a79dd08748', 'REMOTE_PATH=172.16.1.5:/HOSTED-ENGINE', 'ROLE=Master', 'SDUUID=562f0160-7b80-42e2-b248-5754455c40fc', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=14825cb0b944c5c2ae9ed025c82e06730a2d91c4']
>Thread-17::DEBUG::2014-01-28 00:16:07,436::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-17::INFO::2014-01-28 00:16:07,436::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_imageNS already registered
>Thread-17::INFO::2014-01-28 00:16:07,436::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 562f0160-7b80-42e2-b248-5754455c40fc_volumeNS already registered
>Thread-17::DEBUG::2014-01-28 00:16:07,450::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE/562f0160-7b80-42e2-b248-5754455c40fc/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-17::DEBUG::2014-01-28 00:16:07,483::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n478 bytes (478 B) copied, 0.00183443 s, 261 kB/s\n'; <rc> = 0
>Thread-267::DEBUG::2014-01-28 00:16:07,972::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call tasksList with () {} flowID [4ae9a898]
>Thread-267::ERROR::2014-01-28 00:16:08,004::BindingXMLRPC::985::vds::(wrapper) vdsm exception occured
>Traceback (most recent call last):
>  File "/usr/share/vdsm/BindingXMLRPC.py", line 972, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/gluster/api.py", line 54, in wrapper
>    rv = func(*args, **kwargs)
>  File "/usr/share/vdsm/gluster/api.py", line 306, in tasksList
>    status = self.svdsmProxy.glusterTasksList(taskIds)
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in glusterTasksList
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 740, in _callmethod
>    raise convert_to_error(kind, result)
>GlusterCmdExecFailedException: Command execution failed
>error: tasks is not a valid status option
>Usage: volume status [all | <VOLNAME> [nfs|shd|<BRICK>]] [detail|clients|mem|inode|fd|callpool]
>return code: 1
>Thread-268::DEBUG::2014-01-28 00:16:08,239::BindingXMLRPC::159::vds::(wrapper) client [127.0.0.1]
>Thread-268::INFO::2014-01-28 00:16:08,241::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='7e0cc075-eee7-4268-946e-33a79dd08748', conList=[{'port': '', 'connection': '172.16.1.5:/HOSTED-ENGINE', 'iqn': '', 'portal': '', 'user': 'kvm', 'protocol_version': '3', 'password': '******', 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}], options=None)
>Thread-268::DEBUG::2014-01-28 00:16:08,245::hsm::2273::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.1.5:_HOSTED-ENGINE
>Thread-268::DEBUG::2014-01-28 00:16:08,247::hsm::2297::Storage.HSM::(__prefetchDomains) Found SD uuids: ('562f0160-7b80-42e2-b248-5754455c40fc',)
>Thread-268::DEBUG::2014-01-28 00:16:08,247::hsm::2353::Storage.HSM::(connectStorageServer) knownSDs: {562f0160-7b80-42e2-b248-5754455c40fc: storage.nfsSD.findDomain, 436c344c-bc57-441b-b311-c9595c6039e1: storage.nfsSD.findDomain, ff3a0c94-11dc-446d-b093-5f68ad81520d: storage.glusterSD.findDomain}
>Thread-268::INFO::2014-01-28 00:16:08,247::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '63249e13-a6df-4915-bf5f-be3da81463fc'}]}
>Thread-269::DEBUG::2014-01-28 00:16:08,462::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.11]
>Thread-269::INFO::2014-01-28 00:16:08,463::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-269::INFO::2014-01-28 00:16:08,464::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00183443', 'lastCheck': '1.0', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000330166', 'lastCheck': '7.3', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000990767', 'lastCheck': '6.0', 'valid': True}}
>Thread-270::DEBUG::2014-01-28 00:16:08,701::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.11]::call vmGetStats with ('2d425c4c-91b1-4564-b276-f27cc581e6a1',) {}
>Thread-270::DEBUG::2014-01-28 00:16:08,701::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-270::DEBUG::2014-01-28 00:16:08,702::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>Thread-270::DEBUG::2014-01-28 00:16:08,702::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>Thread-270::DEBUG::2014-01-28 00:16:08,702::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-270::DEBUG::2014-01-28 00:16:08,703::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>Thread-270::DEBUG::2014-01-28 00:16:08,703::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-270::DEBUG::2014-01-28 00:16:08,703::BindingXMLRPC::976::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'root', 'memUsage': '54', 'acpiEnable': 'true', 'guestFQDN': u'engine.melb.example.net', 'pid': '18113', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'/', u'total': '25323499520', u'used': '2592231424', u'fs': u'ext4'}, {u'path': u'/boot', u'total': '507744256', u'used': '55643136', u'fs': u'ext4'}], 'network': {u'vnet0': {'macAddr': '00:16:3e:0a:b7:55', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmType': 'kvm', 'guestName': u'engine.melb.example.net', 'elapsedTime': '610953', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'mem_free': '1919536', u'swap_in': '0', u'pageflt': '159', u'mem_total': '3922728', u'mem_unused': '1384816'}, 'cpuSys': '0.86', 'appsList': [u'ovirt-guest-agent-1.0.8-1.el6', u'kernel-2.6.32-431.el6', u'kernel-2.6.32-431.3.1.el6'], 'guestOs': u'2.6.32-431.3.1.el6.x86_64', 'displayType': 'vnc', 'status': 'Up', 'hash': '2950633987041661405', 'vmId': '2d425c4c-91b1-4564-b276-f27cc581e6a1', 'displayIp': '0', 'displayPort': u'5900', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'truesize': '3920601088', 'apparentsize': '26843545600', 'imageID': 'daa7fa08-1354-4b19-a80f-8694ea63f914'}, 'hdc': {'truesize': '0', 'apparentsize': '0'}}, 'monitorResponse': '0', 'statsAge': '0.08', 'cpuUser': '16.97', 'lastLogin': 1390828479.279238, 'guestIPs': u'172.16.0.10', 'netIfaces': [{u'hw': u'00:16:3e:0a:b7:55', u'name': u'eth0', u'inet': [u'172.16.0.10'], u'inet6': [u'fe80::216:3eff:fe0a:b755']}]}]}
>Thread-12::DEBUG::2014-01-28 00:16:08,895::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:16:08,921::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-12::DEBUG::2014-01-28 00:16:10,938::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:16:10,962::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-129::DEBUG::2014-01-28 00:16:11,182::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-129::DEBUG::2014-01-28 00:16:11,205::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n364 bytes (364 B) copied, 0.000336519 s, 1.1 MB/s\n'; <rc> = 0
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:11,397::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:11,398::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:11,398::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:11,398::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:11,399::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:11,399::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:11,400::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:11,400::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:11,400::vm::641::vm.Vm::(_getDiskStats) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc stats not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:11,401::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:11,401::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk vda latency not available
>GuestMonitor-HostedEngine::DEBUG::2014-01-28 00:16:11,401::vm::680::vm.Vm::(_getDiskLatency) vmId=`2d425c4c-91b1-4564-b276-f27cc581e6a1`::Disk hdc latency not available
>Thread-272::DEBUG::2014-01-28 00:16:11,741::BindingXMLRPC::969::vds::(wrapper) client [172.16.0.10]::call volumesList with () {}
>Thread-272::DEBUG::2014-01-28 00:16:11,794::BindingXMLRPC::976::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'HOSTED-ENGINE': {'transportType': ['TCP'], 'uuid': '224dc5cc-3ce9-4db2-84dc-4694c8bd6759', 'bricks': ['gs01.melb.example.net:/data1/hosted-engine'], 'volumeName': 'HOSTED-ENGINE', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'network.remote-dio': 'enable', 'auth.allow': '172.16.*.*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'performance.stat-prefetch': 'off', 'storage.owner-gid': '36'}}, 'VM-DATA': {'transportType': ['TCP'], 'uuid': 'f8e51af1-a0ca-411b-9738-c3e673d6c4e3', 'bricks': ['gs01.melb.example.net:/data1/vm-data'], 'volumeName': 'VM-DATA', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '172.16.*.*', 'performance.cache-size': '1GB', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'performance.io-cache': 'off', 'storage.owner-uid': '36', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-128::DEBUG::2014-01-28 00:16:12,473::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/172.16.1.5:_VM-DATA/ff3a0c94-11dc-446d-b093-5f68ad81520d/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-128::DEBUG::2014-01-28 00:16:12,496::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n504 bytes (504 B) copied, 0.00079311 s, 635 kB/s\n'; <rc> = 0
>Thread-12::DEBUG::2014-01-28 00:16:12,978::utils::555::root::(execCmd) '/sbin/ip -d -o link' (cwd None)
>Thread-12::DEBUG::2014-01-28 00:16:13,001::utils::575::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-274::DEBUG::2014-01-28 00:16:13,450::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-274::INFO::2014-01-28 00:16:13,451::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-274::INFO::2014-01-28 00:16:13,452::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 2}}
>Thread-275::DEBUG::2014-01-28 00:16:13,459::BindingXMLRPC::159::vds::(wrapper) client [172.16.0.10]
>Thread-275::INFO::2014-01-28 00:16:13,461::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', options=None)
>Thread-275::INFO::2014-01-28 00:16:13,468::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': 'ff3a0c94-11dc-446d-b093-5f68ad81520d', 'name': 'NextDC_M1', 'version': '3', 'domains': 'ff3a0c94-11dc-446d-b093-5f68ad81520d:Active,436c344c-bc57-441b-b311-c9595c6039e1:Active', 'pool_status': 'connected', 'isoprefix': '/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/images/11111111-1111-1111-1111-111111111111', 'type': 'GLUSTERFS', 'master_ver': 1, 'lver': 2}, 'dominfo': {'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'status': 'Active', 'diskfree': '917084569600', 'isoprefix': '', 'alerts': [], 'disktotal': '982907879424', 'version': 3}, '436c344c-bc57-441b-b311-c9595c6039e1': {'status': 'Active', 'diskfree': '21444952064', 'isoprefix': '/rhev/data-center/mnt/engine.melb.example.net:_var_lib_exports_iso/436c344c-bc57-441b-b311-c9595c6039e1/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '25323634688', 'version': 0}}}
>Thread-276::INFO::2014-01-28 00:16:14,649::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-276::INFO::2014-01-28 00:16:14,649::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'562f0160-7b80-42e2-b248-5754455c40fc': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00183443', 'lastCheck': '7.2', 'valid': True}, '436c344c-bc57-441b-b311-c9595c6039e1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000336519', 'lastCheck': '3.4', 'valid': True}, 'ff3a0c94-11dc-446d-b093-5f68ad81520d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00079311', 'lastCheck': '2.1', 'valid': True}}

Actions: View

Attachments on bug 1058300: 856063 | 856064 | 858008 | 858009 | 858010 | 867646 | 867647 | 884735 | 909506