Description of problem: VM Migration from 2 NUMA nodes hosts to 1 NUMA node host is failing. Version-Release number of selected component (if applicable): - Engine : CentOS 6.7, standalone bare-metal - oVirt 3.6.2.6 - Hosts : - 1 x CentOS 7.2 Dell Poweredge M610 Nehalem E5540 (1 NUMA node) - 2 x CentOS 7.2 Dell Poweredga R430 Haswell E5-2630 (2 NUMA nodes) How reproducible: Always Steps to Reproduce: 1. Install such a setup 2. Create a new VM, run it on a 2 NUMA nodes host 3. Migrate it to a 1 NUMA node host Actual results: The migration is failing with the error : Thread-6341::ERROR::2016-02-01 10:39:25,364::migration::310::virt.vm::(run) vmId=`49bfc23b-79a6-4f6c-840c-95ea04562513`::Failed to migrate Traceback (most recent call last): File "/usr/share/vdsm/virt/migration.py", line 294, in run self._startUnderlyingMigration(time.time()) File "/usr/share/vdsm/virt/migration.py", line 364, in _startUnderlyingMigration self._perform_migration(duri, muri) File "/usr/share/vdsm/virt/migration.py", line 403, in _perform_migration self._vm._dom.migrateToURI3(duri, params, flags) File "/usr/share/vdsm/virt/virdomain.py", line 68, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 124, in wrapper ret = f(*args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1836, in migrateToURI3 if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self) libvirtError: unsupported configuration: NUMA node 1 is unavailable Expected results: The migration should work. Additional info: Hard to remember but I feel this was not happening on 3.5.x but I couldn't swear. A lot has already been said in the second half of this other BZ : https://bugzilla.redhat.com/show_bug.cgi?id=1147644#c22 but I'm replicating them here for convenience. See below and attachements.
> - output of `lscpu' on all the nodes * === lscpu === [root@serv-vm-adms02 etc]# lscpu Architecture : x86_64 Mode(s) opératoire(s) des processeurs : 32-bit, 64-bit Boutisme : Little Endian Processeur(s) : 16 Liste de processeur(s) en ligne : 0-15 Thread(s) par cœur : 2 Cœur(s) par socket : 4 Socket(s) : 2 Nœud(s) NUMA : 1 Identifiant constructeur : GenuineIntel Famille de processeur : 6 Modèle : 26 Nom de modèle : Intel(R) Xeon(R) CPU E5540 @ 2.53GHz Révision : 5 Vitesse du processeur en MHz : 2527.020 BogoMIPS : 5053.19 Virtualisation : VT-x Cache L1d : 32K Cache L1i : 32K Cache L2 : 256K Cache L3 : 8192K Nœud NUMA 0 de processeur(s) : 0-15 [root@serv-vm-adms04 etc]# lscpu Architecture : x86_64 Mode(s) opératoire(s) des processeurs : 32-bit, 64-bit Boutisme : Little Endian Processeur(s) : 32 Liste de processeur(s) en ligne : 0-31 Thread(s) par cœur : 2 Cœur(s) par socket : 8 Socket(s) : 2 Nœud(s) NUMA : 2 Identifiant constructeur : GenuineIntel Famille de processeur : 6 Modèle : 63 Nom de modèle : Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz Révision : 2 Vitesse du processeur en MHz : 2758.687 BogoMIPS : 4798.47 Virtualisation : VT-x Cache L1d : 32K Cache L1i : 32K Cache L2 : 256K Cache L3 : 20480K Nœud NUMA 0 de processeur(s) : 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30 Nœud NUMA 1 de processeur(s) : 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31 [root@serv-vm-adms05 etc]# lscpu Architecture : x86_64 Mode(s) opératoire(s) des processeurs : 32-bit, 64-bit Boutisme : Little Endian Processeur(s) : 32 Liste de processeur(s) en ligne : 0-31 Thread(s) par cœur : 2 Cœur(s) par socket : 8 Socket(s) : 2 Nœud(s) NUMA : 2 Identifiant constructeur : GenuineIntel Famille de processeur : 6 Modèle : 63 Nom de modèle : Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz Révision : 2 Vitesse du processeur en MHz : 3017.250 BogoMIPS : 4798.83 Virtualisation : VT-x Cache L1d : 32K Cache L1i : 32K Cache L2 : 256K Cache L3 : 20480K Nœud NUMA 0 de processeur(s) : 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30 Nœud NUMA 1 de processeur(s) : 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31
> - output of `vdsClient -s 0 getVdsCaps' on all the nodes serv-vm-adms02 HBAInventory = {'FC': [], 'iSCSI': [{'InitiatorName': 'iqn.2009-03.fr.isere.sdis:serv-vm-adms02'}]} ISCSIInitiatorName = 'iqn.2009-03.fr.isere.sdis:serv-vm-adms02' additionalFeatures = [] autoNumaBalancing = 0 bondings = {'bond0': {'active_slave': 'em1', 'addr': '', 'cfg': {'BONDING_OPTS': 'mode=1 miimon=100', 'BOOTPROTO': 'none', 'DEVICE': 'bond0', 'HOTPLUG': 'no', 'MTU': '1500', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '', 'hwaddr': '00:22:19:68:10:ea', 'ipv4addrs': [], 'ipv6addrs': ['fe80::222:19ff:fe68:10ea/64'], 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '', 'opts': {'miimon': '100', 'mode': '1'}, 'slaves': ['em1', 'em2']}, 'bond1': {'active_slave': 'em3', 'addr': '', 'cfg': {'BONDING_OPTS': 'mode=1 miimon=100', 'BOOTPROTO': 'none', 'BRIDGE': 'ovirtiscsi02', 'DEVICE': 'bond1', 'HOTPLUG': 'no', 'MTU': '9000', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '', 'hwaddr': '00:22:19:68:10:ee', 'ipv4addrs': [], 'ipv6addrs': ['fe80::222:19ff:fe68:10ee/64'], 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '', 'opts': {'miimon': '100', 'mode': '1'}, 'slaves': ['em3', 'em4']}} bridges = {'ovirtiscsi02': {'addr': '192.168.29.11', 'cfg': {'BOOTPROTO': 'none', 'DEFROUTE': 'no', 'DELAY': '0', 'DEVICE': 'ovirtiscsi02', 'HOTPLUG': 'no', 'IPADDR': '192.168.29.11', 'MTU': '9000', 'NETMASK': '255.255.255.0', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes', 'STP': 'off', 'TYPE': 'Bridge'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '', 'ipv4addrs': ['192.168.29.11/24'], 'ipv6addrs': ['fe80::222:19ff:fe68:10ee/64'], 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '255.255.255.0', 'opts': {'ageing_time': '30000', 'bridge_id': '8000.0022196810ee', 'default_pvid': '1', 'forward_delay': '0', 'gc_timer': '2028', 'group_addr': '1:80:c2:0:0:0', 'group_fwd_mask': '0x0', 'hash_elasticity': '4', 'hash_max': '512', 'hello_time': '200', 'hello_timer': '128', 'max_age': '2000', 'multicast_last_member_count': '2', 'multicast_last_member_interval': '100', 'multicast_membership_interval': '26000', 'multicast_querier': '0', 'multicast_querier_interval': '25500', 'multicast_query_interval': '12500', 'multicast_query_response_interval': '1000', 'multicast_router': '1', 'multicast_snooping': '1', 'multicast_startup_query_count': '2', 'multicast_startup_query_interval': '3125', 'nf_call_arptables': '0', 'nf_call_ip6tables': '0', 'nf_call_iptables': '0', 'priority': '32768', 'root_id': '8000.0022196810ee', 'root_path_cost': '0', 'root_port': '0', 'stp_state': '0', 'tcn_timer': '0', 'topology_change': '0', 'topology_change_detected': '0', 'topology_change_timer': '0', 'vlan_filtering': '0'}, 'ports': ['bond1'], 'stp': 'off'}, 'ovirtmgmt': {'addr': '192.168.12.212', 'cfg': {'BOOTPROTO': 'none', 'DEFROUTE': 'yes', 'DELAY': '0', 'DEVICE': 'ovirtmgmt', 'GATEWAY': '192.168.12.254', 'HOTPLUG': 'no', 'IPADDR': '192.168.12.212', 'MTU': '1500', 'NETMASK': '255.255.255.0', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes', 'STP': 'off', 'TYPE': 'Bridge'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '192.168.12.254', 'ipv4addrs': ['192.168.12.212/24'], 'ipv6addrs': ['fe80::222:19ff:fe68:10ea/64'], 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '255.255.255.0', 'opts': {'ageing_time': '30000', 'bridge_id': '8000.0022196810ea', 'default_pvid': '1', 'forward_delay': '0', 'gc_timer': '2027', 'group_addr': '1:80:c2:0:0:0', 'group_fwd_mask': '0x0', 'hash_elasticity': '4', 'hash_max': '512', 'hello_time': '200', 'hello_timer': '127', 'max_age': '2000', 'multicast_last_member_count': '2', 'multicast_last_member_interval': '100', 'multicast_membership_interval': '26000', 'multicast_querier': '0', 'multicast_querier_interval': '25500', 'multicast_query_interval': '12500', 'multicast_query_response_interval': '1000', 'multicast_router': '1', 'multicast_snooping': '1', 'multicast_startup_query_count': '2', 'multicast_startup_query_interval': '3125', 'nf_call_arptables': '0', 'nf_call_ip6tables': '0', 'nf_call_iptables': '0', 'priority': '32768', 'root_id': '8000.0022196810ea', 'root_path_cost': '0', 'root_port': '0', 'stp_state': '0', 'tcn_timer': '0', 'topology_change': '0', 'topology_change_detected': '0', 'topology_change_timer': '0', 'vlan_filtering': '0'}, 'ports': ['bond0.6'], 'stp': 'off'}} clusterLevels = ['3.4', '3.5', '3.6'] cpuCores = '8' cpuFlags = 'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,nopl,xtopology,nonstop_tsc,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,dca,sse4_1,sse4_2,popcnt,lahf_lm,ida,dtherm,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_n270' cpuModel = 'Intel(R) Xeon(R) CPU E5540 @ 2.53GHz' cpuSockets = '2' cpuSpeed = '2527.020' cpuThreads = '16' emulatedMachines = ['pc-i440fx-rhel7.1.0', 'rhel6.3.0', 'pc-q35-rhel7.2.0', 'pc-i440fx-rhel7.0.0', 'rhel6.1.0', 'rhel6.6.0', 'rhel6.2.0', 'pc', 'pc-q35-rhel7.0.0', 'pc-q35-rhel7.1.0', 'q35', 'pc-i440fx-rhel7.2.0', 'rhel6.4.0', 'rhel6.0.0', 'rhel6.5.0'] guestOverhead = '65' hooks = {'before_device_create': {'50_vmfex': {'md5': '074c353cba2c58663e9eb11d97e9003f'}}, 'before_device_migrate_destination': {'50_vmfex': {'md5': '074c353cba2c58663e9eb11d97e9003f'}}, 'before_nic_hotplug': {'50_vmfex': {'md5': '074c353cba2c58663e9eb11d97e9003f'}}} hostdevPassthrough = 'false' kdumpStatus = 0 kvmEnabled = 'true' lastClient = '127.0.0.1' lastClientIface = 'lo' liveMerge = 'true' liveSnapshot = 'true' memSize = '16032' netConfigDirty = 'False' networks = {'ovirtiscsi02': {'addr': '192.168.29.11', 'bridged': True, 'cfg': {'BOOTPROTO': 'none', 'DEFROUTE': 'no', 'DELAY': '0', 'DEVICE': 'ovirtiscsi02', 'HOTPLUG': 'no', 'IPADDR': '192.168.29.11', 'MTU': '9000', 'NETMASK': '255.255.255.0', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes', 'STP': 'off', 'TYPE': 'Bridge'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '', 'iface': 'ovirtiscsi02', 'ipv4addrs': ['192.168.29.11/24'], 'ipv6addrs': ['fe80::222:19ff:fe68:10ee/64'], 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '255.255.255.0', 'ports': ['bond1'], 'stp': 'off'}, 'ovirtmgmt': {'addr': '192.168.12.212', 'bridged': True, 'cfg': {'BOOTPROTO': 'none', 'DEFROUTE': 'yes', 'DELAY': '0', 'DEVICE': 'ovirtmgmt', 'GATEWAY': '192.168.12.254', 'HOTPLUG': 'no', 'IPADDR': '192.168.12.212', 'MTU': '1500', 'NETMASK': '255.255.255.0', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes', 'STP': 'off', 'TYPE': 'Bridge'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '192.168.12.254', 'iface': 'ovirtmgmt', 'ipv4addrs': ['192.168.12.212/24'], 'ipv6addrs': ['fe80::222:19ff:fe68:10ea/64'], 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '255.255.255.0', 'ports': ['bond0.6'], 'stp': 'off'}} nics = {'em1': {'addr': '', 'cfg': {'BOOTPROTO': 'none', 'DEVICE': 'em1', 'HWADDR': '00:22:19:68:10:ea', 'MASTER': 'bond0', 'MTU': '1500', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes', 'SLAVE': 'yes'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '', 'hwaddr': '00:22:19:68:10:ea', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '', 'permhwaddr': '00:22:19:68:10:ea', 'speed': 1000}, 'em2': {'addr': '', 'cfg': {'BOOTPROTO': 'none', 'DEVICE': 'em2', 'HWADDR': '00:22:19:68:10:ec', 'MASTER': 'bond0', 'MTU': '1500', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes', 'SLAVE': 'yes'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '', 'hwaddr': '00:22:19:68:10:ea', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '', 'permhwaddr': '00:22:19:68:10:ec', 'speed': 1000}, 'em3': {'addr': '', 'cfg': {'BOOTPROTO': 'none', 'DEVICE': 'em3', 'HWADDR': '00:22:19:68:10:ee', 'MASTER': 'bond1', 'MTU': '9000', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes', 'SLAVE': 'yes'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '', 'hwaddr': '00:22:19:68:10:ee', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '', 'permhwaddr': '00:22:19:68:10:ee', 'speed': 1000}, 'em4': {'addr': '', 'cfg': {'BOOTPROTO': 'none', 'DEVICE': 'em4', 'HWADDR': '00:22:19:68:10:f0', 'MASTER': 'bond1', 'MTU': '9000', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes', 'SLAVE': 'yes'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '', 'hwaddr': '00:22:19:68:10:ee', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '', 'permhwaddr': '00:22:19:68:10:f0', 'speed': 1000}} numaNodeDistance = {'0': [10]} numaNodes = {'0': {'cpus': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], 'totalMemory': '16032'}} onlineCpus = '0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15' operatingSystem = {'name': 'RHEL', 'release': '2.1511.el7.centos.2.10', 'version': '7'} packages2 = {'glusterfs-cli': {'buildtime': 1447082499, 'release': '1.el7', 'version': '3.7.6'}, 'kernel': {'buildtime': 1453756034.0, 'release': '327.4.5.el7.x86_64', 'version': '3.10.0'}, 'librbd1': {'buildtime': 1448036424, 'release': '3.el7', 'version': '0.80.7'}, 'libvirt': {'buildtime': 1449666817, 'release': '13.el7_2.2', 'version': '1.2.17'}, 'mom': {'buildtime': 1442501481, 'release': '2.el7', 'version': '0.5.1'}, 'qemu-img': {'buildtime': 1450438721, 'release': '31.el7_2.4.1', 'version': '2.3.0'}, 'qemu-kvm': {'buildtime': 1450438721, 'release': '31.el7_2.4.1', 'version': '2.3.0'}, 'spice-server': {'buildtime': 1448026017, 'release': '15.el7', 'version': '0.12.4'}, 'vdsm': {'buildtime': 1453208693, 'release': '0.el7.centos', 'version': '4.17.18'}} reservedMem = '321' rngSources = ['random'] selinux = {'mode': '0'} software_revision = '0' software_version = '4.17' supportedENGINEs = ['3.4', '3.5', '3.6'] uuid = '4C4C4544-0031-3410-8059-B4C04F4C344A' version_name = 'Snow Man' vlans = {'bond0.6': {'addr': '', 'cfg': {'BOOTPROTO': 'none', 'BRIDGE': 'ovirtmgmt', 'DEVICE': 'bond0.6', 'HOTPLUG': 'no', 'MTU': '1500', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes', 'VLAN': 'yes'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '', 'iface': 'bond0', 'ipv4addrs': [], 'ipv6addrs': ['fe80::222:19ff:fe68:10ea/64'], 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '', 'vlanid': 6}} vmTypes = ['kvm'] serv-vm-adms04 HBAInventory = {'FC': [], 'iSCSI': [{'InitiatorName': 'iqn.2009-03.fr.isere.sdis:serv-vm-adms04'}]} ISCSIInitiatorName = 'iqn.2009-03.fr.isere.sdis:serv-vm-adms04' additionalFeatures = [] autoNumaBalancing = 1 bondings = {'bond0': {'active_slave': 'em1', 'addr': '', 'cfg': {'BONDING_OPTS': 'mode=1 miimon=100', 'BOOTPROTO': 'none', 'DEVICE': 'bond0', 'HOTPLUG': 'no', 'MTU': '1500', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '', 'hwaddr': '44:a8:42:1a:ff:ae', 'ipv4addrs': [], 'ipv6addrs': ['fe80::46a8:42ff:fe1a:ffae/64'], 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '', 'opts': {'miimon': '100', 'mode': '1'}, 'slaves': ['em1', 'em2']}, 'bond1': {'active_slave': 'em3', 'addr': '', 'cfg': {'BONDING_OPTS': 'mode=1 miimon=100', 'BOOTPROTO': 'none', 'BRIDGE': 'ovirtiscsi02', 'DEVICE': 'bond1', 'HOTPLUG': 'no', 'MTU': '9000', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '', 'hwaddr': '44:a8:42:1a:ff:b0', 'ipv4addrs': [], 'ipv6addrs': ['fe80::46a8:42ff:fe1a:ffb0/64'], 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '', 'opts': {'miimon': '100', 'mode': '1'}, 'slaves': ['em3', 'em4']}} bridges = {'ovirtiscsi02': {'addr': '192.168.29.16', 'cfg': {'BOOTPROTO': 'none', 'DEFROUTE': 'no', 'DELAY': '0', 'DEVICE': 'ovirtiscsi02', 'HOTPLUG': 'no', 'IPADDR': '192.168.29.16', 'MTU': '9000', 'NETMASK': '255.255.255.0', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes', 'STP': 'off', 'TYPE': 'Bridge'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '', 'ipv4addrs': ['192.168.29.16/24'], 'ipv6addrs': ['fe80::46a8:42ff:fe1a:ffb0/64'], 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '255.255.255.0', 'opts': {'ageing_time': '30000', 'bridge_id': '8000.44a8421affb0', 'default_pvid': '1', 'forward_delay': '0', 'gc_timer': '1264', 'group_addr': '1:80:c2:0:0:0', 'group_fwd_mask': '0x0', 'hash_elasticity': '4', 'hash_max': '512', 'hello_time': '200', 'hello_timer': '155', 'max_age': '2000', 'multicast_last_member_count': '2', 'multicast_last_member_interval': '100', 'multicast_membership_interval': '26000', 'multicast_querier': '0', 'multicast_querier_interval': '25500', 'multicast_query_interval': '12500', 'multicast_query_response_interval': '1000', 'multicast_router': '1', 'multicast_snooping': '1', 'multicast_startup_query_count': '2', 'multicast_startup_query_interval': '3125', 'nf_call_arptables': '0', 'nf_call_ip6tables': '0', 'nf_call_iptables': '0', 'priority': '32768', 'root_id': '8000.44a8421affb0', 'root_path_cost': '0', 'root_port': '0', 'stp_state': '0', 'tcn_timer': '0', 'topology_change': '0', 'topology_change_detected': '0', 'topology_change_timer': '0', 'vlan_filtering': '0'}, 'ports': ['bond1', 'vnet0', 'vnet3'], 'stp': 'off'}, 'ovirtmgmt': {'addr': '192.168.12.17', 'cfg': {'BOOTPROTO': 'none', 'DEFROUTE': 'yes', 'DELAY': '0', 'DEVICE': 'ovirtmgmt', 'GATEWAY': '192.168.12.254', 'HOTPLUG': 'no', 'IPADDR': '192.168.12.17', 'MTU': '1500', 'NETMASK': '255.255.255.0', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes', 'STP': 'off', 'TYPE': 'Bridge'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '192.168.12.254', 'ipv4addrs': ['192.168.12.17/24'], 'ipv6addrs': ['fe80::46a8:42ff:fe1a:ffae/64'], 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '255.255.255.0', 'opts': {'ageing_time': '30000', 'bridge_id': '8000.44a8421affae', 'default_pvid': '1', 'forward_delay': '0', 'gc_timer': '1264', 'group_addr': '1:80:c2:0:0:0', 'group_fwd_mask': '0x0', 'hash_elasticity': '4', 'hash_max': '512', 'hello_time': '200', 'hello_timer': '55', 'max_age': '2000', 'multicast_last_member_count': '2', 'multicast_last_member_interval': '100', 'multicast_membership_interval': '26000', 'multicast_querier': '0', 'multicast_querier_interval': '25500', 'multicast_query_interval': '12500', 'multicast_query_response_interval': '1000', 'multicast_router': '1', 'multicast_snooping': '1', 'multicast_startup_query_count': '2', 'multicast_startup_query_interval': '3125', 'nf_call_arptables': '0', 'nf_call_ip6tables': '0', 'nf_call_iptables': '0', 'priority': '32768', 'root_id': '8000.44a8421affae', 'root_path_cost': '0', 'root_port': '0', 'stp_state': '0', 'tcn_timer': '0', 'topology_change': '0', 'topology_change_detected': '0', 'topology_change_timer': '0', 'vlan_filtering': '0'}, 'ports': ['vnet1', 'vnet2', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'bond0.6'], 'stp': 'off'}} clusterLevels = ['3.4', '3.5', '3.6'] cpuCores = '16' cpuFlags = 'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,nopl,xtopology,nonstop_tsc,aperfmperf,eagerfpu,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,fma,cx16,xtpr,pdcm,pcid,dca,sse4_1,sse4_2,x2apic,movbe,popcnt,tsc_deadline_timer,aes,xsave,avx,f16c,rdrand,lahf_lm,abm,ida,arat,epb,pln,pts,dtherm,tpr_shadow,vnmi,flexpriority,ept,vpid,fsgsbase,tsc_adjust,bmi1,avx2,smep,bmi2,erms,invpcid,cqm,xsaveopt,cqm_llc,cqm_occup_llc,model_Haswell-noTSX,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_IvyBridge,model_Westmere,model_n270,model_SandyBridge' cpuModel = 'Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz' cpuSockets = '2' cpuSpeed = '3192.562' cpuThreads = '32' emulatedMachines = ['pc-i440fx-rhel7.1.0', 'rhel6.3.0', 'pc-q35-rhel7.2.0', 'pc-i440fx-rhel7.0.0', 'rhel6.1.0', 'rhel6.6.0', 'rhel6.2.0', 'pc', 'pc-q35-rhel7.0.0', 'pc-q35-rhel7.1.0', 'q35', 'pc-i440fx-rhel7.2.0', 'rhel6.4.0', 'rhel6.0.0', 'rhel6.5.0'] guestOverhead = '65' hooks = {'before_device_create': {'50_vmfex': {'md5': '074c353cba2c58663e9eb11d97e9003f'}}, 'before_device_migrate_destination': {'50_vmfex': {'md5': '074c353cba2c58663e9eb11d97e9003f'}}, 'before_nic_hotplug': {'50_vmfex': {'md5': '074c353cba2c58663e9eb11d97e9003f'}}} hostdevPassthrough = 'false' kdumpStatus = 0 kvmEnabled = 'true' lastClient = '127.0.0.1' lastClientIface = 'lo' liveMerge = 'true' liveSnapshot = 'true' memSize = '193342' netConfigDirty = 'False' networks = {'ovirtiscsi02': {'addr': '192.168.29.16', 'bridged': True, 'cfg': {'BOOTPROTO': 'none', 'DEFROUTE': 'no', 'DELAY': '0', 'DEVICE': 'ovirtiscsi02', 'HOTPLUG': 'no', 'IPADDR': '192.168.29.16', 'MTU': '9000', 'NETMASK': '255.255.255.0', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes', 'STP': 'off', 'TYPE': 'Bridge'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '', 'iface': 'ovirtiscsi02', 'ipv4addrs': ['192.168.29.16/24'], 'ipv6addrs': ['fe80::46a8:42ff:fe1a:ffb0/64'], 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '255.255.255.0', 'ports': ['bond1', 'vnet0', 'vnet3'], 'stp': 'off'}, 'ovirtmgmt': {'addr': '192.168.12.17', 'bridged': True, 'cfg': {'BOOTPROTO': 'none', 'DEFROUTE': 'yes', 'DELAY': '0', 'DEVICE': 'ovirtmgmt', 'GATEWAY': '192.168.12.254', 'HOTPLUG': 'no', 'IPADDR': '192.168.12.17', 'MTU': '1500', 'NETMASK': '255.255.255.0', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes', 'STP': 'off', 'TYPE': 'Bridge'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '192.168.12.254', 'iface': 'ovirtmgmt', 'ipv4addrs': ['192.168.12.17/24'], 'ipv6addrs': ['fe80::46a8:42ff:fe1a:ffae/64'], 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '255.255.255.0', 'ports': ['vnet1', 'vnet2', 'vnet4', 'vnet5', 'vnet6', 'vnet7', 'bond0.6'], 'stp': 'off'}} nics = {'em1': {'addr': '', 'cfg': {'BOOTPROTO': 'none', 'DEVICE': 'em1', 'HWADDR': '44:a8:42:1a:ff:ae', 'MASTER': 'bond0', 'MTU': '1500', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes', 'SLAVE': 'yes'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '', 'hwaddr': '44:a8:42:1a:ff:ae', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '', 'permhwaddr': '44:a8:42:1a:ff:ae', 'speed': 1000}, 'em2': {'addr': '', 'cfg': {'BOOTPROTO': 'none', 'DEVICE': 'em2', 'HWADDR': '44:a8:42:1a:ff:af', 'MASTER': 'bond0', 'MTU': '1500', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes', 'SLAVE': 'yes'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '', 'hwaddr': '44:a8:42:1a:ff:ae', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '', 'permhwaddr': '44:a8:42:1a:ff:af', 'speed': 1000}, 'em3': {'addr': '', 'cfg': {'BOOTPROTO': 'none', 'DEVICE': 'em3', 'HWADDR': '44:a8:42:1a:ff:b0', 'MASTER': 'bond1', 'MTU': '9000', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes', 'SLAVE': 'yes'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '', 'hwaddr': '44:a8:42:1a:ff:b0', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '', 'permhwaddr': '44:a8:42:1a:ff:b0', 'speed': 1000}, 'em4': {'addr': '', 'cfg': {'BOOTPROTO': 'none', 'DEVICE': 'em4', 'HWADDR': '44:a8:42:1a:ff:b1', 'MASTER': 'bond1', 'MTU': '9000', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes', 'SLAVE': 'yes'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '', 'hwaddr': '44:a8:42:1a:ff:b0', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '', 'permhwaddr': '44:a8:42:1a:ff:b1', 'speed': 1000}} numaNodeDistance = {'0': [10, 21], '1': [21, 10]} numaNodes = {'0': {'cpus': [0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30], 'totalMemory': '130975'}, '1': {'cpus': [1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31], 'totalMemory': '65536'}} onlineCpus = '0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31' operatingSystem = {'name': 'RHEL', 'release': '2.1511.el7.centos.2.10', 'version': '7'} packages2 = {'glusterfs-cli': {'buildtime': 1447082499, 'release': '1.el7', 'version': '3.7.6'}, 'kernel': {'buildtime': 1453756034.0, 'release': '327.4.5.el7.x86_64', 'version': '3.10.0'}, 'librbd1': {'buildtime': 1448036424, 'release': '3.el7', 'version': '0.80.7'}, 'libvirt': {'buildtime': 1449666817, 'release': '13.el7_2.2', 'version': '1.2.17'}, 'mom': {'buildtime': 1442501481, 'release': '2.el7', 'version': '0.5.1'}, 'qemu-img': {'buildtime': 1450438721, 'release': '31.el7_2.4.1', 'version': '2.3.0'}, 'qemu-kvm': {'buildtime': 1450438721, 'release': '31.el7_2.4.1', 'version': '2.3.0'}, 'spice-server': {'buildtime': 1448026017, 'release': '15.el7', 'version': '0.12.4'}, 'vdsm': {'buildtime': 1453208693, 'release': '0.el7.centos', 'version': '4.17.18'}} reservedMem = '321' rngSources = ['random'] selinux = {'mode': '-1'} software_revision = '0' software_version = '4.17' supportedENGINEs = ['3.4', '3.5', '3.6'] uuid = '4C4C4544-0039-3410-8033-C7C04F383532' version_name = 'Snow Man' vlans = {'bond0.6': {'addr': '', 'cfg': {'BOOTPROTO': 'none', 'BRIDGE': 'ovirtmgmt', 'DEVICE': 'bond0.6', 'HOTPLUG': 'no', 'MTU': '1500', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes', 'VLAN': 'yes'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '', 'iface': 'bond0', 'ipv4addrs': [], 'ipv6addrs': ['fe80::46a8:42ff:fe1a:ffae/64'], 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '', 'vlanid': 6}} vmTypes = ['kvm'] serv-vm-adms05 HBAInventory = {'FC': [], 'iSCSI': [{'InitiatorName': 'iqn.2009-03.fr.isere.sdis:serv-vm-adms05'}]} ISCSIInitiatorName = 'iqn.2009-03.fr.isere.sdis:serv-vm-adms05' additionalFeatures = [] autoNumaBalancing = 1 bondings = {'bond0': {'active_slave': 'em1', 'addr': '', 'cfg': {'BONDING_OPTS': 'mode=1 miimon=100', 'BOOTPROTO': 'none', 'DEVICE': 'bond0', 'HOTPLUG': 'no', 'MTU': '1500', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '', 'hwaddr': '44:a8:42:1a:df:4e', 'ipv4addrs': [], 'ipv6addrs': ['fe80::46a8:42ff:fe1a:df4e/64'], 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '', 'opts': {'miimon': '100', 'mode': '1'}, 'slaves': ['em1', 'em2']}, 'bond1': {'active_slave': 'em3', 'addr': '', 'cfg': {'BONDING_OPTS': 'mode=1 miimon=100', 'BOOTPROTO': 'none', 'BRIDGE': 'ovirtiscsi02', 'DEVICE': 'bond1', 'HOTPLUG': 'no', 'MTU': '9000', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '', 'hwaddr': '44:a8:42:1a:df:50', 'ipv4addrs': [], 'ipv6addrs': ['fe80::46a8:42ff:fe1a:df50/64'], 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '', 'opts': {'miimon': '100', 'mode': '1'}, 'slaves': ['em3', 'em4']}} bridges = {'ovirtiscsi02': {'addr': '192.168.29.17', 'cfg': {'BOOTPROTO': 'none', 'DEFROUTE': 'no', 'DELAY': '0', 'DEVICE': 'ovirtiscsi02', 'HOTPLUG': 'no', 'IPADDR': '192.168.29.17', 'MTU': '9000', 'NETMASK': '255.255.255.0', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes', 'STP': 'off', 'TYPE': 'Bridge'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '', 'ipv4addrs': ['192.168.29.17/24'], 'ipv6addrs': ['fe80::46a8:42ff:fe1a:df50/64'], 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '255.255.255.0', 'opts': {'ageing_time': '30000', 'bridge_id': '8000.44a8421adf50', 'default_pvid': '1', 'forward_delay': '0', 'gc_timer': '26947', 'group_addr': '1:80:c2:0:0:0', 'group_fwd_mask': '0x0', 'hash_elasticity': '4', 'hash_max': '512', 'hello_time': '200', 'hello_timer': '185', 'max_age': '2000', 'multicast_last_member_count': '2', 'multicast_last_member_interval': '100', 'multicast_membership_interval': '26000', 'multicast_querier': '0', 'multicast_querier_interval': '25500', 'multicast_query_interval': '12500', 'multicast_query_response_interval': '1000', 'multicast_router': '1', 'multicast_snooping': '1', 'multicast_startup_query_count': '2', 'multicast_startup_query_interval': '3125', 'nf_call_arptables': '0', 'nf_call_ip6tables': '0', 'nf_call_iptables': '0', 'priority': '32768', 'root_id': '8000.44a8421adf50', 'root_path_cost': '0', 'root_port': '0', 'stp_state': '0', 'tcn_timer': '0', 'topology_change': '0', 'topology_change_detected': '0', 'topology_change_timer': '0', 'vlan_filtering': '0'}, 'ports': ['bond1', 'vnet7'], 'stp': 'off'}, 'ovirtmgmt': {'addr': '192.168.12.18', 'cfg': {'BOOTPROTO': 'none', 'DEFROUTE': 'yes', 'DELAY': '0', 'DEVICE': 'ovirtmgmt', 'GATEWAY': '192.168.12.254', 'HOTPLUG': 'no', 'IPADDR': '192.168.12.18', 'MTU': '1500', 'NETMASK': '255.255.255.0', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes', 'STP': 'off', 'TYPE': 'Bridge'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '192.168.12.254', 'ipv4addrs': ['192.168.12.18/24'], 'ipv6addrs': ['fe80::46a8:42ff:fe1a:df4e/64'], 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '255.255.255.0', 'opts': {'ageing_time': '30000', 'bridge_id': '8000.44a8421adf4e', 'default_pvid': '1', 'forward_delay': '0', 'gc_timer': '694', 'group_addr': '1:80:c2:0:0:0', 'group_fwd_mask': '0x0', 'hash_elasticity': '4', 'hash_max': '512', 'hello_time': '200', 'hello_timer': '85', 'max_age': '2000', 'multicast_last_member_count': '2', 'multicast_last_member_interval': '100', 'multicast_membership_interval': '26000', 'multicast_querier': '0', 'multicast_querier_interval': '25500', 'multicast_query_interval': '12500', 'multicast_query_response_interval': '1000', 'multicast_router': '1', 'multicast_snooping': '1', 'multicast_startup_query_count': '2', 'multicast_startup_query_interval': '3125', 'nf_call_arptables': '0', 'nf_call_ip6tables': '0', 'nf_call_iptables': '0', 'priority': '32768', 'root_id': '8000.44a8421adf4e', 'root_path_cost': '0', 'root_port': '0', 'stp_state': '0', 'tcn_timer': '0', 'topology_change': '0', 'topology_change_detected': '0', 'topology_change_timer': '0', 'vlan_filtering': '0'}, 'ports': ['vnet0', 'vnet1', 'vnet6', 'vnet8', 'vnet12', 'bond0.6'], 'stp': 'off'}} clusterLevels = ['3.4', '3.5', '3.6'] cpuCores = '16' cpuFlags = 'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,nopl,xtopology,nonstop_tsc,aperfmperf,eagerfpu,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,fma,cx16,xtpr,pdcm,pcid,dca,sse4_1,sse4_2,x2apic,movbe,popcnt,tsc_deadline_timer,aes,xsave,avx,f16c,rdrand,lahf_lm,abm,ida,arat,epb,pln,pts,dtherm,tpr_shadow,vnmi,flexpriority,ept,vpid,fsgsbase,tsc_adjust,bmi1,avx2,smep,bmi2,erms,invpcid,cqm,xsaveopt,cqm_llc,cqm_occup_llc,model_Haswell-noTSX,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_IvyBridge,model_Westmere,model_n270,model_SandyBridge' cpuModel = 'Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz' cpuSockets = '2' cpuSpeed = '2411.156' cpuThreads = '32' emulatedMachines = ['pc-i440fx-rhel7.1.0', 'rhel6.3.0', 'pc-q35-rhel7.2.0', 'pc-i440fx-rhel7.0.0', 'rhel6.1.0', 'rhel6.6.0', 'rhel6.2.0', 'pc', 'pc-q35-rhel7.0.0', 'pc-q35-rhel7.1.0', 'q35', 'pc-i440fx-rhel7.2.0', 'rhel6.4.0', 'rhel6.0.0', 'rhel6.5.0'] guestOverhead = '65' hooks = {'before_device_create': {'50_vmfex': {'md5': '074c353cba2c58663e9eb11d97e9003f'}}, 'before_device_migrate_destination': {'50_vmfex': {'md5': '074c353cba2c58663e9eb11d97e9003f'}}, 'before_nic_hotplug': {'50_vmfex': {'md5': '074c353cba2c58663e9eb11d97e9003f'}}} hostdevPassthrough = 'false' kdumpStatus = 0 kvmEnabled = 'true' lastClient = '127.0.0.1' lastClientIface = 'lo' liveMerge = 'true' liveSnapshot = 'true' memSize = '193342' netConfigDirty = 'False' networks = {'ovirtiscsi02': {'addr': '192.168.29.17', 'bridged': True, 'cfg': {'BOOTPROTO': 'none', 'DEFROUTE': 'no', 'DELAY': '0', 'DEVICE': 'ovirtiscsi02', 'HOTPLUG': 'no', 'IPADDR': '192.168.29.17', 'MTU': '9000', 'NETMASK': '255.255.255.0', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes', 'STP': 'off', 'TYPE': 'Bridge'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '', 'iface': 'ovirtiscsi02', 'ipv4addrs': ['192.168.29.17/24'], 'ipv6addrs': ['fe80::46a8:42ff:fe1a:df50/64'], 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '255.255.255.0', 'ports': ['bond1', 'vnet7'], 'stp': 'off'}, 'ovirtmgmt': {'addr': '192.168.12.18', 'bridged': True, 'cfg': {'BOOTPROTO': 'none', 'DEFROUTE': 'yes', 'DELAY': '0', 'DEVICE': 'ovirtmgmt', 'GATEWAY': '192.168.12.254', 'HOTPLUG': 'no', 'IPADDR': '192.168.12.18', 'MTU': '1500', 'NETMASK': '255.255.255.0', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes', 'STP': 'off', 'TYPE': 'Bridge'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '192.168.12.254', 'iface': 'ovirtmgmt', 'ipv4addrs': ['192.168.12.18/24'], 'ipv6addrs': ['fe80::46a8:42ff:fe1a:df4e/64'], 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '255.255.255.0', 'ports': ['vnet0', 'vnet1', 'vnet6', 'vnet8', 'vnet12', 'bond0.6'], 'stp': 'off'}} nics = {'em1': {'addr': '', 'cfg': {'BOOTPROTO': 'none', 'DEVICE': 'em1', 'MASTER': 'bond0', 'MTU': '1500', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes', 'SLAVE': 'yes'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '', 'hwaddr': '44:a8:42:1a:df:4e', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '', 'permhwaddr': '44:a8:42:1a:df:4e', 'speed': 1000}, 'em2': {'addr': '', 'cfg': {'BOOTPROTO': 'none', 'DEVICE': 'em2', 'MASTER': 'bond0', 'MTU': '1500', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes', 'SLAVE': 'yes'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '', 'hwaddr': '44:a8:42:1a:df:4e', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '', 'permhwaddr': '44:a8:42:1a:df:4f', 'speed': 1000}, 'em3': {'addr': '', 'cfg': {'BOOTPROTO': 'none', 'DEVICE': 'em3', 'MASTER': 'bond1', 'MTU': '9000', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes', 'SLAVE': 'yes'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '', 'hwaddr': '44:a8:42:1a:df:50', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '', 'permhwaddr': '44:a8:42:1a:df:50', 'speed': 1000}, 'em4': {'addr': '', 'cfg': {'BOOTPROTO': 'none', 'DEVICE': 'em4', 'MASTER': 'bond1', 'MTU': '9000', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes', 'SLAVE': 'yes'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '', 'hwaddr': '44:a8:42:1a:df:50', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '', 'permhwaddr': '44:a8:42:1a:df:51', 'speed': 1000}} numaNodeDistance = {'0': [10, 21], '1': [21, 10]} numaNodes = {'0': {'cpus': [0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30], 'totalMemory': '130975'}, '1': {'cpus': [1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31], 'totalMemory': '65536'}} onlineCpus = '0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31' operatingSystem = {'name': 'RHEL', 'release': '2.1511.el7.centos.2.10', 'version': '7'} packages2 = {'glusterfs-cli': {'buildtime': 1447082499, 'release': '1.el7', 'version': '3.7.6'}, 'kernel': {'buildtime': 1453756034.0, 'release': '327.4.5.el7.x86_64', 'version': '3.10.0'}, 'librbd1': {'buildtime': 1448036424, 'release': '3.el7', 'version': '0.80.7'}, 'libvirt': {'buildtime': 1449666817, 'release': '13.el7_2.2', 'version': '1.2.17'}, 'mom': {'buildtime': 1442501481, 'release': '2.el7', 'version': '0.5.1'}, 'qemu-img': {'buildtime': 1450438721, 'release': '31.el7_2.4.1', 'version': '2.3.0'}, 'qemu-kvm': {'buildtime': 1450438721, 'release': '31.el7_2.4.1', 'version': '2.3.0'}, 'spice-server': {'buildtime': 1448026017, 'release': '15.el7', 'version': '0.12.4'}, 'vdsm': {'buildtime': 1453208693, 'release': '0.el7.centos', 'version': '4.17.18'}} reservedMem = '321' rngSources = ['random'] selinux = {'mode': '-1'} software_revision = '0' software_version = '4.17' supportedENGINEs = ['3.4', '3.5', '3.6'] uuid = '4C4C4544-0042-3410-8033-B3C04F383532' version_name = 'Snow Man' vlans = {'bond0.6': {'addr': '', 'cfg': {'BOOTPROTO': 'none', 'BRIDGE': 'ovirtmgmt', 'DEVICE': 'bond0.6', 'HOTPLUG': 'no', 'MTU': '1500', 'NM_CONTROLLED': 'no', 'ONBOOT': 'yes', 'VLAN': 'yes'}, 'dhcpv4': False, 'dhcpv6': False, 'gateway': '', 'iface': 'bond0', 'ipv4addrs': [], 'ipv6addrs': ['fe80::46a8:42ff:fe1a:df4e/64'], 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '', 'vlanid': 6}} vmTypes = ['kvm']
> virsh -r dumpxml $DOMAIN_ID > 4. please provide the dumped xml - dumpxml : http://ur1.ca/ogvx9 : [root@serv-vm-adms04 tmp]# virsh -r dumpxml serv-lin-adms1 <domain type='kvm' id='7100'> <name>serv-lin-adms1</name> <uuid>23d90f3b-352e-4ea4-8c6f-dcea3d34dd73</uuid> <metadata xmlns:ovirt="http://ovirt.org/vm/tune/1.0"> <ovirt:qos/> </metadata> <maxMemory slots='16' unit='KiB'>4294967296</maxMemory> <memory unit='KiB'>1048576</memory> <currentMemory unit='KiB'>1048576</currentMemory> <vcpu placement='static' current='1'>16</vcpu> <cputune> <shares>1020</shares> </cputune> <numatune> <memory mode='interleave' nodeset='0-1'/> </numatune> <resource> <partition>/machine</partition> </resource> <sysinfo type='smbios'> <system> <entry name='manufacturer'>oVirt</entry> <entry name='product'>oVirt Node</entry> <entry name='version'>7-2.1511.el7.centos.2.10</entry> <entry name='serial'>4C4C4544-0042-3410-8033-B3C04F383532</entry> <entry name='uuid'>23d90f3b-352e-4ea4-8c6f-dcea3d34dd73</entry> </system> </sysinfo> <os> <type arch='x86_64' machine='pc-i440fx-rhel7.2.0'>hvm</type> <boot dev='hd'/> <smbios mode='sysinfo'/> </os> <features> <acpi/> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>Nehalem</model> <topology sockets='16' cores='1' threads='1'/> <numa> <cell id='0' cpus='0' memory='1048576' unit='KiB'/> </numa> </cpu> <clock offset='variable' adjustment='0' basis='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source startupPolicy='optional'/> <backingStore/> <target dev='hdc' bus='ide'/> <readonly/> <serial></serial> <alias name='ide0-1-0'/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk> <disk type='block' device='disk' snapshot='no'> <driver name='qemu' type='qcow2' cache='none' error_policy='stop' io='native'/> <source dev='/rhev/data-center/00000002-0002-0002-0002-0000000003ca/f7bf2585-1f74-4655-9118-a790e9abc70d/images/d96b2076-1bdf-4960-bce4-3ed804319b65/1df08cf5-6ef1-4ecf-9673-72caf27c20ab'/> <backingStore/> <target dev='vda' bus='virtio'/> <serial>d96b2076-1bdf-4960-bce4-3ed804319b65</serial> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </disk> <controller type='scsi' index='0' model='virtio-scsi'> <alias name='scsi0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <controller type='virtio-serial' index='0' ports='16'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <controller type='ide' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='usb' index='0'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <interface type='bridge'> <mac address='00:1a:4a:cc:cf:23'/> <source bridge='ovirtmgmt'/> <target dev='vnet4'/> <model type='virtio'/> <filterref filter='vdsm-no-mac-spoofing'/> <link state='up'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channels/23d90f3b-352e-4ea4-8c6f-dcea3d34dd73.com.redhat.rhevm.vdsm'/> <target type='virtio' name='com.redhat.rhevm.vdsm' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channels/23d90f3b-352e-4ea4-8c6f-dcea3d34dd73.org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/> <alias name='channel1'/> <address type='virtio-serial' controller='0' bus='0' port='2'/> </channel> <input type='tablet' bus='usb'> <alias name='input0'/> </input> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='5902' autoport='yes' listen='192.168.12.17' passwdValidTo='1970-01-01T00:00:01'> <listen type='network' address='192.168.12.17' network='vdsm-ovirtmgmt'/> </graphics> <video> <model type='cirrus' vram='32768' heads='1'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> </domain>
- source : http://ur1.ca/ogvy7 Thread-6590::DEBUG::2016-02-01 13:51:14,578::fileSD::173::Storage.Misc.excCmd::(getReadDelay) /usr/bin/taskset --cpu-list 0-31 /usr/bin/dd if=/rhev/data-center/mnt/serv-mvm-adms01.sdis.isere.fr:_home_vmexports01/d3966e25-4872-41ee-81a7-121c7c9f7347/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) Thread-6591::DEBUG::2016-02-01 13:51:14,587::fileSD::173::Storage.Misc.excCmd::(getReadDelay) /usr/bin/taskset --cpu-list 0-31 /usr/bin/dd if=/rhev/data-center/mnt/serv-mvm-adms01.sdis.isere.fr:_home_iso/c4475304-caab-48cd-adb2-c70b4af7a259/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) Thread-6590::DEBUG::2016-02-01 13:51:14,594::fileSD::173::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n385 bytes (385 B) copied, 0.000322544 s, 1.2 MB/s\n'; <rc> = 0 Thread-6591::DEBUG::2016-02-01 13:51:14,597::fileSD::173::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n372 bytes (372 B) copied, 0.000278233 s, 1.3 MB/s\n'; <rc> = 0 Thread-6593::DEBUG::2016-02-01 13:51:16,930::blockSD::422::Storage.Misc.excCmd::(getReadDelay) /usr/bin/taskset --cpu-list 0-31 /usr/bin/dd if=/dev/b681cc9d-1afb-4f36-b394-af5cb644616f/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) Thread-6593::DEBUG::2016-02-01 13:51:16,938::blockSD::422::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000673858 s, 6.1 MB/s\n'; <rc> = 0 Thread-6589::DEBUG::2016-02-01 13:51:16,958::blockSD::422::Storage.Misc.excCmd::(getReadDelay) /usr/bin/taskset --cpu-list 0-31 /usr/bin/dd if=/dev/f7bf2585-1f74-4655-9118-a790e9abc70d/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) Thread-6589::DEBUG::2016-02-01 13:51:16,966::blockSD::422::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000499478 s, 8.2 MB/s\n'; <rc> = 0 jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:19,064::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'VM.migrate' in bridge with {u'params': {u'tunneled': u'false', u'dstqemu': u'192.168.29.11', u'autoConverge': u'false', u'src': u'serv-vm-adms04', u'dst': u'serv-vm-adms02:54321', u'vmId': u'49bfc23b-79a6-4f6c-840c-95ea04562513', u'abortOnError': u'true', u'compressed': u'false', u'method': u'online'}, u'vmID': u'49bfc23b-79a6-4f6c-840c-95ea04562513'} jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:19,064::API::546::vds::(migrate) {u'tunneled': u'false', u'dstqemu': u'192.168.29.11', u'autoConverge': u'false', u'src': u'serv-vm-adms04', u'dst': u'serv-vm-adms02:54321', u'vmId': u'49bfc23b-79a6-4f6c-840c-95ea04562513', u'abortOnError': u'true', u'compressed': u'false', u'method': u'online'} jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:19,066::__init__::206::jsonrpc.Notification::(emit) Sending event {"params": {"notify_time": 4806990620, "49bfc23b-79a6-4f6c-840c-95ea04562513": {"status": "Migration Source"}}, "jsonrpc": "2.0", "method": "|virt|VM_status|49bfc23b-79a6-4f6c-840c-95ea04562513"} jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:19,066::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'VM.migrate' in bridge with True Thread-7306::DEBUG::2016-02-01 13:51:19,078::migration::147::virt.vm::(_setupVdsConnection) vmId=`49bfc23b-79a6-4f6c-840c-95ea04562513`::Initiating connection with destination Thread-7306::DEBUG::2016-02-01 13:51:19,079::stompreactor::389::jsonrpc.AsyncoreClient::(send) Sending response JsonRpc (StompReactor)::DEBUG::2016-02-01 13:51:19,080::stomp::477::yajsonrpc.protocols.stomp.AsyncClient::(_process_connected) Stomp connection established Thread-7306::DEBUG::2016-02-01 13:51:19,082::migration::159::virt.vm::(_setupVdsConnection) vmId=`49bfc23b-79a6-4f6c-840c-95ea04562513`::Destination server is: serv-vm-adms02:54321 Thread-7306::DEBUG::2016-02-01 13:51:19,083::migration::202::virt.vm::(_prepareGuest) vmId=`49bfc23b-79a6-4f6c-840c-95ea04562513`::Migration started Thread-7306::DEBUG::2016-02-01 13:51:19,091::migration::286::virt.vm::(run) vmId=`49bfc23b-79a6-4f6c-840c-95ea04562513`::migration semaphore acquired after 0 seconds Thread-7306::DEBUG::2016-02-01 13:51:19,129::stompreactor::389::jsonrpc.AsyncoreClient::(send) Sending response Thread-7306::INFO::2016-02-01 13:51:19,669::migration::335::virt.vm::(_startUnderlyingMigration) vmId=`49bfc23b-79a6-4f6c-840c-95ea04562513`::Creation of destination VM took: 0 seconds Thread-7306::INFO::2016-02-01 13:51:19,669::migration::354::virt.vm::(_startUnderlyingMigration) vmId=`49bfc23b-79a6-4f6c-840c-95ea04562513`::starting migration to qemu+tls://serv-vm-adms02/system with miguri tcp://192.168.29.11 Thread-7307::DEBUG::2016-02-01 13:51:19,670::migration::443::virt.vm::(run) vmId=`49bfc23b-79a6-4f6c-840c-95ea04562513`::migration downtime thread started (10 steps) Thread-7307::DEBUG::2016-02-01 13:51:19,670::migration::466::virt.vm::(_set_downtime) vmId=`49bfc23b-79a6-4f6c-840c-95ea04562513`::setting migration downtime to 51 Thread-7308::DEBUG::2016-02-01 13:51:19,670::migration::500::virt.vm::(monitor_migration) vmId=`49bfc23b-79a6-4f6c-840c-95ea04562513`::starting migration monitor thread Thread-7306::DEBUG::2016-02-01 13:51:19,719::migration::558::virt.vm::(stop) vmId=`49bfc23b-79a6-4f6c-840c-95ea04562513`::stopping migration monitor thread Thread-7306::DEBUG::2016-02-01 13:51:19,719::migration::453::virt.vm::(stop) vmId=`49bfc23b-79a6-4f6c-840c-95ea04562513`::stopping migration downtime thread Thread-7306::ERROR::2016-02-01 13:51:19,719::migration::208::virt.vm::(_recover) vmId=`49bfc23b-79a6-4f6c-840c-95ea04562513`::unsupported configuration: NUMA node 1 is unavailable Thread-7306::DEBUG::2016-02-01 13:51:19,720::stompreactor::389::jsonrpc.AsyncoreClient::(send) Sending response Thread-7307::DEBUG::2016-02-01 13:51:19,720::migration::450::virt.vm::(run) vmId=`49bfc23b-79a6-4f6c-840c-95ea04562513`::migration downtime thread exiting Thread-7306::DEBUG::2016-02-01 13:51:20,051::__init__::206::jsonrpc.Notification::(emit) Sending event {"params": {"notify_time": 4806991610, "49bfc23b-79a6-4f6c-840c-95ea04562513": {"status": "Migration Source"}}, "jsonrpc": "2.0", "method": "|virt|VM_status|49bfc23b-79a6-4f6c-840c-95ea04562513"} Thread-7306::ERROR::2016-02-01 13:51:20,052::migration::310::virt.vm::(run) vmId=`49bfc23b-79a6-4f6c-840c-95ea04562513`::Failed to migrate Traceback (most recent call last): File "/usr/share/vdsm/virt/migration.py", line 294, in run self._startUnderlyingMigration(time.time()) File "/usr/share/vdsm/virt/migration.py", line 364, in _startUnderlyingMigration self._perform_migration(duri, muri) File "/usr/share/vdsm/virt/migration.py", line 403, in _perform_migration self._vm._dom.migrateToURI3(duri, params, flags) File "/usr/share/vdsm/virt/virdomain.py", line 68, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 124, in wrapper ret = f(*args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1836, in migrateToURI3 if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self) libvirtError: unsupported configuration: NUMA node 1 is unavailable jsonrpc.Executor/6::DEBUG::2016-02-01 13:51:20,691::task::595::Storage.TaskManager.Task::(_updateState) Task=`eaf8e2a8-657b-4b55-b375-8fd66c34665a`::moving from state init -> state preparing jsonrpc.Executor/6::INFO::2016-02-01 13:51:20,692::logUtils::48::dispatcher::(wrapper) Run and protect: repoStats(options=None) jsonrpc.Executor/6::INFO::2016-02-01 13:51:20,692::logUtils::51::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'f7bf2585-1f74-4655-9118-a790e9abc70d': {'code': 0, 'actual': True, 'version': 3, 'acquired': True, 'delay': '0.000499478', 'lastCheck': '3.7', 'valid': True}, u'd3966e25-4872-41ee-81a7-121c7c9f7347': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000322544', 'lastCheck': '6.1', 'valid': True}, u'c4475304-caab-48cd-adb2-c70b4af7a259': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000278233', 'lastCheck': '6.1', 'valid': True}, u'b681cc9d-1afb-4f36-b394-af5cb644616f': {'code': 0, 'actual': True, 'version': 3, 'acquired': True, 'delay': '0.000673858', 'lastCheck': '3.8', 'valid': True}} jsonrpc.Executor/6::DEBUG::2016-02-01 13:51:20,692::task::1191::Storage.TaskManager.Task::(prepare) Task=`eaf8e2a8-657b-4b55-b375-8fd66c34665a`::finished: {u'f7bf2585-1f74-4655-9118-a790e9abc70d': {'code': 0, 'actual': True, 'version': 3, 'acquired': True, 'delay': '0.000499478', 'lastCheck': '3.7', 'valid': True}, u'd3966e25-4872-41ee-81a7-121c7c9f7347': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000322544', 'lastCheck': '6.1', 'valid': True}, u'c4475304-caab-48cd-adb2-c70b4af7a259': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000278233', 'lastCheck': '6.1', 'valid': True}, u'b681cc9d-1afb-4f36-b394-af5cb644616f': {'code': 0, 'actual': True, 'version': 3, 'acquired': True, 'delay': '0.000673858', 'lastCheck': '3.8', 'valid': True}} jsonrpc.Executor/6::DEBUG::2016-02-01 13:51:20,692::task::595::Storage.TaskManager.Task::(_updateState) Task=`eaf8e2a8-657b-4b55-b375-8fd66c34665a`::moving from state preparing -> state finished jsonrpc.Executor/6::DEBUG::2016-02-01 13:51:20,692::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} jsonrpc.Executor/6::DEBUG::2016-02-01 13:51:20,692::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/6::DEBUG::2016-02-01 13:51:20,692::task::993::Storage.TaskManager.Task::(_decref) Task=`eaf8e2a8-657b-4b55-b375-8fd66c34665a`::ref 0 aborting False Reactor thread::INFO::2016-02-01 13:51:23,500::protocoldetector::72::ProtocolDetector.AcceptorImpl::(handle_accept) Accepting connection from 127.0.0.1:51739 Reactor thread::DEBUG::2016-02-01 13:51:23,505::protocoldetector::82::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::INFO::2016-02-01 13:51:23,506::protocoldetector::118::ProtocolDetector.Detector::(handle_read) Detected protocol xml from 127.0.0.1:51739 Reactor thread::DEBUG::2016-02-01 13:51:23,506::bindingxmlrpc::1297::XmlDetector::(handle_socket) xml over http detected from ('127.0.0.1', 51739) BindingXMLRPC::INFO::2016-02-01 13:51:23,506::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting request handler for 127.0.0.1:51739 Thread-7309::INFO::2016-02-01 13:51:23,506::xmlrpc::84::vds.XMLRPCServer::(_process_requests) Request handler for 127.0.0.1:51739 started Thread-7309::INFO::2016-02-01 13:51:23,522::xmlrpc::92::vds.XMLRPCServer::(_process_requests) Request handler for 127.0.0.1:51739 stopped Thread-6590::DEBUG::2016-02-01 13:51:24,596::fileSD::173::Storage.Misc.excCmd::(getReadDelay) /usr/bin/taskset --cpu-list 0-31 /usr/bin/dd if=/rhev/data-center/mnt/serv-mvm-adms01.sdis.isere.fr:_home_vmexports01/d3966e25-4872-41ee-81a7-121c7c9f7347/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) Thread-6591::DEBUG::2016-02-01 13:51:24,605::fileSD::173::Storage.Misc.excCmd::(getReadDelay) /usr/bin/taskset --cpu-list 0-31 /usr/bin/dd if=/rhev/data-center/mnt/serv-mvm-adms01.sdis.isere.fr:_home_iso/c4475304-caab-48cd-adb2-c70b4af7a259/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) Thread-6590::DEBUG::2016-02-01 13:51:24,614::fileSD::173::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n385 bytes (385 B) copied, 0.000320073 s, 1.2 MB/s\n'; <rc> = 0 Thread-6591::DEBUG::2016-02-01 13:51:24,617::fileSD::173::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n372 bytes (372 B) copied, 0.000309749 s, 1.2 MB/s\n'; <rc> = 0 Thread-6593::DEBUG::2016-02-01 13:51:26,940::blockSD::422::Storage.Misc.excCmd::(getReadDelay) /usr/bin/taskset --cpu-list 0-31 /usr/bin/dd if=/dev/b681cc9d-1afb-4f36-b394-af5cb644616f/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) Thread-6593::DEBUG::2016-02-01 13:51:26,949::blockSD::422::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758067 s, 5.4 MB/s\n'; <rc> = 0 Thread-6589::DEBUG::2016-02-01 13:51:26,967::blockSD::422::Storage.Misc.excCmd::(getReadDelay) /usr/bin/taskset --cpu-list 0-31 /usr/bin/dd if=/dev/f7bf2585-1f74-4655-9118-a790e9abc70d/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) Thread-6589::DEBUG::2016-02-01 13:51:26,974::blockSD::422::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000491324 s, 8.3 MB/s\n'; <rc> = 0 jsonrpc.Executor/0::DEBUG::2016-02-01 13:51:27,174::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'Host.getVMFullList' in bridge with {u'vmList': [u'49bfc23b-79a6-4f6c-840c-95ea04562513']} jsonrpc.Executor/0::DEBUG::2016-02-01 13:51:27,175::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'Host.getVMFullList' in bridge with [{u'acpiEnable': u'true', u'emulatedMachine': u'pc-i440fx-rhel7.2.0', u'afterMigrationStatus': u'', u'tabletEnable': u'true', 'pid': '107037', u'memGuaranteedSize': 1024, u'transparentHugePages': u'true', u'displayPort': u'5904', u'displaySecurePort': '-1', u'spiceSslCipherSuite': u'DEFAULT', u'cpuType': u'Nehalem', u'smp': u'2', u'guestNumaNodes': [{u'nodeIndex': 0, u'cpus': u'0,1', u'memory': u'1024'}], u'smartcardEnable': u'false', u'numaTune': {u'nodeset': u'1,0', u'mode': u'interleave'}, u'custom': {u'device_90e6ab00-d9e0-4d13-9b2b-da53872f48d5device_bfec0f5c-150a-4e0c-94a8-56bb5ffd506e': u"VmDevice:{id='VmDeviceId:{deviceId='bfec0f5c-150a-4e0c-94a8-56bb5ffd506e', vmId='49bfc23b-79a6-4f6c-840c-95ea04562513'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false'}", u'device_90e6ab00-d9e0-4d13-9b2b-da53872f48d5device_bfec0f5c-150a-4e0c-94a8-56bb5ffd506edevice_185fa779-0f85-4a4e-9633-f8c2c81fc746': u"VmDevice:{id='VmDeviceId:{deviceId='185fa779-0f85-4a4e-9633-f8c2c81fc746', vmId='49bfc23b-79a6-4f6c-840c-95ea04562513'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false'}", u'device_90e6ab00-d9e0-4d13-9b2b-da53872f48d5': u"VmDevice:{id='VmDeviceId:{deviceId='90e6ab00-d9e0-4d13-9b2b-da53872f48d5', vmId='49bfc23b-79a6-4f6c-840c-95ea04562513'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false'}"}, u'vmType': u'kvm', u'memSize': 1024, u'smpCoresPerSocket': u'1', u'vmName': u'serv-vpn-adms1', u'nice': u'0', u'status': 'Up', u'maxMemSize': 4194304, u'bootMenuEnable': u'false', u'vmId': u'49bfc23b-79a6-4f6c-840c-95ea04562513', u'displayIp': '192.168.12.17', u'smpThreadsPerCore': u'1', u'guestDiskMapping': {u'3ccc0ba0-44f9-4a71-a': {u'name': u'/dev/vda'}, u'QEMU_DVD-ROM': {u'name': u'/dev/sr0'}}, u'spiceSecureChannels': u'smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard', u'nicModel': u'rtl8139,pv', u'keyboardLayout': u'fr', u'kvmEnable': u'true', u'pitReinjection': u'false', u'displayNetwork': u'ovirtmgmt', u'devices': [{u'target': 1048576, u'specParams': {u'model': u'virtio'}, u'alias': u'balloon0', 'deviceType': u'balloon', u'deviceId': u'57055dc0-e7a1-434d-a6f7-5ebe141511b5', u'address': {u'slot': u'0x07', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'device': u'memballoon', u'type': u'balloon'}, {u'device': u'unix', u'alias': u'channel0', 'deviceType': u'channel', u'type': u'channel', u'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'1'}}, {u'device': u'unix', u'alias': u'channel1', 'deviceType': u'channel', u'type': u'channel', u'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'2'}}, {u'index': u'0', u'alias': u'scsi0', u'specParams': {}, 'deviceType': u'controller', u'deviceId': u'b624e28c-7a07-4c1f-b819-7f2afad89079', u'address': {u'slot': u'0x04', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'device': u'scsi', u'model': u'virtio-scsi', u'type': u'controller'}, {u'specParams': {}, u'alias': u'virtio-serial0', 'deviceType': u'controller', u'deviceId': u'a54ef659-aefc-4c64-8211-02ae09fa45e4', u'address': {u'slot': u'0x05', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'device': u'virtio-serial', u'type': u'controller'}, {u'device': u'usb', u'alias': u'usb', 'deviceType': u'controller', u'type': u'controller', u'address': {u'slot': u'0x01', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x2'}}, {u'device': u'ide', u'alias': u'ide', 'deviceType': u'controller', u'type': u'controller', u'address': {u'slot': u'0x01', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x1'}}, {u'specParams': {u'vram': u'32768', u'heads': u'1'}, u'alias': u'video0', 'deviceType': u'video', u'deviceId': u'eb8ae28f-2505-48fd-b850-6043074d32ff', u'address': {u'slot': u'0x02', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'device': u'cirrus', u'type': u'video'}, {u'specParams': {u'displayNetwork': u'ovirtmgmt', u'displayIp': '192.168.12.17'}, 'deviceType': u'graphics', u'deviceId': u'00000032-0032-0032-0032-00000000013c', u'device': u'vnc', u'type': u'graphics', u'port': u'5904'}, {u'nicModel': u'pv', u'macAddr': u'00:1a:4a:cc:00:05', u'linkActive': True, u'network': u'ovirtmgmt', u'specParams': {u'inbound': {}, u'outbound': {}}, u'filter': u'vdsm-no-mac-spoofing', u'alias': u'net0', 'deviceType': u'interface', u'deviceId': u'8a209ba0-46e0-434b-9b90-461e32959cda', u'address': {u'slot': u'0x03', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'device': u'bridge', u'type': u'interface', u'name': u'vnet6'}, {u'index': u'2', u'iface': u'ide', u'name': u'hdc', u'alias': u'ide0-1-0', u'specParams': {u'path': u''}, u'readonly': 'True', 'deviceType': u'disk', u'deviceId': u'1b7bc6a6-45db-40d1-a80a-11a3db93f25a', u'address': {u'bus': u'1', u'controller': u'0', u'type': u'drive', u'target': u'0', u'unit': u'0'}, u'device': u'cdrom', u'shared': u'false', u'path': '', u'type': u'disk'}, {u'poolID': u'00000002-0002-0002-0002-0000000003ca', u'volumeInfo': {'domainID': u'f7bf2585-1f74-4655-9118-a790e9abc70d', 'volType': 'path', 'leaseOffset': 123731968, 'volumeID': '75016b46-a4c3-46d2-8908-0fbc7204e9ac', 'leasePath': '/dev/f7bf2585-1f74-4655-9118-a790e9abc70d/leases', 'imageID': u'3ccc0ba0-44f9-4a71-a3c5-193dd364c87e', 'path': u'/rhev/data-center/mnt/blockSD/f7bf2585-1f74-4655-9118-a790e9abc70d/images/3ccc0ba0-44f9-4a71-a3c5-193dd364c87e/75016b46-a4c3-46d2-8908-0fbc7204e9ac'}, u'index': 0, u'iface': u'virtio', u'apparentsize': '6442450944', u'specParams': {}, u'imageID': u'3ccc0ba0-44f9-4a71-a3c5-193dd364c87e', u'readonly': 'False', 'deviceType': u'disk', u'shared': u'false', u'truesize': '6442450944', u'type': u'disk', u'domainID': u'f7bf2585-1f74-4655-9118-a790e9abc70d', u'reqsize': u'0', u'format': u'cow', u'deviceId': u'3ccc0ba0-44f9-4a71-a3c5-193dd364c87e', u'address': {u'slot': u'0x06', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'device': u'disk', u'path': u'/rhev/data-center/00000002-0002-0002-0002-0000000003ca/f7bf2585-1f74-4655-9118-a790e9abc70d/images/3ccc0ba0-44f9-4a71-a3c5-193dd364c87e/75016b46-a4c3-46d2-8908-0fbc7204e9ac', u'propagateErrors': u'off', u'optional': u'false', u'name': u'vda', u'bootOrder': u'1', u'volumeID': u'75016b46-a4c3-46d2-8908-0fbc7204e9ac', u'alias': u'virtio-disk0', u'volumeChain': [{'domainID': u'f7bf2585-1f74-4655-9118-a790e9abc70d', 'volType': 'path', 'leaseOffset': 123731968, 'volumeID': '75016b46-a4c3-46d2-8908-0fbc7204e9ac', 'leasePath': '/dev/f7bf2585-1f74-4655-9118-a790e9abc70d/leases', 'imageID': u'3ccc0ba0-44f9-4a71-a3c5-193dd364c87e', 'path': u'/rhev/data-center/mnt/blockSD/f7bf2585-1f74-4655-9118-a790e9abc70d/images/3ccc0ba0-44f9-4a71-a3c5-193dd364c87e/75016b46-a4c3-46d2-8908-0fbc7204e9ac'}]}], u'display': 'vnc', u'timeOffset': u'3600', u'maxVCpus': u'16', 'clientIp': u'', u'statusTime': '4806998730', u'maxMemSlots': 16}] jsonrpc.Executor/2::DEBUG::2016-02-01 13:51:28,249::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'VM.getMigrationStatus' in bridge with {u'vmID': u'49bfc23b-79a6-4f6c-840c-95ea04562513'} jsonrpc.Executor/2::DEBUG::2016-02-01 13:51:28,250::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'VM.getMigrationStatus' in bridge with {'status': {'message': 'Fatal error during migration', 'code': 12}, 'progress': 0}
- target : http://ur1.ca/ogvy9 Thread-741::DEBUG::2016-02-01 13:51:15,273::blockSD::422::Storage.Misc.excCmd::(getReadDelay) /usr/bin/taskset --cpu-list 0-15 /usr/bin/dd if=/dev/b681cc9d-1afb-4f36-b394-af5cb644616f/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) Thread-741::DEBUG::2016-02-01 13:51:15,283::blockSD::422::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837655 s, 4.9 MB/s\n'; <rc> = 0 Thread-739::DEBUG::2016-02-01 13:51:15,361::fileSD::173::Storage.Misc.excCmd::(getReadDelay) /usr/bin/taskset --cpu-list 0-15 /usr/bin/dd if=/rhev/data-center/mnt/serv-mvm-adms01.sdis.isere.fr:_home_iso/c4475304-caab-48cd-adb2-c70b4af7a259/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) Thread-738::DEBUG::2016-02-01 13:51:15,370::fileSD::173::Storage.Misc.excCmd::(getReadDelay) /usr/bin/taskset --cpu-list 0-15 /usr/bin/dd if=/rhev/data-center/mnt/serv-mvm-adms01.sdis.isere.fr:_home_vmexports01/d3966e25-4872-41ee-81a7-121c7c9f7347/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) Thread-739::DEBUG::2016-02-01 13:51:15,379::fileSD::173::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n372 bytes (372 B) copied, 0.000435299 s, 855 kB/s\n'; <rc> = 0 Thread-738::DEBUG::2016-02-01 13:51:15,382::fileSD::173::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n385 bytes (385 B) copied, 0.000378542 s, 1.0 MB/s\n'; <rc> = 0 Thread-737::DEBUG::2016-02-01 13:51:15,620::blockSD::422::Storage.Misc.excCmd::(getReadDelay) /usr/bin/taskset --cpu-list 0-15 /usr/bin/dd if=/dev/f7bf2585-1f74-4655-9118-a790e9abc70d/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) Thread-737::DEBUG::2016-02-01 13:51:15,629::blockSD::422::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000560646 s, 7.3 MB/s\n'; <rc> = 0 jsonrpc.Executor/2::DEBUG::2016-02-01 13:51:15,678::task::595::Storage.TaskManager.Task::(_updateState) Task=`b0e8bf04-14f8-455a-a8ca-27ed7a41f550`::moving from state init -> state preparing jsonrpc.Executor/2::INFO::2016-02-01 13:51:15,678::logUtils::48::dispatcher::(wrapper) Run and protect: repoStats(options=None) jsonrpc.Executor/2::INFO::2016-02-01 13:51:15,678::logUtils::51::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'f7bf2585-1f74-4655-9118-a790e9abc70d': {'code': 0, 'actual': True, 'version': 3, 'acquired': True, 'delay': '0.000560646', 'lastCheck': '0.0', 'valid': True}, u'd3966e25-4872-41ee-81a7-121c7c9f7347': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000378542', 'lastCheck': '0.3', 'valid': True}, u'c4475304-caab-48cd-adb2-c70b4af7a259': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000435299', 'lastCheck': '0.3', 'valid': True}, u'b681cc9d-1afb-4f36-b394-af5cb644616f': {'code': 0, 'actual': True, 'version': 3, 'acquired': True, 'delay': '0.000837655', 'lastCheck': '0.4', 'valid': True}} jsonrpc.Executor/2::DEBUG::2016-02-01 13:51:15,678::task::1191::Storage.TaskManager.Task::(prepare) Task=`b0e8bf04-14f8-455a-a8ca-27ed7a41f550`::finished: {u'f7bf2585-1f74-4655-9118-a790e9abc70d': {'code': 0, 'actual': True, 'version': 3, 'acquired': True, 'delay': '0.000560646', 'lastCheck': '0.0', 'valid': True}, u'd3966e25-4872-41ee-81a7-121c7c9f7347': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000378542', 'lastCheck': '0.3', 'valid': True}, u'c4475304-caab-48cd-adb2-c70b4af7a259': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000435299', 'lastCheck': '0.3', 'valid': True}, u'b681cc9d-1afb-4f36-b394-af5cb644616f': {'code': 0, 'actual': True, 'version': 3, 'acquired': True, 'delay': '0.000837655', 'lastCheck': '0.4', 'valid': True}} jsonrpc.Executor/2::DEBUG::2016-02-01 13:51:15,678::task::595::Storage.TaskManager.Task::(_updateState) Task=`b0e8bf04-14f8-455a-a8ca-27ed7a41f550`::moving from state preparing -> state finished jsonrpc.Executor/2::DEBUG::2016-02-01 13:51:15,678::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} jsonrpc.Executor/2::DEBUG::2016-02-01 13:51:15,678::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/2::DEBUG::2016-02-01 13:51:15,679::task::993::Storage.TaskManager.Task::(_decref) Task=`b0e8bf04-14f8-455a-a8ca-27ed7a41f550`::ref 0 aborting False Reactor thread::INFO::2016-02-01 13:51:17,140::protocoldetector::72::ProtocolDetector.AcceptorImpl::(handle_accept) Accepting connection from 127.0.0.1:35104 Reactor thread::DEBUG::2016-02-01 13:51:17,147::protocoldetector::82::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::INFO::2016-02-01 13:51:17,147::protocoldetector::118::ProtocolDetector.Detector::(handle_read) Detected protocol xml from 127.0.0.1:35104 Reactor thread::DEBUG::2016-02-01 13:51:17,147::bindingxmlrpc::1297::XmlDetector::(handle_socket) xml over http detected from ('127.0.0.1', 35104) BindingXMLRPC::INFO::2016-02-01 13:51:17,148::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting request handler for 127.0.0.1:35104 Thread-775::INFO::2016-02-01 13:51:17,148::xmlrpc::84::vds.XMLRPCServer::(_process_requests) Request handler for 127.0.0.1:35104 started Thread-775::INFO::2016-02-01 13:51:17,149::xmlrpc::92::vds.XMLRPCServer::(_process_requests) Request handler for 127.0.0.1:35104 stopped Reactor thread::INFO::2016-02-01 13:51:19,071::protocoldetector::72::ProtocolDetector.AcceptorImpl::(handle_accept) Accepting connection from 192.168.12.17:52591 Reactor thread::DEBUG::2016-02-01 13:51:19,077::protocoldetector::82::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::INFO::2016-02-01 13:51:19,078::protocoldetector::118::ProtocolDetector.Detector::(handle_read) Detected protocol stomp from 192.168.12.17:52591 Reactor thread::INFO::2016-02-01 13:51:19,079::stompreactor::101::Broker.StompAdapter::(_cmd_connect) Processing CONNECT request Reactor thread::DEBUG::2016-02-01 13:51:19,079::stompreactor::470::protocoldetector.StompDetector::(handle_socket) Stomp detected from ('192.168.12.17', 52591) JsonRpc (StompReactor)::INFO::2016-02-01 13:51:19,080::stompreactor::128::Broker.StompAdapter::(_cmd_subscribe) Subscribe command received jsonrpc.Executor/3::DEBUG::2016-02-01 13:51:19,080::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'Host.ping' in bridge with [] jsonrpc.Executor/3::DEBUG::2016-02-01 13:51:19,081::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'Host.ping' in bridge with True jsonrpc.Executor/0::DEBUG::2016-02-01 13:51:19,131::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'VM.migrationCreate' in bridge with [u'49bfc23b-79a6-4f6c-840c-95ea04562513', {u'status': u'Up', u'acpiEnable': u'true', u'emulatedMachine': u'pc-i440fx-rhel7.2.0', u'afterMigrationStatus': u'', u'tabletEnable': u'true', u'vmId': u'49bfc23b-79a6-4f6c-840c-95ea04562513', u'memGuaranteedSize': 1024, u'transparentHugePages': u'true', u'displayPort': u'5904', u'displaySecurePort': u'-1', u'spiceSslCipherSuite': u'DEFAULT', u'cpuType': u'Nehalem', u'smp': u'2', u'migrationDest': u'libvirt', u'guestNumaNodes': [{u'nodeIndex': 0, u'cpus': u'0,1', u'memory': u'1024'}], u'smartcardEnable': u'false', u'numaTune': {u'nodeset': u'1,0', u'mode': u'interleave'}, u'custom': {u'device_90e6ab00-d9e0-4d13-9b2b-da53872f48d5device_bfec0f5c-150a-4e0c-94a8-56bb5ffd506e': u"VmDevice:{id='VmDeviceId:{deviceId='bfec0f5c-150a-4e0c-94a8-56bb5ffd506e', vmId='49bfc23b-79a6-4f6c-840c-95ea04562513'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false'}", u'device_90e6ab00-d9e0-4d13-9b2b-da53872f48d5device_bfec0f5c-150a-4e0c-94a8-56bb5ffd506edevice_185fa779-0f85-4a4e-9633-f8c2c81fc746': u"VmDevice:{id='VmDeviceId:{deviceId='185fa779-0f85-4a4e-9633-f8c2c81fc746', vmId='49bfc23b-79a6-4f6c-840c-95ea04562513'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false'}", u'device_90e6ab00-d9e0-4d13-9b2b-da53872f48d5': u"VmDevice:{id='VmDeviceId:{deviceId='90e6ab00-d9e0-4d13-9b2b-da53872f48d5', vmId='49bfc23b-79a6-4f6c-840c-95ea04562513'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false'}"}, u'vmType': u'kvm', u'_srcDomXML': u'<domain type=\'kvm\' id=\'7102\'>\n <name>serv-vpn-adms1</name>\n <uuid>49bfc23b-79a6-4f6c-840c-95ea04562513</uuid>\n <metadata xmlns:ovirt="http://ovirt.org/vm/tune/1.0">\n <ovirt:qos/>\n </metadata>\n <maxMemory slots=\'16\' unit=\'KiB\'>4294967296</maxMemory>\n <memory unit=\'KiB\'>1048576</memory>\n <currentMemory unit=\'KiB\'>1048576</currentMemory>\n <vcpu placement=\'static\' current=\'2\'>16</vcpu>\n <cputune>\n <shares>1020</shares>\n </cputune>\n <numatune>\n <memory mode=\'interleave\' nodeset=\'0-1\'/>\n </numatune>\n <resource>\n <partition>/machine</partition>\n </resource>\n <sysinfo type=\'smbios\'>\n <system>\n <entry name=\'manufacturer\'>oVirt</entry>\n <entry name=\'product\'>oVirt Node</entry>\n <entry name=\'version\'>7-2.1511.el7.centos.2.10</entry>\n <entry name=\'serial\'>4C4C4544-0039-3410-8033-C7C04F383532</entry>\n <entry name=\'uuid\'>49bfc23b-79a6-4f6c-840c-95ea04562513</entry>\n </system>\n </sysinfo>\n <os>\n <type arch=\'x86_64\' machine=\'pc-i440fx-rhel7.2.0\'>hvm</type>\n <smbios mode=\'sysinfo\'/>\n </os>\n <features>\n <acpi/>\n </features>\n <cpu mode=\'custom\' match=\'exact\'>\n <model fallback=\'allow\'>Nehalem</model>\n <topology sockets=\'16\' cores=\'1\' threads=\'1\'/>\n <numa>\n <cell id=\'0\' cpus=\'0-1\' memory=\'1048576\' unit=\'KiB\'/>\n </numa>\n </cpu>\n <clock offset=\'variable\' adjustment=\'3600\' basis=\'utc\'>\n <timer name=\'rtc\' tickpolicy=\'catchup\'/>\n <timer name=\'pit\' tickpolicy=\'delay\'/>\n <timer name=\'hpet\' present=\'no\'/>\n </clock>\n <on_poweroff>destroy</on_poweroff>\n <on_reboot>restart</on_reboot>\n <on_crash>destroy</on_crash>\n <devices>\n <emulator>/usr/libexec/qemu-kvm</emulator>\n <disk type=\'file\' device=\'cdrom\'>\n <driver name=\'qemu\' type=\'raw\'/>\n <source startupPolicy=\'optional\'/>\n <backingStore/>\n <target dev=\'hdc\' bus=\'ide\'/>\n <readonly/>\n <serial></serial>\n <alias name=\'ide0-1-0\'/>\n <address type=\'drive\' controller=\'0\' bus=\'1\' target=\'0\' unit=\'0\'/>\n </disk>\n <disk type=\'block\' device=\'disk\' snapshot=\'no\'>\n <driver name=\'qemu\' type=\'qcow2\' cache=\'none\' error_policy=\'stop\' io=\'native\'/>\n <source dev=\'/rhev/data-center/00000002-0002-0002-0002-0000000003ca/f7bf2585-1f74-4655-9118-a790e9abc70d/images/3ccc0ba0-44f9-4a71-a3c5-193dd364c87e/75016b46-a4c3-46d2-8908-0fbc7204e9ac\'/>\n <backingStore/>\n <target dev=\'vda\' bus=\'virtio\'/>\n <serial>3ccc0ba0-44f9-4a71-a3c5-193dd364c87e</serial>\n <boot order=\'1\'/>\n <alias name=\'virtio-disk0\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x06\' function=\'0x0\'/>\n </disk>\n <controller type=\'scsi\' index=\'0\' model=\'virtio-scsi\'>\n <alias name=\'scsi0\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x04\' function=\'0x0\'/>\n </controller>\n <controller type=\'virtio-serial\' index=\'0\' ports=\'16\'>\n <alias name=\'virtio-serial0\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x05\' function=\'0x0\'/>\n </controller>\n <controller type=\'ide\' index=\'0\'>\n <alias name=\'ide\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x01\' function=\'0x1\'/>\n </controller>\n <controller type=\'usb\' index=\'0\'>\n <alias name=\'usb\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x01\' function=\'0x2\'/>\n </controller>\n <controller type=\'pci\' index=\'0\' model=\'pci-root\'>\n <alias name=\'pci.0\'/>\n </controller>\n <interface type=\'bridge\'>\n <mac address=\'00:1a:4a:cc:00:05\'/>\n <source bridge=\'ovirtmgmt\'/>\n <target dev=\'vnet6\'/>\n <model type=\'virtio\'/>\n <filterref filter=\'vdsm-no-mac-spoofing\'/>\n <link state=\'up\'/>\n <alias name=\'net0\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x03\' function=\'0x0\'/>\n </interface>\n <channel type=\'unix\'>\n <source mode=\'bind\' path=\'/var/lib/libvirt/qemu/channels/49bfc23b-79a6-4f6c-840c-95ea04562513.com.redhat.rhevm.vdsm\'/>\n <target type=\'virtio\' name=\'com.redhat.rhevm.vdsm\' state=\'connected\'/>\n <alias name=\'channel0\'/>\n <address type=\'virtio-serial\' controller=\'0\' bus=\'0\' port=\'1\'/>\n </channel>\n <channel type=\'unix\'>\n <source mode=\'bind\' path=\'/var/lib/libvirt/qemu/channels/49bfc23b-79a6-4f6c-840c-95ea04562513.org.qemu.guest_agent.0\'/>\n <target type=\'virtio\' name=\'org.qemu.guest_agent.0\' state=\'connected\'/>\n <alias name=\'channel1\'/>\n <address type=\'virtio-serial\' controller=\'0\' bus=\'0\' port=\'2\'/>\n </channel>\n <input type=\'tablet\' bus=\'usb\'>\n <alias name=\'input0\'/>\n </input>\n <input type=\'mouse\' bus=\'ps2\'/>\n <input type=\'keyboard\' bus=\'ps2\'/>\n <graphics type=\'vnc\' port=\'5904\' autoport=\'yes\' listen=\'192.168.12.17\' passwdValidTo=\'1970-01-01T00:00:01\'>\n <listen type=\'network\' address=\'192.168.12.17\' network=\'vdsm-ovirtmgmt\'/>\n </graphics>\n <video>\n <model type=\'cirrus\' vram=\'32768\' heads=\'1\'/>\n <alias name=\'video0\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x02\' function=\'0x0\'/>\n </video>\n <memballoon model=\'virtio\'>\n <alias name=\'balloon0\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x07\' function=\'0x0\'/>\n </memballoon>\n </devices>\n</domain>\n', u'memSize': 1024, u'smpCoresPerSocket': u'1', u'vmName': u'serv-vpn-adms1', u'nice': u'0', u'username': u'None', u'maxMemSize': 4194304, u'bootMenuEnable': u'false', u'nicModel': u'rtl8139,pv', u'displayIp': u'192.168.12.17', u'keyboardLayout': u'fr', u'smpThreadsPerCore': u'1', u'guestDiskMapping': {u'3ccc0ba0-44f9-4a71-a': {u'name': u'/dev/vda'}, u'QEMU_DVD-ROM': {u'name': u'/dev/sr0'}}, u'clientIp': u'', u'guestFQDN': u'serv-vpn-adms1.sdis.isere.fr', u'elapsedTimeOffset': 11588.329982042313, u'kvmEnable': u'true', u'pitReinjection': u'false', u'spiceSecureChannels': u'smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard', u'displayNetwork': u'ovirtmgmt', u'devices': [{u'target': 1048576, u'alias': u'balloon0', u'specParams': {u'model': u'virtio'}, u'deviceId': u'57055dc0-e7a1-434d-a6f7-5ebe141511b5', u'address': {u'slot': u'0x07', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'device': u'memballoon', u'type': u'balloon'}, {u'device': u'unix', u'alias': u'channel0', u'type': u'channel', u'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'1'}}, {u'device': u'unix', u'alias': u'channel1', u'type': u'channel', u'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'2'}}, {u'index': u'0', u'alias': u'scsi0', u'specParams': {}, u'deviceId': u'b624e28c-7a07-4c1f-b819-7f2afad89079', u'address': {u'slot': u'0x04', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'device': u'scsi', u'model': u'virtio-scsi', u'type': u'controller'}, {u'alias': u'virtio-serial0', u'specParams': {}, u'deviceId': u'a54ef659-aefc-4c64-8211-02ae09fa45e4', u'address': {u'slot': u'0x05', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'device': u'virtio-serial', u'type': u'controller'}, {u'device': u'usb', u'alias': u'usb', u'type': u'controller', u'address': {u'slot': u'0x01', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x2'}}, {u'device': u'ide', u'alias': u'ide', u'type': u'controller', u'address': {u'slot': u'0x01', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x1'}}, {u'alias': u'video0', u'specParams': {u'vram': u'32768', u'heads': u'1'}, u'deviceId': u'eb8ae28f-2505-48fd-b850-6043074d32ff', u'address': {u'slot': u'0x02', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'device': u'cirrus', u'type': u'video'}, {u'device': u'vnc', u'specParams': {u'displayNetwork': u'ovirtmgmt', u'displayIp': u'192.168.12.17'}, u'type': u'graphics', u'deviceId': u'00000032-0032-0032-0032-00000000013c', u'port': u'5904'}, {u'nicModel': u'pv', u'macAddr': u'00:1a:4a:cc:00:05', u'linkActive': True, u'network': u'ovirtmgmt', u'alias': u'net0', u'filter': u'vdsm-no-mac-spoofing', u'specParams': {u'inbound': {}, u'outbound': {}}, u'deviceId': u'8a209ba0-46e0-434b-9b90-461e32959cda', u'address': {u'slot': u'0x03', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'device': u'bridge', u'type': u'interface', u'name': u'vnet6'}, {u'index': u'2', u'iface': u'ide', u'name': u'hdc', u'alias': u'ide0-1-0', u'specParams': {u'path': u''}, u'readonly': u'True', u'deviceId': u'1b7bc6a6-45db-40d1-a80a-11a3db93f25a', u'address': {u'bus': u'1', u'controller': u'0', u'type': u'drive', u'target': u'0', u'unit': u'0'}, u'device': u'cdrom', u'shared': u'false', u'path': u'', u'type': u'disk'}, {u'poolID': u'00000002-0002-0002-0002-0000000003ca', u'reqsize': u'0', u'index': 0, u'iface': u'virtio', u'apparentsize': u'6442450944', u'alias': u'virtio-disk0', u'imageID': u'3ccc0ba0-44f9-4a71-a3c5-193dd364c87e', u'readonly': u'False', u'shared': u'false', u'truesize': u'6442450944', u'type': u'disk', u'domainID': u'f7bf2585-1f74-4655-9118-a790e9abc70d', u'volumeInfo': {u'domainID': u'f7bf2585-1f74-4655-9118-a790e9abc70d', u'volType': u'path', u'leaseOffset': 123731968, u'volumeID': u'75016b46-a4c3-46d2-8908-0fbc7204e9ac', u'leasePath': u'/dev/f7bf2585-1f74-4655-9118-a790e9abc70d/leases', u'imageID': u'3ccc0ba0-44f9-4a71-a3c5-193dd364c87e', u'path': u'/rhev/data-center/mnt/blockSD/f7bf2585-1f74-4655-9118-a790e9abc70d/images/3ccc0ba0-44f9-4a71-a3c5-193dd364c87e/75016b46-a4c3-46d2-8908-0fbc7204e9ac'}, u'format': u'cow', u'deviceId': u'3ccc0ba0-44f9-4a71-a3c5-193dd364c87e', u'address': {u'slot': u'0x06', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'device': u'disk', u'path': u'/rhev/data-center/00000002-0002-0002-0002-0000000003ca/f7bf2585-1f74-4655-9118-a790e9abc70d/images/3ccc0ba0-44f9-4a71-a3c5-193dd364c87e/75016b46-a4c3-46d2-8908-0fbc7204e9ac', u'propagateErrors': u'off', u'optional': u'false', u'name': u'vda', u'bootOrder': u'1', u'volumeID': u'75016b46-a4c3-46d2-8908-0fbc7204e9ac', u'specParams': {}, u'volumeChain': [{u'domainID': u'f7bf2585-1f74-4655-9118-a790e9abc70d', u'volType': u'path', u'leaseOffset': 123731968, u'volumeID': u'75016b46-a4c3-46d2-8908-0fbc7204e9ac', u'leasePath': u'/dev/f7bf2585-1f74-4655-9118-a790e9abc70d/leases', u'imageID': u'3ccc0ba0-44f9-4a71-a3c5-193dd364c87e', u'path': u'/rhev/data-center/mnt/blockSD/f7bf2585-1f74-4655-9118-a790e9abc70d/images/3ccc0ba0-44f9-4a71-a3c5-193dd364c87e/75016b46-a4c3-46d2-8908-0fbc7204e9ac'}]}], u'display': u'vnc', u'timeOffset': u'3600', u'maxVCpus': u'16', u'guestIPs': u'192.168.12.24', u'statusTime': u'4806990640', u'maxMemSlots': 16}] jsonrpc.Executor/0::DEBUG::2016-02-01 13:51:19,133::API::581::vds::(migrationCreate) Migration create jsonrpc.Executor/0::DEBUG::2016-02-01 13:51:19,138::vm::4633::virt.vm::(waitForMigrationDestinationPrepare) vmId=`49bfc23b-79a6-4f6c-840c-95ea04562513`::migration destination: waiting for VM creation Thread-776::DEBUG::2016-02-01 13:51:19,138::vm::696::virt.vm::(_startUnderlyingVm) vmId=`49bfc23b-79a6-4f6c-840c-95ea04562513`::Start jsonrpc.Executor/0::DEBUG::2016-02-01 13:51:19,138::vm::4638::virt.vm::(waitForMigrationDestinationPrepare) vmId=`49bfc23b-79a6-4f6c-840c-95ea04562513`::migration destination: waiting 36s for path preparation Thread-776::INFO::2016-02-01 13:51:19,138::vm::1840::virt.vm::(_run) vmId=`49bfc23b-79a6-4f6c-840c-95ea04562513`::VM wrapper has started Thread-776::DEBUG::2016-02-01 13:51:19,139::task::595::Storage.TaskManager.Task::(_updateState) Task=`dece8635-fd4a-4c52-8c36-fda24ae3482c`::moving from state init -> state preparing Thread-776::INFO::2016-02-01 13:51:19,140::logUtils::48::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID=u'f7bf2585-1f74-4655-9118-a790e9abc70d', spUUID=u'00000002-0002-0002-0002-0000000003ca', imgUUID=u'3ccc0ba0-44f9-4a71-a3c5-193dd364c87e', volUUID=u'75016b46-a4c3-46d2-8908-0fbc7204e9ac', options=None) Thread-776::INFO::2016-02-01 13:51:19,140::logUtils::51::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '6442450944', 'apparentsize': '6442450944'} Thread-776::DEBUG::2016-02-01 13:51:19,140::task::1191::Storage.TaskManager.Task::(prepare) Task=`dece8635-fd4a-4c52-8c36-fda24ae3482c`::finished: {'truesize': '6442450944', 'apparentsize': '6442450944'} Thread-776::DEBUG::2016-02-01 13:51:19,140::task::595::Storage.TaskManager.Task::(_updateState) Task=`dece8635-fd4a-4c52-8c36-fda24ae3482c`::moving from state preparing -> state finished Thread-776::DEBUG::2016-02-01 13:51:19,140::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-776::DEBUG::2016-02-01 13:51:19,140::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-776::DEBUG::2016-02-01 13:51:19,141::task::993::Storage.TaskManager.Task::(_decref) Task=`dece8635-fd4a-4c52-8c36-fda24ae3482c`::ref 0 aborting False Thread-776::INFO::2016-02-01 13:51:19,141::clientIF::375::vds::(prepareVolumePath) prepared volume path: Thread-776::DEBUG::2016-02-01 13:51:19,141::task::595::Storage.TaskManager.Task::(_updateState) Task=`e30c4c19-556a-424e-b709-e054c0e9aa17`::moving from state init -> state preparing Thread-776::INFO::2016-02-01 13:51:19,141::logUtils::48::dispatcher::(wrapper) Run and protect: prepareImage(sdUUID=u'f7bf2585-1f74-4655-9118-a790e9abc70d', spUUID=u'00000002-0002-0002-0002-0000000003ca', imgUUID=u'3ccc0ba0-44f9-4a71-a3c5-193dd364c87e', leafUUID=u'75016b46-a4c3-46d2-8908-0fbc7204e9ac') Thread-776::DEBUG::2016-02-01 13:51:19,142::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.f7bf2585-1f74-4655-9118-a790e9abc70d`ReqID=`e66e1626-06ba-4cd5-b8ee-5f22c6f9215f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3213' at 'prepareImage' Thread-776::DEBUG::2016-02-01 13:51:19,142::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.f7bf2585-1f74-4655-9118-a790e9abc70d' for lock type 'shared' Thread-776::DEBUG::2016-02-01 13:51:19,142::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.f7bf2585-1f74-4655-9118-a790e9abc70d' is free. Now locking as 'shared' (1 active user) Thread-776::DEBUG::2016-02-01 13:51:19,142::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.f7bf2585-1f74-4655-9118-a790e9abc70d`ReqID=`e66e1626-06ba-4cd5-b8ee-5f22c6f9215f`::Granted request Thread-776::DEBUG::2016-02-01 13:51:19,142::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`e30c4c19-556a-424e-b709-e054c0e9aa17`::_resourcesAcquired: Storage.f7bf2585-1f74-4655-9118-a790e9abc70d (shared) Thread-776::DEBUG::2016-02-01 13:51:19,143::task::993::Storage.TaskManager.Task::(_decref) Task=`e30c4c19-556a-424e-b709-e054c0e9aa17`::ref 1 aborting False Thread-776::DEBUG::2016-02-01 13:51:19,143::lvm::427::Storage.OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex Thread-776::DEBUG::2016-02-01 13:51:19,143::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-15 /usr/bin/sudo -n /usr/sbin/lvm lvs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600601600c302700fae12990bc5de411|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags f7bf2585-1f74-4655-9118-a790e9abc70d (cwd None) Thread-776::DEBUG::2016-02-01 13:51:19,264::lvm::290::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n'; <rc> = 0 Thread-776::DEBUG::2016-02-01 13:51:19,265::lvm::462::Storage.LVM::(_reloadlvs) lvs reloaded Thread-776::DEBUG::2016-02-01 13:51:19,265::lvm::462::Storage.OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex Thread-776::DEBUG::2016-02-01 13:51:19,266::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-15 /usr/bin/sudo -n /usr/sbin/lvm lvchange --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600601600c302700fae12990bc5de411|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --autobackup n --available y f7bf2585-1f74-4655-9118-a790e9abc70d/75016b46-a4c3-46d2-8908-0fbc7204e9ac (cwd None) Thread-776::DEBUG::2016-02-01 13:51:19,524::lvm::290::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n'; <rc> = 0 Thread-776::DEBUG::2016-02-01 13:51:19,525::lvm::513::Storage.OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex Thread-776::DEBUG::2016-02-01 13:51:19,525::lvm::525::Storage.OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex Thread-776::DEBUG::2016-02-01 13:51:19,525::fileUtils::143::Storage.fileUtils::(createdir) Creating directory: /var/run/vdsm/storage/f7bf2585-1f74-4655-9118-a790e9abc70d/3ccc0ba0-44f9-4a71-a3c5-193dd364c87e mode: None Thread-776::WARNING::2016-02-01 13:51:19,525::fileUtils::152::Storage.fileUtils::(createdir) Dir /var/run/vdsm/storage/f7bf2585-1f74-4655-9118-a790e9abc70d/3ccc0ba0-44f9-4a71-a3c5-193dd364c87e already exists Thread-776::DEBUG::2016-02-01 13:51:19,526::blockSD::1101::Storage.StorageDomain::(createImageLinks) Creating symlink from /dev/f7bf2585-1f74-4655-9118-a790e9abc70d/75016b46-a4c3-46d2-8908-0fbc7204e9ac to /var/run/vdsm/storage/f7bf2585-1f74-4655-9118-a790e9abc70d/3ccc0ba0-44f9-4a71-a3c5-193dd364c87e/75016b46-a4c3-46d2-8908-0fbc7204e9ac Thread-776::DEBUG::2016-02-01 13:51:19,526::blockSD::1106::Storage.StorageDomain::(createImageLinks) img run vol already exists: /var/run/vdsm/storage/f7bf2585-1f74-4655-9118-a790e9abc70d/3ccc0ba0-44f9-4a71-a3c5-193dd364c87e/75016b46-a4c3-46d2-8908-0fbc7204e9ac Thread-776::DEBUG::2016-02-01 13:51:19,526::blockSD::1071::Storage.StorageDomain::(linkBCImage) Creating symlink from /var/run/vdsm/storage/f7bf2585-1f74-4655-9118-a790e9abc70d/3ccc0ba0-44f9-4a71-a3c5-193dd364c87e to /rhev/data-center/00000002-0002-0002-0002-0000000003ca/f7bf2585-1f74-4655-9118-a790e9abc70d/images/3ccc0ba0-44f9-4a71-a3c5-193dd364c87e Thread-776::DEBUG::2016-02-01 13:51:19,526::blockSD::1077::Storage.StorageDomain::(linkBCImage) path to image directory already exists: /rhev/data-center/00000002-0002-0002-0002-0000000003ca/f7bf2585-1f74-4655-9118-a790e9abc70d/images/3ccc0ba0-44f9-4a71-a3c5-193dd364c87e Thread-776::DEBUG::2016-02-01 13:51:19,527::lvm::427::Storage.OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex Thread-776::DEBUG::2016-02-01 13:51:19,527::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-15 /usr/bin/sudo -n /usr/sbin/lvm lvs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600601600c302700fae12990bc5de411|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags f7bf2585-1f74-4655-9118-a790e9abc70d (cwd None) Thread-776::DEBUG::2016-02-01 13:51:19,645::lvm::290::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n'; <rc> = 0 Thread-776::DEBUG::2016-02-01 13:51:19,646::lvm::462::Storage.LVM::(_reloadlvs) lvs reloaded Thread-776::DEBUG::2016-02-01 13:51:19,646::lvm::462::Storage.OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex Thread-776::INFO::2016-02-01 13:51:19,647::logUtils::51::dispatcher::(wrapper) Run and protect: prepareImage, Return response: {'info': {'domainID': u'f7bf2585-1f74-4655-9118-a790e9abc70d', 'volType': 'path', 'leaseOffset': 123731968, 'path': u'/rhev/data-center/mnt/blockSD/f7bf2585-1f74-4655-9118-a790e9abc70d/images/3ccc0ba0-44f9-4a71-a3c5-193dd364c87e/75016b46-a4c3-46d2-8908-0fbc7204e9ac', 'volumeID': '75016b46-a4c3-46d2-8908-0fbc7204e9ac', 'leasePath': '/dev/f7bf2585-1f74-4655-9118-a790e9abc70d/leases', 'imageID': u'3ccc0ba0-44f9-4a71-a3c5-193dd364c87e'}, 'path': u'/rhev/data-center/00000002-0002-0002-0002-0000000003ca/f7bf2585-1f74-4655-9118-a790e9abc70d/images/3ccc0ba0-44f9-4a71-a3c5-193dd364c87e/75016b46-a4c3-46d2-8908-0fbc7204e9ac', 'imgVolumesInfo': [{'domainID': u'f7bf2585-1f74-4655-9118-a790e9abc70d', 'volType': 'path', 'leaseOffset': 123731968, 'path': u'/rhev/data-center/mnt/blockSD/f7bf2585-1f74-4655-9118-a790e9abc70d/images/3ccc0ba0-44f9-4a71-a3c5-193dd364c87e/75016b46-a4c3-46d2-8908-0fbc7204e9ac', 'volumeID': '75016b46-a4c3-46d2-8908-0fbc7204e9ac', 'leasePath': '/dev/f7bf2585-1f74-4655-9118-a790e9abc70d/leases', 'imageID': u'3ccc0ba0-44f9-4a71-a3c5-193dd364c87e'}]} Thread-776::DEBUG::2016-02-01 13:51:19,647::task::1191::Storage.TaskManager.Task::(prepare) Task=`e30c4c19-556a-424e-b709-e054c0e9aa17`::finished: {'info': {'domainID': u'f7bf2585-1f74-4655-9118-a790e9abc70d', 'volType': 'path', 'leaseOffset': 123731968, 'path': u'/rhev/data-center/mnt/blockSD/f7bf2585-1f74-4655-9118-a790e9abc70d/images/3ccc0ba0-44f9-4a71-a3c5-193dd364c87e/75016b46-a4c3-46d2-8908-0fbc7204e9ac', 'volumeID': '75016b46-a4c3-46d2-8908-0fbc7204e9ac', 'leasePath': '/dev/f7bf2585-1f74-4655-9118-a790e9abc70d/leases', 'imageID': u'3ccc0ba0-44f9-4a71-a3c5-193dd364c87e'}, 'path': u'/rhev/data-center/00000002-0002-0002-0002-0000000003ca/f7bf2585-1f74-4655-9118-a790e9abc70d/images/3ccc0ba0-44f9-4a71-a3c5-193dd364c87e/75016b46-a4c3-46d2-8908-0fbc7204e9ac', 'imgVolumesInfo': [{'domainID': u'f7bf2585-1f74-4655-9118-a790e9abc70d', 'volType': 'path', 'leaseOffset': 123731968, 'path': u'/rhev/data-center/mnt/blockSD/f7bf2585-1f74-4655-9118-a790e9abc70d/images/3ccc0ba0-44f9-4a71-a3c5-193dd364c87e/75016b46-a4c3-46d2-8908-0fbc7204e9ac', 'volumeID': '75016b46-a4c3-46d2-8908-0fbc7204e9ac', 'leasePath': '/dev/f7bf2585-1f74-4655-9118-a790e9abc70d/leases', 'imageID': u'3ccc0ba0-44f9-4a71-a3c5-193dd364c87e'}]} Thread-776::DEBUG::2016-02-01 13:51:19,647::task::595::Storage.TaskManager.Task::(_updateState) Task=`e30c4c19-556a-424e-b709-e054c0e9aa17`::moving from state preparing -> state finished Thread-776::DEBUG::2016-02-01 13:51:19,647::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.f7bf2585-1f74-4655-9118-a790e9abc70d': < ResourceRef 'Storage.f7bf2585-1f74-4655-9118-a790e9abc70d', isValid: 'True' obj: 'None'>} Thread-776::DEBUG::2016-02-01 13:51:19,647::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-776::DEBUG::2016-02-01 13:51:19,647::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.f7bf2585-1f74-4655-9118-a790e9abc70d' Thread-776::DEBUG::2016-02-01 13:51:19,648::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.f7bf2585-1f74-4655-9118-a790e9abc70d' (0 active users) Thread-776::DEBUG::2016-02-01 13:51:19,648::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.f7bf2585-1f74-4655-9118-a790e9abc70d' is free, finding out if anyone is waiting for it. Thread-776::DEBUG::2016-02-01 13:51:19,648::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f7bf2585-1f74-4655-9118-a790e9abc70d', Clearing records. Thread-776::DEBUG::2016-02-01 13:51:19,648::task::993::Storage.TaskManager.Task::(_decref) Task=`e30c4c19-556a-424e-b709-e054c0e9aa17`::ref 0 aborting False Thread-776::INFO::2016-02-01 13:51:19,648::clientIF::375::vds::(prepareVolumePath) prepared volume path: /rhev/data-center/00000002-0002-0002-0002-0000000003ca/f7bf2585-1f74-4655-9118-a790e9abc70d/images/3ccc0ba0-44f9-4a71-a3c5-193dd364c87e/75016b46-a4c3-46d2-8908-0fbc7204e9ac Thread-776::DEBUG::2016-02-01 13:51:19,653::core::51::virt.vm::(__init__) vmId=`49bfc23b-79a6-4f6c-840c-95ea04562513`::Ignoring param (target, 1048576) in Balloon jsonrpc.Executor/0::DEBUG::2016-02-01 13:51:19,665::API::599::vds::(migrationCreate) Destination VM creation succeeded jsonrpc.Executor/0::DEBUG::2016-02-01 13:51:19,665::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'VM.migrationCreate' in bridge with {'migrationPort': 0, 'params': {u'username': u'None', u'acpiEnable': u'true', u'emulatedMachine': u'pc-i440fx-rhel7.2.0', u'afterMigrationStatus': u'', u'tabletEnable': u'true', 'pid': '0', u'memGuaranteedSize': 1024, u'transparentHugePages': u'true', u'displaySecurePort': '-1', u'spiceSslCipherSuite': u'DEFAULT', u'cpuType': u'Nehalem', u'custom': {u'device_90e6ab00-d9e0-4d13-9b2b-da53872f48d5device_bfec0f5c-150a-4e0c-94a8-56bb5ffd506e': u"VmDevice:{id='VmDeviceId:{deviceId='bfec0f5c-150a-4e0c-94a8-56bb5ffd506e', vmId='49bfc23b-79a6-4f6c-840c-95ea04562513'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false'}", u'device_90e6ab00-d9e0-4d13-9b2b-da53872f48d5device_bfec0f5c-150a-4e0c-94a8-56bb5ffd506edevice_185fa779-0f85-4a4e-9633-f8c2c81fc746': u"VmDevice:{id='VmDeviceId:{deviceId='185fa779-0f85-4a4e-9633-f8c2c81fc746', vmId='49bfc23b-79a6-4f6c-840c-95ea04562513'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false'}", u'device_90e6ab00-d9e0-4d13-9b2b-da53872f48d5': u"VmDevice:{id='VmDeviceId:{deviceId='90e6ab00-d9e0-4d13-9b2b-da53872f48d5', vmId='49bfc23b-79a6-4f6c-840c-95ea04562513'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false'}"}, u'migrationDest': u'libvirt', u'guestNumaNodes': [{u'nodeIndex': 0, u'cpus': u'0,1', u'memory': u'1024'}], u'numaTune': {u'nodeset': u'1,0', u'mode': u'interleave'}, u'smp': u'2', u'vmType': u'kvm', u'smpThreadsPerCore': u'1', u'memSize': 1024, u'smpCoresPerSocket': u'1', u'vmName': u'serv-vpn-adms1', u'nice': u'0', u'guestFQDN': u'serv-vpn-adms1.sdis.isere.fr', u'maxMemSize': 4194304, u'bootMenuEnable': u'false', u'guestDiskMapping': {}, u'vmId': u'49bfc23b-79a6-4f6c-840c-95ea04562513', u'displayIp': '192.168.12.212', u'displayPort': '-1', u'guestIPs': u'192.168.12.24', u'smartcardEnable': u'false', u'spiceSecureChannels': u'smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard', u'maxMemSlots': 16, u'nicModel': u'rtl8139,pv', u'keyboardLayout': u'fr', u'kvmEnable': u'true', u'pitReinjection': u'false', u'displayNetwork': u'ovirtmgmt', u'devices': [{u'target': 1048576, u'specParams': {u'model': u'virtio'}, u'alias': u'balloon0', 'deviceType': u'balloon', u'deviceId': u'57055dc0-e7a1-434d-a6f7-5ebe141511b5', u'address': {u'slot': u'0x07', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'device': u'memballoon', u'type': u'balloon'}, {u'device': u'unix', u'alias': u'channel0', 'deviceType': u'channel', u'type': u'channel', u'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'1'}}, {u'device': u'unix', u'alias': u'channel1', 'deviceType': u'channel', u'type': u'channel', u'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'2'}}, {u'index': u'0', u'alias': u'scsi0', u'specParams': {}, 'deviceType': u'controller', u'deviceId': u'b624e28c-7a07-4c1f-b819-7f2afad89079', u'address': {u'slot': u'0x04', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'device': u'scsi', u'model': u'virtio-scsi', u'type': u'controller'}, {u'specParams': {}, u'alias': u'virtio-serial0', 'deviceType': u'controller', u'deviceId': u'a54ef659-aefc-4c64-8211-02ae09fa45e4', u'address': {u'slot': u'0x05', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'device': u'virtio-serial', u'type': u'controller'}, {u'device': u'usb', u'alias': u'usb', 'deviceType': u'controller', u'type': u'controller', u'address': {u'slot': u'0x01', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x2'}}, {u'device': u'ide', u'alias': u'ide', 'deviceType': u'controller', u'type': u'controller', u'address': {u'slot': u'0x01', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x1'}}, {u'specParams': {u'vram': u'32768', u'heads': u'1'}, u'alias': u'video0', 'deviceType': u'video', u'deviceId': u'eb8ae28f-2505-48fd-b850-6043074d32ff', u'address': {u'slot': u'0x02', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'device': u'cirrus', u'type': u'video'}, {u'specParams': {u'displayNetwork': u'ovirtmgmt', u'displayIp': u'192.168.12.17'}, 'deviceType': u'graphics', u'deviceId': u'00000032-0032-0032-0032-00000000013c', u'device': u'vnc', u'type': u'graphics', u'port': u'5904'}, {u'nicModel': u'pv', u'macAddr': u'00:1a:4a:cc:00:05', u'linkActive': True, u'network': u'ovirtmgmt', u'specParams': {u'inbound': {}, u'outbound': {}}, u'filter': u'vdsm-no-mac-spoofing', u'alias': u'net0', 'deviceType': u'interface', u'deviceId': u'8a209ba0-46e0-434b-9b90-461e32959cda', u'address': {u'slot': u'0x03', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'device': u'bridge', u'type': u'interface', u'name': u'vnet6'}, {u'index': u'2', u'iface': u'ide', u'name': u'hdc', u'alias': u'ide0-1-0', u'specParams': {u'path': u''}, u'readonly': u'True', 'deviceType': u'disk', u'deviceId': u'1b7bc6a6-45db-40d1-a80a-11a3db93f25a', u'address': {u'bus': u'1', u'controller': u'0', u'type': u'drive', u'target': u'0', u'unit': u'0'}, u'device': u'cdrom', u'shared': u'false', u'path': u'', u'type': u'disk'}, {u'poolID': u'00000002-0002-0002-0002-0000000003ca', u'volumeInfo': {u'domainID': u'f7bf2585-1f74-4655-9118-a790e9abc70d', u'volType': u'path', u'leaseOffset': 123731968, u'volumeID': u'75016b46-a4c3-46d2-8908-0fbc7204e9ac', u'leasePath': u'/dev/f7bf2585-1f74-4655-9118-a790e9abc70d/leases', u'imageID': u'3ccc0ba0-44f9-4a71-a3c5-193dd364c87e', u'path': u'/rhev/data-center/mnt/blockSD/f7bf2585-1f74-4655-9118-a790e9abc70d/images/3ccc0ba0-44f9-4a71-a3c5-193dd364c87e/75016b46-a4c3-46d2-8908-0fbc7204e9ac'}, u'index': 0, u'iface': u'virtio', u'apparentsize': u'6442450944', u'specParams': {}, u'imageID': u'3ccc0ba0-44f9-4a71-a3c5-193dd364c87e', u'readonly': u'False', 'deviceType': u'disk', u'shared': u'false', u'truesize': u'6442450944', u'type': u'disk', u'domainID': u'f7bf2585-1f74-4655-9118-a790e9abc70d', u'reqsize': u'0', u'format': u'cow', u'deviceId': u'3ccc0ba0-44f9-4a71-a3c5-193dd364c87e', u'address': {u'slot': u'0x06', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'device': u'disk', u'path': u'/rhev/data-center/00000002-0002-0002-0002-0000000003ca/f7bf2585-1f74-4655-9118-a790e9abc70d/images/3ccc0ba0-44f9-4a71-a3c5-193dd364c87e/75016b46-a4c3-46d2-8908-0fbc7204e9ac', u'propagateErrors': u'off', u'optional': u'false', u'name': u'vda', u'bootOrder': u'1', u'volumeID': u'75016b46-a4c3-46d2-8908-0fbc7204e9ac', u'alias': u'virtio-disk0', u'volumeChain': [{u'domainID': u'f7bf2585-1f74-4655-9118-a790e9abc70d', u'volType': u'path', u'leaseOffset': 123731968, u'volumeID': u'75016b46-a4c3-46d2-8908-0fbc7204e9ac', u'leasePath': u'/dev/f7bf2585-1f74-4655-9118-a790e9abc70d/leases', u'imageID': u'3ccc0ba0-44f9-4a71-a3c5-193dd364c87e', u'path': u'/rhev/data-center/mnt/blockSD/f7bf2585-1f74-4655-9118-a790e9abc70d/images/3ccc0ba0-44f9-4a71-a3c5-193dd364c87e/75016b46-a4c3-46d2-8908-0fbc7204e9ac'}]}], u'status': 'Migration Destination', u'timeOffset': u'3600', u'maxVCpus': u'16', 'clientIp': u'', u'statusTime': '4349227970', u'display': 'vnc'}} Thread-776::DEBUG::2016-02-01 13:51:19,668::vm::2857::virt.vm::(_waitForUnderlyingMigration) vmId=`49bfc23b-79a6-4f6c-840c-95ea04562513`::Waiting 21600 seconds for end of migration jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:19,721::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'VM.destroy' in bridge with [u'49bfc23b-79a6-4f6c-840c-95ea04562513'] jsonrpc.Executor/4::INFO::2016-02-01 13:51:19,721::API::341::vds::(destroy) vmContainerLock acquired by vm 49bfc23b-79a6-4f6c-840c-95ea04562513 jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:19,721::vm::3861::virt.vm::(destroy) vmId=`49bfc23b-79a6-4f6c-840c-95ea04562513`::destroy Called jsonrpc.Executor/4::INFO::2016-02-01 13:51:19,722::vm::3793::virt.vm::(releaseVm) vmId=`49bfc23b-79a6-4f6c-840c-95ea04562513`::Release VM resources jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:19,722::vmchannels::214::vds::(unregister) Delete fileno 38 from listener. jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:19,722::task::595::Storage.TaskManager.Task::(_updateState) Task=`8df4f819-2505-45f4-bb7c-e972ec7b880a`::moving from state init -> state preparing jsonrpc.Executor/4::INFO::2016-02-01 13:51:19,723::logUtils::48::dispatcher::(wrapper) Run and protect: teardownImage(sdUUID=u'f7bf2585-1f74-4655-9118-a790e9abc70d', spUUID=u'00000002-0002-0002-0002-0000000003ca', imgUUID=u'3ccc0ba0-44f9-4a71-a3c5-193dd364c87e', volUUID=None) jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:19,723::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.f7bf2585-1f74-4655-9118-a790e9abc70d`ReqID=`20dc729e-99cb-4813-9e71-a03c9f54d2a9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3265' at 'teardownImage' jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:19,723::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.f7bf2585-1f74-4655-9118-a790e9abc70d' for lock type 'shared' jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:19,723::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.f7bf2585-1f74-4655-9118-a790e9abc70d' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:19,723::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.f7bf2585-1f74-4655-9118-a790e9abc70d`ReqID=`20dc729e-99cb-4813-9e71-a03c9f54d2a9`::Granted request jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:19,724::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`8df4f819-2505-45f4-bb7c-e972ec7b880a`::_resourcesAcquired: Storage.f7bf2585-1f74-4655-9118-a790e9abc70d (shared) jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:19,724::task::993::Storage.TaskManager.Task::(_decref) Task=`8df4f819-2505-45f4-bb7c-e972ec7b880a`::ref 1 aborting False jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:19,724::lvm::427::Storage.OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:19,724::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-15 /usr/bin/sudo -n /usr/sbin/lvm lvs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600601600c302700fae12990bc5de411|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags f7bf2585-1f74-4655-9118-a790e9abc70d (cwd None) Thread-776::ERROR::2016-02-01 13:51:19,733::vm::752::virt.vm::(_startUnderlyingVm) vmId=`49bfc23b-79a6-4f6c-840c-95ea04562513`::Failed to start a migration destination vm Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 721, in _startUnderlyingVm self._completeIncomingMigration() File "/usr/share/vdsm/virt/vm.py", line 2830, in _completeIncomingMigration self._incomingMigrationFinished.isSet(), usedTimeout) File "/usr/share/vdsm/virt/vm.py", line 2889, in _attachLibvirtDomainAfterMigration raise MigrationError(e.get_error_message()) MigrationError: Domaine non trouvé : aucun domaine ne correspond à l'UUID '49bfc23b-79a6-4f6c-840c-95ea04562513' Thread-776::INFO::2016-02-01 13:51:19,733::vm::1324::virt.vm::(setDownStatus) vmId=`49bfc23b-79a6-4f6c-840c-95ea04562513`::Changed state to Down: VM failed to migrate (code=8) Thread-776::DEBUG::2016-02-01 13:51:19,734::__init__::206::jsonrpc.Notification::(emit) Sending event {"params": {"notify_time": 4349228570, "49bfc23b-79a6-4f6c-840c-95ea04562513": {"status": "Down", "timeOffset": "3600", "exitReason": 8, "exitMessage": "VM failed to migrate", "exitCode": 1}}, "jsonrpc": "2.0", "method": "|virt|VM_status|49bfc23b-79a6-4f6c-840c-95ea04562513"} jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:19,895::lvm::290::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n'; <rc> = 0 jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:19,896::lvm::462::Storage.LVM::(_reloadlvs) lvs reloaded jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:19,896::lvm::462::Storage.OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:19,897::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-15 /usr/bin/sudo -n /usr/sbin/lvm lvchange --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600601600c302700fae12990bc5de411|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --autobackup n --available n f7bf2585-1f74-4655-9118-a790e9abc70d/75016b46-a4c3-46d2-8908-0fbc7204e9ac (cwd None) jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:20,037::lvm::290::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n'; <rc> = 0 jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:20,037::lvm::513::Storage.OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:20,037::lvm::525::Storage.OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex jsonrpc.Executor/4::INFO::2016-02-01 13:51:20,038::logUtils::51::dispatcher::(wrapper) Run and protect: teardownImage, Return response: None jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:20,038::task::1191::Storage.TaskManager.Task::(prepare) Task=`8df4f819-2505-45f4-bb7c-e972ec7b880a`::finished: None jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:20,038::task::595::Storage.TaskManager.Task::(_updateState) Task=`8df4f819-2505-45f4-bb7c-e972ec7b880a`::moving from state preparing -> state finished jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:20,038::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.f7bf2585-1f74-4655-9118-a790e9abc70d': < ResourceRef 'Storage.f7bf2585-1f74-4655-9118-a790e9abc70d', isValid: 'True' obj: 'None'>} jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:20,038::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:20,039::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.f7bf2585-1f74-4655-9118-a790e9abc70d' jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:20,039::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.f7bf2585-1f74-4655-9118-a790e9abc70d' (0 active users) jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:20,039::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.f7bf2585-1f74-4655-9118-a790e9abc70d' is free, finding out if anyone is waiting for it. jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:20,039::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f7bf2585-1f74-4655-9118-a790e9abc70d', Clearing records. jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:20,039::task::993::Storage.TaskManager.Task::(_decref) Task=`8df4f819-2505-45f4-bb7c-e972ec7b880a`::ref 0 aborting False jsonrpc.Executor/4::WARNING::2016-02-01 13:51:20,040::utils::138::root::(rmFile) File: /var/lib/libvirt/qemu/channels/49bfc23b-79a6-4f6c-840c-95ea04562513.com.redhat.rhevm.vdsm already removed jsonrpc.Executor/4::WARNING::2016-02-01 13:51:20,040::utils::138::root::(rmFile) File: /var/lib/libvirt/qemu/channels/49bfc23b-79a6-4f6c-840c-95ea04562513.org.qemu.guest_agent.0 already removed jsonrpc.Executor/4::WARNING::2016-02-01 13:51:20,040::vm::1699::virt.vm::(_cleanupStatsCache) vmId=`49bfc23b-79a6-4f6c-840c-95ea04562513`::timestamp already removed from stats cache jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:20,041::task::595::Storage.TaskManager.Task::(_updateState) Task=`a27424c4-4cec-47cc-9f87-2dc651a9047f`::moving from state init -> state preparing jsonrpc.Executor/4::INFO::2016-02-01 13:51:20,041::logUtils::48::dispatcher::(wrapper) Run and protect: inappropriateDevices(thiefId=u'49bfc23b-79a6-4f6c-840c-95ea04562513') jsonrpc.Executor/4::INFO::2016-02-01 13:51:20,042::logUtils::51::dispatcher::(wrapper) Run and protect: inappropriateDevices, Return response: None jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:20,042::task::1191::Storage.TaskManager.Task::(prepare) Task=`a27424c4-4cec-47cc-9f87-2dc651a9047f`::finished: None jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:20,042::task::595::Storage.TaskManager.Task::(_updateState) Task=`a27424c4-4cec-47cc-9f87-2dc651a9047f`::moving from state preparing -> state finished jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:20,042::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:20,042::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:20,042::task::993::Storage.TaskManager.Task::(_decref) Task=`a27424c4-4cec-47cc-9f87-2dc651a9047f`::ref 0 aborting False jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:20,043::vm::3856::virt.vm::(deleteVm) vmId=`49bfc23b-79a6-4f6c-840c-95ea04562513`::Total desktops after destroy of 49bfc23b-79a6-4f6c-840c-95ea04562513 is 0 jsonrpc.Executor/4::DEBUG::2016-02-01 13:51:20,043::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'VM.destroy' in bridge with True VM Channels Listener::DEBUG::2016-02-01 13:51:20,456::vmchannels::121::vds::(_do_del_channels) fileno 38 was removed from listener. Thread-741::DEBUG::2016-02-01 13:51:25,286::blockSD::422::Storage.Misc.excCmd::(getReadDelay) /usr/bin/taskset --cpu-list 0-15 /usr/bin/dd if=/dev/b681cc9d-1afb-4f36-b394-af5cb644616f/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) Thread-741::DEBUG::2016-02-01 13:51:25,296::blockSD::422::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853544 s, 4.8 MB/s\n'; <rc> = 0 Thread-739::DEBUG::2016-02-01 13:51:25,383::fileSD::173::Storage.Misc.excCmd::(getReadDelay) /usr/bin/taskset --cpu-list 0-15 /usr/bin/dd if=/rhev/data-center/mnt/serv-mvm-adms01.sdis.isere.fr:_home_iso/c4475304-caab-48cd-adb2-c70b4af7a259/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) Thread-738::DEBUG::2016-02-01 13:51:25,392::fileSD::173::Storage.Misc.excCmd::(getReadDelay) /usr/bin/taskset --cpu-list 0-15 /usr/bin/dd if=/rhev/data-center/mnt/serv-mvm-adms01.sdis.isere.fr:_home_vmexports01/d3966e25-4872-41ee-81a7-121c7c9f7347/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) Thread-739::DEBUG::2016-02-01 13:51:25,401::fileSD::173::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n372 bytes (372 B) copied, 0.000478946 s, 777 kB/s\n'; <rc> = 0 Thread-738::DEBUG::2016-02-01 13:51:25,404::fileSD::173::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n385 bytes (385 B) copied, 0.000441253 s, 873 kB/s\n'; <rc> = 0 Thread-737::DEBUG::2016-02-01 13:51:25,631::blockSD::422::Storage.Misc.excCmd::(getReadDelay) /usr/bin/taskset --cpu-list 0-15 /usr/bin/dd if=/dev/f7bf2585-1f74-4655-9118-a790e9abc70d/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) Thread-737::DEBUG::2016-02-01 13:51:25,640::blockSD::422::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000600277 s, 6.8 MB/s\n'; <rc> = 0 jsonrpc.Executor/6::DEBUG::2016-02-01 13:51:31,809::task::595::Storage.TaskManager.Task::(_updateState) Task=`a98b2d80-a94f-4b87-988f-8e64bdea4c22`::moving from state init -> state preparing jsonrpc.Executor/6::INFO::2016-02-01 13:51:31,810::logUtils::48::dispatcher::(wrapper) Run and protect: repoStats(options=None) jsonrpc.Executor/6::INFO::2016-02-01 13:51:31,810::logUtils::51::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'f7bf2585-1f74-4655-9118-a790e9abc70d': {'code': 0, 'actual': True, 'version': 3, 'acquired': True, 'delay': '0.000600277', 'lastCheck': '6.2', 'valid': True}, u'd3966e25-4872-41ee-81a7-121c7c9f7347': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000441253', 'lastCheck': '6.4', 'valid': True}, u'c4475304-caab-48cd-adb2-c70b4af7a259': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000478946', 'lastCheck': '6.4', 'valid': True}, u'b681cc9d-1afb-4f36-b394-af5cb644616f': {'code': 0, 'actual': True, 'version': 3, 'acquired': True, 'delay': '0.000853544', 'lastCheck': '6.5', 'valid': True}} jsonrpc.Executor/6::DEBUG::2016-02-01 13:51:31,810::task::1191::Storage.TaskManager.Task::(prepare) Task=`a98b2d80-a94f-4b87-988f-8e64bdea4c22`::finished: {u'f7bf2585-1f74-4655-9118-a790e9abc70d': {'code': 0, 'actual': True, 'version': 3, 'acquired': True, 'delay': '0.000600277', 'lastCheck': '6.2', 'valid': True}, u'd3966e25-4872-41ee-81a7-121c7c9f7347': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000441253', 'lastCheck': '6.4', 'valid': True}, u'c4475304-caab-48cd-adb2-c70b4af7a259': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000478946', 'lastCheck': '6.4', 'valid': True}, u'b681cc9d-1afb-4f36-b394-af5cb644616f': {'code': 0, 'actual': True, 'version': 3, 'acquired': True, 'delay': '0.000853544', 'lastCheck': '6.5', 'valid': True}} jsonrpc.Executor/6::DEBUG::2016-02-01 13:51:31,810::task::595::Storage.TaskManager.Task::(_updateState) Task=`a98b2d80-a94f-4b87-988f-8e64bdea4c22`::moving from state preparing -> state finished jsonrpc.Executor/6::DEBUG::2016-02-01 13:51:31,810::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} jsonrpc.Executor/6::DEBUG::2016-02-01 13:51:31,810::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/6::DEBUG::2016-02-01 13:51:31,810::task::993::Storage.TaskManager.Task::(_decref) Task=`a98b2d80-a94f-4b87-988f-8e64bdea4c22`::ref 0 aborting False Reactor thread::INFO::2016-02-01 13:51:32,166::protocoldetector::72::ProtocolDetector.AcceptorImpl::(handle_accept) Accepting connection from 127.0.0.1:35110 Reactor thread::DEBUG::2016-02-01 13:51:32,173::protocoldetector::82::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::INFO::2016-02-01 13:51:32,173::protocoldetector::118::ProtocolDetector.Detector::(handle_read) Detected protocol xml from 127.0.0.1:35110 Reactor thread::DEBUG::2016-02-01 13:51:32,173::bindingxmlrpc::1297::XmlDetector::(handle_socket) xml over http detected from ('127.0.0.1', 35110) BindingXMLRPC::INFO::2016-02-01 13:51:32,174::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting request handler for 127.0.0.1:35110 Thread-778::INFO::2016-02-01 13:51:32,174::xmlrpc::84::vds.XMLRPCServer::(_process_requests) Request handler for 127.0.0.1:35110 started Thread-778::INFO::2016-02-01 13:51:32,175::xmlrpc::92::vds.XMLRPCServer::(_process_requests) Request handler for 127.0.0.1:35110 stopped
- engine : http://ur1.ca/ogvz9 2016-02-01 13:51:18,690 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (default task-185) [345d438f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[49bfc23b-79a6-4f6c-840c-95ea04562513=<VM, ACTION_TYPE_FAILED_VM_IS_BEING_MIGRATED$VmName serv-vpn-adms1>]', sharedLocks='null'}' 2016-02-01 13:51:18,863 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-185) [345d438f] Candidate host 'serv-vm-adms04' ('0f67d1cb-f98d-48e7-a878-5a94dddc1a7e') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'Migration' (correlation id: null) 2016-02-01 13:51:18,888 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (org.ovirt.thread.pool-8-thread-28) [345d438f] Running command: MigrateVmToServerCommand internal: false. Entities affected : ID: 49bfc23b-79a6-4f6c-840c-95ea04562513 Type: VMAction group MIGRATE_VM with role type USER 2016-02-01 13:51:19,061 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-8-thread-28) [345d438f] START, MigrateVDSCommand( MigrateVDSCommandParameters:{runAsync='true', hostId='0f67d1cb-f98d-48e7-a878-5a94dddc1a7e', vmId='49bfc23b-79a6-4f6c-840c-95ea04562513', srcHost='serv-vm-adms04', dstVdsId='3eaac3ad-9713-42e8-bfaf-0506b3a3c023', dstHost='serv-vm-adms02:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='false', migrateCompressed='false', consoleAddress='null'}), log id: 5813dd46 2016-02-01 13:51:19,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-8-thread-28) [345d438f] START, MigrateBrokerVDSCommand(HostName = serv-vm-adms04, MigrateVDSCommandParameters:{runAsync='true', hostId='0f67d1cb-f98d-48e7-a878-5a94dddc1a7e', vmId='49bfc23b-79a6-4f6c-840c-95ea04562513', srcHost='serv-vm-adms04', dstVdsId='3eaac3ad-9713-42e8-bfaf-0506b3a3c023', dstHost='serv-vm-adms02:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='false', migrateCompressed='false', consoleAddress='null'}), log id: 244bd7 2016-02-01 13:51:20,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-8-thread-28) [345d438f] FINISH, MigrateBrokerVDSCommand, log id: 244bd7 2016-02-01 13:51:20,100 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-8-thread-28) [345d438f] FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 5813dd46 2016-02-01 13:51:20,113 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-28) [345d438f] Correlation ID: 345d438f, Job ID: 9a03dfc8-8938-4adc-8e71-ea3bd0be4414, Call Stack: null, Custom Event ID: -1, Message: Migration started (VM: serv-vpn-adms1, Source: serv-vm-adms04, Destination: serv-vm-adms02, User: necarnot.fr). 2016-02-01 13:51:27,124 INFO [org.ovirt.engine.core.vdsbroker.VmAnalyzer] (DefaultQuartzScheduler_Worker-26) [401a804e] VM '49bfc23b-79a6-4f6c-840c-95ea04562513'(serv-vpn-adms1) moved from 'MigratingFrom' --> 'Up' 2016-02-01 13:51:27,124 INFO [org.ovirt.engine.core.vdsbroker.VmAnalyzer] (DefaultQuartzScheduler_Worker-26) [401a804e] Adding VM '49bfc23b-79a6-4f6c-840c-95ea04562513' to re-run list 2016-02-01 13:51:27,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler_Worker-26) [401a804e] START, FullListVDSCommand(HostName = , FullListVDSCommandParameters:{runAsync='true', hostId='0f67d1cb-f98d-48e7-a878-5a94dddc1a7e', vds='Host[,0f67d1cb-f98d-48e7-a878-5a94dddc1a7e]', vmIds='[49bfc23b-79a6-4f6c-840c-95ea04562513]'}), log id: 3a87a307 2016-02-01 13:51:28,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler_Worker-26) [401a804e] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.2.0, afterMigrationStatus=, tabletEnable=true, pid=107037, memGuaranteedSize=1024, transparentHugePages=true, displayPort=5904, displaySecurePort=-1, spiceSslCipherSuite=DEFAULT, cpuType=Nehalem, smp=2, guestNumaNodes=[Ljava.lang.Object;@247a7735, smartcardEnable=false, numaTune={nodeset=1,0, mode=interleave}, custom={device_90e6ab00-d9e0-4d13-9b2b-da53872f48d5device_bfec0f5c-150a-4e0c-94a8-56bb5ffd506e=VmDevice:{id='VmDeviceId:{deviceId='bfec0f5c-150a-4e0c-94a8-56bb5ffd506e', vmId='49bfc23b-79a6-4f6c-840c-95ea04562513'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false'}, device_90e6ab00-d9e0-4d13-9b2b-da53872f48d5device_bfec0f5c-150a-4e0c-94a8-56bb5ffd506edevice_185fa779-0f85-4a4e-9633-f8c2c81fc746=VmDevice:{id='VmDeviceId:{deviceId='185fa779-0f85-4a4e-9633-f8c2c81fc746', vmId='49bfc23b-79a6-4f6c-840c-95ea04562513'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false'}, device_90e6ab00-d9e0-4d13-9b2b-da53872f48d5=VmDevice:{id='VmDeviceId:{deviceId='90e6ab00-d9e0-4d13-9b2b-da53872f48d5', vmId='49bfc23b-79a6-4f6c-840c-95ea04562513'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false'}}, vmType=kvm, memSize=1024, smpCoresPerSocket=1, vmName=serv-vpn-adms1, nice=0, status=Up, maxMemSize=4194304, bootMenuEnable=false, vmId=49bfc23b-79a6-4f6c-840c-95ea04562513, displayIp=192.168.12.17, smpThreadsPerCore=1, guestDiskMapping={3ccc0ba0-44f9-4a71-a={name=/dev/vda}, QEMU_DVD-ROM={name=/dev/sr0}}, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, nicModel=rtl8139,pv, keyboardLayout=fr, kvmEnable=true, pitReinjection=false, displayNetwork=ovirtmgmt, devices=[Ljava.lang.Object;@64284b48, display=vnc, timeOffset=3600, maxVCpus=16, clientIp=, statusTime=4806998730, maxMemSlots=16}], log id: 3a87a307 2016-02-01 13:51:28,179 INFO [org.ovirt.engine.core.vdsbroker.VmsMonitoring] (DefaultQuartzScheduler_Worker-26) [401a804e] Received a vnc Device without an address when processing VM 49bfc23b-79a6-4f6c-840c-95ea04562513 devices, skipping device: {specParams={displayNetwork=ovirtmgmt, displayIp=192.168.12.17}, deviceType=graphics, deviceId=00000032-0032-0032-0032-00000000013c, device=vnc, type=graphics, port=5904} 2016-02-01 13:51:28,180 ERROR [org.ovirt.engine.core.vdsbroker.VmsMonitoring] (DefaultQuartzScheduler_Worker-26) [401a804e] VM '49bfc23b-79a6-4f6c-840c-95ea04562513' managed non pluggable device was removed unexpectedly from libvirt: 'VmDevice:{id='VmDeviceId:{deviceId='00000032-0032-0032-0032-00000000013c', vmId='49bfc23b-79a6-4f6c-840c-95ea04562513'}', device='vnc', type='GRAPHICS', bootOrder='0', specParams='[]', address='', managed='true', plugged='false', readOnly='false', deviceAlias='null', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false'}' 2016-02-01 13:51:28,197 ERROR [org.ovirt.engine.core.vdsbroker.VmsMonitoring] (DefaultQuartzScheduler_Worker-26) [401a804e] Rerun VM '49bfc23b-79a6-4f6c-840c-95ea04562513'. Called from VDS 'serv-vm-adms04' 2016-02-01 13:51:28,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (org.ovirt.thread.pool-8-thread-31) [401a804e] START, MigrateStatusVDSCommand(HostName = serv-vm-adms04, MigrateStatusVDSCommandParameters:{runAsync='true', hostId='0f67d1cb-f98d-48e7-a878-5a94dddc1a7e', vmId='49bfc23b-79a6-4f6c-840c-95ea04562513'}), log id: 434a1833 2016-02-01 13:51:28,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (org.ovirt.thread.pool-8-thread-31) [401a804e] FINISH, MigrateStatusVDSCommand, log id: 434a1833 2016-02-01 13:51:28,303 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-31) [401a804e] Correlation ID: 345d438f, Job ID: 9a03dfc8-8938-4adc-8e71-ea3bd0be4414, Call Stack: null, Custom Event ID: -1, Message: Migration failed (VM: serv-vpn-adms1, Source: serv-vm-adms04, Destination: serv-vm-adms02). 2016-02-01 13:51:28,305 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (org.ovirt.thread.pool-8-thread-31) [401a804e] Lock freed to object 'EngineLock:{exclusiveLocks='[49bfc23b-79a6-4f6c-840c-95ea04562513=<VM, ACTION_TYPE_FAILED_VM_IS_BEING_MIGRATED$VmName serv-vpn-adms1>]', sharedLocks='null'}'
Nicolas Ecarnot 2016-02-01 09:00:24 EST (In reply to Nicolas Ecarnot from comment #30) > > > Do you advice me to try to create a new VM and tests again? > > > > That would help, yes. > > I'm on my way doing that. Been there, tried that : When migrating a brand new pristine VM, the error is exactly the same : libvirtError: unsupported configuration: NUMA node 1 is unavailable (that one can understand as my previous sendings are showing there is NO NUMA node 1 in the target host). Now, the questions are : - should we solve the issue on the target node about NUMA? - should we fix oVirt to prevent its crash in such a case? - both?
libvirtError: unsupported configuration: NUMA node 1 is unavailable This is coming from a layer below oVirt- libvirt. What we need to see is is we're using the right argument (ie- should we use '0' instead of '1').
Can you please provide your libvirt and vdsm log files for the relevant time?
(In reply to Doron Fediuck from comment #9) > Can you please provide your libvirt and vdsm log files for the relevant time? Hi Doron, About the vdsm logs, I already provided everything above. About the libvirt logs, here is the relevant one of the relevant host : 2016-02-03 09:57:08.616+0000: starting up libvirt version: 1.2.17, package: 13.el7_2.2 (CentOS BuildSystem <http://bugs.centos.org>, 2015-12- 09-13:08:22, worker1.bsys.centos.org), qemu version: 2.3.0 (qemu-kvm-ev-2.3.0-31.el7_2.4.1) LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name pleaseMigrateMe -S -machine p c-i440fx-rhel7.2.0,accel=kvm,usb=off -cpu Nehalem -m size=2097152k,slots=16,maxmem=4294967296k -realtime mlock=off -smp 2,maxcpus=16,sockets= 16,cores=1,threads=1 -numa node,nodeid=0,cpus=0-1,mem=2048 -uuid 488c616b-0c2c-4e14-91b0-e8c3debb6209 -smbios type=1,manufacturer=oVirt,produ ct=oVirt Node,version=7-2.1511.el7.centos.2.10,serial=4C4C4544-0039-3410-8033-C7C04F383532,uuid=488c616b-0c2c-4e14-91b0-e8c3debb6209 -no-user -config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-pleaseMigrateMe/monitor.sock,server,nowait -mon chardev= charmonitor,id=monitor,mode=control -rtc base=2016-02-03T10:57:08,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdow n -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,add r=0x4 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/00000002-0002-0002-0002-0000000003ca/f7bf2585-1f74-4655-9118-a790e9abc70d/images/318f04b2-d2f2-4880-9d47-80915e7f2c1f /5771a9a9-6a1e-482d-8cfc-94a26691d14e,if=none,id=drive-virtio-disk0,format=qcow2,serial=318f04b2-d2f2-4880-9d47-80915e7f2c1f,cache=none,werro r=stop,rerror=stop,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=30,id=hostnet0,vhost=on,vhostfd=37 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:cc:00:06,bus=pci.0,addr=0x3 -chardev s ocket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/488c616b-0c2c-4e14-91b0-e8c3debb6209.com.redhat.rhevm.vdsm,server,nowait -device vi rtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/ lib/libvirt/qemu/channels/488c616b-0c2c-4e14-91b0-e8c3debb6209.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0 .0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0 -vnc 192.168.12.17:5,password -device cirru s-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on 2016-02-03T09:57:08.648058Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: 2 3 4 5 6 7 8 9 10 11 12 13 14 15 2016-02-03T09:57:08.648201Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config
(In reply to Doron Fediuck from comment #8) > libvirtError: unsupported configuration: NUMA node 1 is unavailable > > This is coming from a layer below oVirt- libvirt. > What we need to see is is we're using the right argument (ie- should we use > '0' > instead of '1'). Is this something a mere user can change and try, or is it hard coded? (I this is something that can be changed in postgresql, I can do that)
Well, the domain XML says <numatune> <memory mode='interleave' nodeset='0-1'/> </numatune> if you want to migrate it to a host with only one NUMA node, the XML needs to be changed using either a migration hook on the destination host or via a changed XML passed to the libvirt's migration API.
(In reply to Jiri Denemark from comment #12) > Well, the domain XML says > > <numatune> > <memory mode='interleave' nodeset='0-1'/> > </numatune> > > if you want to migrate it to a host with only one NUMA node, the XML needs > to be changed using either a migration hook on the destination host or via a > changed XML passed to the libvirt's migration API. Is this the answer Redhat will provide to thousands of RHEV customers who are using mixed-CPU hosts and that have merely a clue of what a hook is? May you comment what is written on comment #8 please?
That was the answer to oVirt developers (comment #8) how to deal with that. In other words, it's VDSM (rather than the end users) which should take care of changing the XML during migration.
(In reply to Jiri Denemark from comment #14) > That was the answer to oVirt developers (comment #8) how to deal with that. > In other words, it's VDSM (rather than the end users) which should take care > of changing the XML during migration. I thought *this* BZ was the best way to make it to the oVirt developers. If it's not, may you provide the best way? This is a blocker.
Nicalos, I believe this is an important issue, but we do use the bugzilla for intra-team communication too. Jiri is a libvirt maintainer and he merely provided the technical solution needed to solve this from the libvirt side. We will provide the vdsm hook if we decide that it is the right fix. Nobody here expects customers to write their own. But first we have to find out where the issue really is, because the bug might be somewhere else... The migration should not have been allowed (1) in the first place if the engine was aware (2) of the NUMA pinning. We have to check which component set the nodeset list or if it got some implicit value (3). All numbered places might be the primary cause for this issue. Please be patient, it takes more than three days to diagnose, code and release the fix for a multi-package (vdsm, engine, libvirt) + NUMA related issue. I also have one question of my own: Did you configure NUMA pinning for the VM or selected a numa topology manually (preferred, interleaved, strict..)?
(In reply to Martin Sivák from comment #16) > Nicalos, I believe this is an important issue, but we do use the bugzilla > for intra-team communication too. Jiri is a libvirt maintainer and he merely > provided the technical solution needed to solve this from the libvirt side. Perfect. > We will provide the vdsm hook if we decide that it is the right fix. Nobody > here expects customers to write their own. Glad to read that. > But first we have to find out where the issue really is, because the bug > might be somewhere else... The migration should not have been allowed (1) in > the first place if the engine was aware (2) of the NUMA pinning. We have to > check which component set the nodeset list or if it got some implicit value > (3). All numbered places might be the primary cause for this issue. Before hitting this bug, I wasn't even aware of what NUMA meant, so I'm pretty sure I did not force any NUMA pinning in this DC. Now understand this all a little more, I still don't understand why "0" is not set by default in the XML parameters when migrating, as NUMA node 0 will always be available (!) > Please be patient, it takes more than three days to diagnose, code and > release the fix for a multi-package (vdsm, engine, libvirt) + NUMA related > issue. I'll be patient, no issue, I know it can take time. > I also have one question of my own: Did you configure NUMA pinning for the > VM or selected a numa topology manually (preferred, interleaved, strict..)? No pinning. To be sure this bug makes it to the right people, do I have to push it to the libvirt devs, or will you take this responsibility (I'll be happy to help).
> Now understand this all a little more, I still don't understand why "0" is > not set by default in the XML parameters when migrating, as NUMA node 0 will > always be available (!) First, 0 does not have to be available - the CPU can be offline (we also support CPU hotplug). But the issue here is that the VM was started on a host with two NUMA nodes (0 and 1) and is migrated to a host with just a single node (0). And libvirt complains that it does not know what to do with the node no. 1 (implicitly) configured in the VM. This is the whole reason we have the issue.. we currently do not know who or how set the required numa cpus for the VM this way. It may very well be implicit to allow the VM to be scheduled to any available CPU. Sadly it causes trouble with migration. > To be sure this bug makes it to the right people, do I have to push it to > the libvirt devs, or will you take this responsibility (I'll be happy to > help). It already is assigned to the right people.
(In reply to Martin Sivák from comment #18) > But the issue here is that the VM was started on a host with two NUMA nodes > (0 and 1) and is migrated to a host with just a single node (0). I just tested the opposite : - run (once) it on the ONE-numa-node host - migrate to the TWO-numa-node host - it is failing as well At present, I'm trying to closely parse the log to see whether it is the same issue.
Michal I think that one numa node is the node we create for hot plug memory. Martin this means we need a numa policy unit to prevent those situations.
I believe we should not create artificial barriers for migration. If we need to configure NUMA for hotplug purposes we should also make sure it won´t hinder standard migration. Users won't understand why a VM can't be migrated when they do not care about NUMA at all.
You're talking about two different things here... Memory hotplug requires a *guest* NUMA node to be configured (in /domain/cpu/numa element in domain XML), which is irrelevant in this case and should not have any effect on ability to migrate the domain to another host. The problem in this BZ is that the domain's memory is explicitly pinned to a specified set of *host* NUMA nodes (in /domain/numatune/memory), which means the domain can only be started on a host which has the specified NUMA nodes. Thus migrating it to another host may fail depending on the host's NUMA topology.
(In reply to Nicolas Ecarnot from comment #17) > (In reply to Martin Sivák from comment #16) > > Nicalos, I believe this is an important issue, but we do use the bugzilla > > for intra-team communication too. Jiri is a libvirt maintainer and he merely > > provided the technical solution needed to solve this from the libvirt side. > > Perfect. > > > We will provide the vdsm hook if we decide that it is the right fix. Nobody > > here expects customers to write their own. > > Glad to read that. > > > But first we have to find out where the issue really is, because the bug > > might be somewhere else... The migration should not have been allowed (1) in > > the first place if the engine was aware (2) of the NUMA pinning. We have to > > check which component set the nodeset list or if it got some implicit value > > (3). All numbered places might be the primary cause for this issue. > > Before hitting this bug, I wasn't even aware of what NUMA meant, so I'm > pretty sure I did not force any NUMA pinning in this DC. > Now understand this all a little more, I still don't understand why "0" is > not set by default in the XML parameters when migrating, as NUMA node 0 will > always be available (!) > > > Please be patient, it takes more than three days to diagnose, code and > > release the fix for a multi-package (vdsm, engine, libvirt) + NUMA related > > issue. > > I'll be patient, no issue, I know it can take time. > > > I also have one question of my own: Did you configure NUMA pinning for the > > VM or selected a numa topology manually (preferred, interleaved, strict..)? > > No pinning. > Could reproduce your issue. I created a VM and started it on a host with two NUMA cores. I did no pinning at all. After that I did a dump of the xml description and I saw a numa pinning there. Some debugging revealed that ovirt-engine is creating the numa mapping when 1) at least one guest numa node exists 2) a numaTune mode is set (no matter if INTERLEAVE, ...) The virtual NUMA node which we are creating in combination with the default value of INTERLEAVE for numa Tune mode is triggering this behavior. See method NumaSettingFactory#buildVmNumatuneSetting for the actual code. My first impression is that the numaTune mode default value should be null in the engine if no pinning is requested. Just removing the whole <numatune> section in the ovirt-engine broker code via debugger gave me a perfectly fine starting and migratable VM. > > > To be sure this bug makes it to the right people, do I have to push it to > the libvirt devs, or will you take this responsibility (I'll be happy to > help).
Unless all hosts in a cluster have exactly the same numa topology this bug will break every migration to hosts with different numa topology.
Raising priority: Libvirt's default policy is 'strict' so I believe this is working we have single index in the nodeset. @Roman if you can share you log with xml snippet of working VM. What would be the safest configuration that will be agnostic to underlying nodeset so migration is guaranteed (at least from topology perspective)?
(In reply to Roy Golan from comment #25) > Raising priority: > > Libvirt's default policy is 'strict' so I believe this is working we have > single index in the nodeset. @Roman if you can share you log with xml > snippet of working VM. > > What would be the safest configuration that will be agnostic to underlying > nodeset so migration is guaranteed (at least from topology perspective)? Please notice the other way around from comment #9 (https://bugzilla.redhat.com/show_bug.cgi?id=1303842#c19). Also failing from one to many NUMA nodes. May Roman test this scenario, please?
Roy, just don't add any <numatune/> element in the XML. In other words, remove <numatune> <memory mode='interleave' nodeset='0-1'/> </numatune> from the XML. The default is not to pin domains to any NUMA node, which is equivalent to mode='interleave' and nodeset covering all host's NUMA nodes.
(In reply to Jiri Denemark from comment #27) > Roy, just don't add any <numatune/> element in the XML. In other words, > remove > > <numatune> > <memory mode='interleave' nodeset='0-1'/> > </numatune> > > from the XML. Great! That is also what I tested and verified. > The default is not to pin domains to any NUMA node, which is equivalent to > mode='interleave' and nodeset covering all host's NUMA nodes.
my version change was a sad mistake. this is obviously a blocker
Verified on rhevm-3.6.3.2-0.1.el6.noarch 1) Add two hosts(one with one NUMA node and second with two NUMA nodes) 2) Create new vm and starts it on the host with two NUMA nodes 3) Migrate vm on the host with one NUMA node and back Now I also do not see element <numatune> under dumpxml.