Created attachment 1541899 [details] engine, vdsm , qemu logs Description of problem:Migration of VM with 'Pass-Through host CPU' results with VM's Pause state and exception on the following attempt to run. Version-Release number of selected component (if applicable): ovirt-engine-4.3.0.5-0.0.master.20190214203537.git92a2f86.el7.noarch How reproducible:100% Steps to Reproduce: 1. Configure VM with 'pass-through host CPU'. 2. Start VM on host1 (the lscpu below, number of cpu flags=85 ). -ok 3. Migrate the VM to host2(the lscpu below, number of cpu flags=130 ). - ok 4. Migrate back to the host1 In the attached logs the are two scenarios - for HP VM (VM name = 'golden_env_mixed_virtio_0') and Desktop VM (golden_env_mixed_virtio_4) . Actual results: Migration ends with the VM is Pause state (moved from 'MigratingTo' --> 'Paused'). Trying to run the VM from this state fails with .General Exception: ("internal error: unable to execute QEMU command 'cont': Resetting the Virtual Machine is required",), code = 100 (Failed with error GeneralException and code 100) Expected results: successful migration. Additional info: compute-ge-6.scl.lab.tlv.redhat.com with host1 (puma42.scl.lab.tlv.redhat.com 10.35.160.165) and host2 (ocelot03.qa.lab.tlv.redhat.com 10.35.30.3) host1 lscpu lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 24 On-line CPU(s) list: 0-23 Thread(s) per core: 2 Core(s) per socket: 6 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 45 Model name: Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz Stepping: 7 CPU MHz: 2645.617 CPU max MHz: 2800.0000 CPU min MHz: 1200.0000 BogoMIPS: 4600.09 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 15360K NUMA node0 CPU(s): 0-5,12-17 NUMA node1 CPU(s): 6-11,18-23 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts spec_ctrl intel_stibp flush_l1d lscpu of host2 Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 32 On-line CPU(s) list: 0-31 Thread(s) per core: 2 Core(s) per socket: 16 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz Stepping: 4 CPU MHz: 2101.000 CPU max MHz: 2101.0000 CPU min MHz: 1000.0000 BogoMIPS: 4200.00 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 1024K L3 cache: 22528K NUMA node0 CPU(s): 0-31 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke spec_ctrl intel_stibp flush_l1d
I'm more surprised this is allowed at all. Sharon, shouldn't pass-through pin to the host?
(In reply to Ryan Barry from comment #1) > I'm more surprised this is allowed at all. Sharon, shouldn't pass-through > pin to the host? Right, for enabling cpu pass-through you must pin the VM to a host and that's how it is also configured here (not specifically mentioned by Polina, but is derived from step 2 for reproducing). Pinning a vm to a host determines on which host(s) the vm will start running and doesn't determine the migration destination hosts. i.e. When the scheduler filters hosts to migrate to, he checks all hosts in cluster and not just the ones set in "assigned to" field. According to what I saw there is an "internal error" in libvirt when migrating the vm back from host2 "Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz" to host1 "Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz", so I think that we need to understand from libvirt what is the exact problem here or at least ask for a more meaningful error in logs. That's way we'll understand if there is a real not supported scenario that should be blocked by the engine's scheduler or it's a QEMU probelm that should be supported. Do you know who in libvirt can answer that?
migration destination is started with -cpu host, so it will get different flags every time you start it, regardless if it's a subset or not. We need to change the check in CpuLevelFilterPolicyUnit to compare the flags to match exactly.
verified on U/S and D/S ovirt-engine-4.3.3.2-0.1.el7.noarch ovirt-engine-4.3.3.3-0.0.master.20190403112931.gitbef023e.el7.noarch Migration of VMs with 'pass-through host CPU' mode is not allowed now for hosts with different sets of flags
Hi Eli , It means that if you choose 'Migrate' for such VM, the only choice for Destination Host in the 'Migrate VM(s)' window will be from the hosts with the same set of CPU flags. Hosts with a different number of flags don't appear in this list and could not be chosen for migration.
This bugzilla is included in oVirt 4.3.3 release, published on April 16th 2019. Since the problem described in this bug report should be resolved in oVirt 4.3.3 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.