Bug 2186367

Didn't reproduce this issue with Win2019 seabios guest with Hyper-v role installed inside guest.

Test on Nested Hyper-V on KVM with CPU model 'Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz', Sandy Bridge EP, but not Ice Lake, cannot reproduce this issue about "Win11 guest cannot boot up with vmx=on"

Both L1 and L2 vms start up normally.
 
L0:   
    Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
    RHEL9 with 5.14.0-298.el9.x86_64
    qemu-kvm-common-7.2.0-14.el9_2.x86_64    
L1: Windows 11 uefi firmware with enabled Hyper-V role
L2: RHEL-9 Gen1 and Gen2 


[root@dell-per730-32 scripts]# /usr/libexec/qemu-kvm -name w11-uefi -m 8G -smp 8 -rtc base=localtime,driftfix=none  -boot order=cd,menu=on -monitor stdio -M q35,smm=on,accel=kvm -vga std -vnc :8 -global driver=cfi.pflash01,property=secure,value=on -drive file=/usr/share/OVMF/OVMF_CODE.secboot.fd,if=pflash,format=raw,readonly=on,unit=0 -drive file=/home/test/OVMF_VARSw11.fd,if=pflash,format=raw,unit=1 -netdev tap,script=/etc/qemu-ifup,downscript=no,id=hostnet0,vhost=on,queues=4 -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x3 -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x3.0x1  -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:52:11:36:1f:61,bus=pci.2,mq=on,vectors=10 -blockdev driver=file,cache.direct=off,cache.no-flush=on,filename=/home/images/w11-uefi.qcow2,node-name=system_file -blockdev driver=qcow2,node-name=drive_system_disk,file=system_file -object iothread,id=thread0 -device virtio-blk-pci,iothread=thread0,drive=drive_system_disk,id=system_disk,bootindex=0,bus=pci.1 -usb -device usb-tablet -cpu Haswell-noTSX,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vpindex,hv_runtime,hv_crash,hv_time,hv_synic,hv_stimer,hv_tlbflush,hv_ipi,hv_reset,hv_frequencies,hv_reenlightenment,hv_stimer_direct,hv_evmcs,hv_emsr_bitmap,vmx=on


Thank you so much.
Best Regards,
Xuemin

Tested it on Intel(R) Xeon(R) Silver 4314 with vmx=on, not hit this issue.

 (1/2) Host_RHEL.m9.u3.ovmf.qcow2.virtio_blk.up.virtio_net.Guest.Win11.x86_64.io-github-autotest-qemu.boot.q35: STARTED
 (1/2) Host_RHEL.m9.u3.ovmf.qcow2.virtio_blk.up.virtio_net.Guest.Win11.x86_64.io-github-autotest-qemu.boot.q35: PASS (100.64 s)
 (2/2) Host_RHEL.m9.u3.ovmf.qcow2.virtio_scsi.up.virtio_net.Guest.Win11.x86_64.io-github-autotest-qemu.boot.q35: STARTED
 (2/2) Host_RHEL.m9.u3.ovmf.qcow2.virtio_scsi.up.virtio_net.Guest.Win11.x86_64.io-github-autotest-qemu.boot.q35: PASS (117.33 s)


Versions:
kernel-5.14.0-289.el9.x86_64
qemu-kvm-8.0.0-0.rc1.el9.candidate
edk2-ovmf-20230301gitf80f052277c8-2.el9.noarch
swtpm-0.8.0-1.el9.x86_64


Host cpu model:

# lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         46 bits physical, 57 bits virtual
  Byte Order:            Little Endian
CPU(s):                  64
  On-line CPU(s) list:   0-63
Vendor ID:               GenuineIntel
  BIOS Vendor ID:        Intel
  Model name:            Intel(R) Xeon(R) Silver 4314 CPU @ 2.40GHz
    BIOS Model name:     Intel(R) Xeon(R) Silver 4314 CPU @ 2.40GHz
    CPU family:          6
    Model:               106
    Thread(s) per core:  2
    Core(s) per socket:  16
    Socket(s):           2
    Stepping:            6
    CPU max MHz:         3400.0000
    CPU min MHz:         800.0000
    BogoMIPS:            4800.00
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clf
                         lush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm c
                         onstant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc c
                         puid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 s
                         dbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadl
                         ine_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault e
                         pb cat_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced 
                         tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 
                         smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifm
                         a clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsave
                         c xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lo
                         ck_detect wbnoinvd dtherm ida arat pln pts avx512vbmi umip pku ospke avx512
                         _vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq 
                         la57 rdpid fsrm md_clear pconfig flush_l1d arch_capabilities
Virtualization features: 
  Virtualization:        VT-x
Caches (sum of all):     
  L1d:                   1.5 MiB (32 instances)
  L1i:                   1 MiB (32 instances)
  L2:                    40 MiB (32 instances)
  L3:                    48 MiB (2 instances)
NUMA:                    
  NUMA node(s):          2
  NUMA node0 CPU(s):     0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52
                         ,54,56,58,60,62
  NUMA node1 CPU(s):     1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53
                         ,55,57,59,61,63
Vulnerabilities:         
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Mitigation; Clear CPU buffers; SMT vulnerable
  Retbleed:              Not affected
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW se
                         quence
  Srbds:                 Not affected
  Tsx async abort:       Not affected



Guest cpu command lines:
-cpu 'Icelake-Server',ds=on,ss=on,dtes64=on,vmx=on,pdcm=on,hypervisor=on,tsc-adjust=on,avx512ifma=on,sha-ni=on,rdpid=on,fsrm=on,md-clear=on,stibp=on,arch-capabilities=on,xsaves=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,rdctl-no=on,ibrs-all=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on,tsx-ctrl=on,hle=off,rtm=off,mpx=off,intel-pt=off,hv_stimer,hv_synic,hv_vpindex,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_frequencies,hv_runtime,hv_tlbflush,hv_reenlightenment,hv_stimer_direct,hv_ipi,kvm_pv_unhalt=on \



Hi Nana,

Did you tested it with libvirt?  There was a existing bug against icelake cpu model (Bug 2034454 - Windows guest stucks at the tiancore icon as libvirt doesn't add newer cpu model)


Hi Luyao,

Could you help share the progress of bug 2034454? Any update of it? Thanks.


cc Qianqiang.

(In reply to Xueqiang Wei from comment #3)
...
> 
> 
> Hi Luyao,
> 
> Could you help share the progress of bug 2034454? Any update of it? Thanks.
> 

Hi Xueqiang,

Sorry, There is no update for bug 2034454 and also no update for the dependent bug 1856522.

I want to clarify the test result. With the same environment and qemu command lines mentioned in Comment 3, I retested with Hyper-v role installed inside guest, hit the same issue as Nana on Icelake host.

Tested ovmf guests win10/win2019/win11/win2022 with hyper-v role installed, only win11 guest can reproduce this issue.
And once I uninstall hyper-v role, guest works well again.

cpu command line: -cpu 'Icelake-Server-noTSX',kvm_pv_unhalt=on,vmx=on 


Hi Vitaly,


Would you please help take a look at this bug? 
It is related to specific windows guest (win11) with hyper-v role installed inside guest, is this the nested issue?

Thanks for your time.


Best regards
Nana

(In reply to liunana from comment #6)
> Tested ovmf guests win10/win2019/win11/win2022 with hyper-v role installed,
> only win11 guest can reproduce this issue.
> And once I uninstall hyper-v role, guest works well again.
> 
> cpu command line: -cpu 'Icelake-Server-noTSX',kvm_pv_unhalt=on,vmx=on 
> 
> 
> Hi Vitaly,
> 
> 
> Would you please help take a look at this bug? 
> It is related to specific windows guest (win11) with hyper-v role installed
> inside guest, is this the nested issue?

Icelake-Sever CPU models before 'v6' (and 'Icelake-Server-noTSX' is an alias for 
'Icelake-Server-v2') are known to not work with Win11 with Hyper-V role enabled:
https://bugzilla.redhat.com/show_bug.cgi?id=2038051
is this the same issue or a new one? Could you please start with upgrading the CPU
model you use to 'v6'?

(In reply to Vitaly Kuznetsov from comment #7)
> (In reply to liunana from comment #6)
> > Tested ovmf guests win10/win2019/win11/win2022 with hyper-v role installed,
> > only win11 guest can reproduce this issue.
> > And once I uninstall hyper-v role, guest works well again.
> > 
> > cpu command line: -cpu 'Icelake-Server-noTSX',kvm_pv_unhalt=on,vmx=on 
> > 
> > 
> > Hi Vitaly,
> > 
> > 
> > Would you please help take a look at this bug? 
> > It is related to specific windows guest (win11) with hyper-v role installed
> > inside guest, is this the nested issue?
> 
> Icelake-Sever CPU models before 'v6' (and 'Icelake-Server-noTSX' is an alias
> for 
> 'Icelake-Server-v2') are known to not work with Win11 with Hyper-V role
> enabled:
> Red Hathttps://bugzilla.redhat.com/show_bug.cgi?id=2038051
> is this the same issue or a new one? Could you please start with upgrading
> the CPU
> model you use to 'v6'?

Yes, Win11 guest wokrs well with "-cpu 'Icelake-Server-v6',vmx=on". 
And it also works well with "-cpu 'Icelake-Server-noTSX',vmx=on,vmx-page-walk-5=on".

They should be the same issue, thanks.


Best regards
Nana

(In reply to liunana from comment #8)
> 
> Yes, Win11 guest wokrs well with "-cpu 'Icelake-Server-v6',vmx=on". 
> And it also works well with "-cpu
> 'Icelake-Server-noTSX',vmx=on,vmx-page-walk-5=on".
> 
> They should be the same issue, thanks.
> 

Thanks for the confirmation!

*** This bug has been marked as a duplicate of bug 2038051 ***