Bug 1091818 - Windows guest booting failed with apicv and hv_vapic
Summary: Windows guest booting failed with apicv and hv_vapic
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: kernel
Version: 7.0
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Paolo Bonzini
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Keywords: TestOnly
: 1129225 (view as bug list)
Depends On: 1116936
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-04-28 06:41 UTC by huiqingding
Modified: 2015-03-05 12:03 UTC (History)
12 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2015-03-05 12:03:24 UTC


Attachments (Terms of Use)
guest screenshot when booting failed (3.01 KB, image/png)
2014-04-28 06:42 UTC, huiqingding
no flags Details
tested patch (3.82 KB, patch)
2014-05-20 11:20 UTC, Paolo Bonzini
no flags Details | Diff


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2015:0290 normal SHIPPED_LIVE Important: kernel security, bug fix, and enhancement update 2015-03-05 16:13:58 UTC

Description huiqingding 2014-04-28 06:41:40 UTC
Description of problem:
Boot Win2012r2/win7-64bit guest with "-cpu Haswell/SandyBridge,hv_vapic" on IvyBridge host, the guests cannot boot successfully.

Version-Release number of selected component (if applicable):
kernel-3.10.0-121.el7.x86_64
qemu-kvm-rhev-1.5.3-60.el7ev.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Boot a win2012r2/win7-64 guest with "+x2apic,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_time"
# /usr/libexec/qemu-kvm -cpu Haswell,+x2apic,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_time -enable-kvm -m 4096 -smp 4,sockets=2,cores=2,threads=1,maxcpus=160 -name rhel7 -rtc base=localtime,clock=host,driftfix=slew  -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 -chardev socket,id=channel1,path=/tmp/test1,server,nowait -device virtserialport,chardev=channel1,name=com.redhat.com,bus=virtio-serial0.0,id=port1  -drive file=/home/win7-64.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,serial=40c061dd-5d60-4fc5-865f-55db700407f0,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=virtio-netpci0,mac=54:52:1b:35:2c:01,bus=pci.0,addr=0x5 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -serial unix:/tmp/ttyS0,server,nowait -spice port=5900,disable-ticketing -vga qxl -global qxl-vga.vram_size=67108864 -monitor stdio -qmp tcp:0:4445,server,nowait -vnc :1 
2.
3.

Actual results:
after step1, guest cannot boot and the scrrenshot, please refer to the attchment file.

Expected results:
guest can boot successfully.

Additional info:
1. on SandyBridge host, use the same command line , win2012r2 and win7-64 guest can boot successfully.
2. on amd Opteron_G2 host, use the same command line , win2012r2 and win7-64 guest can boot successfully.
2. on IvyBridge host, only with "-cpu Haswell/SandyBridge,+x2apic,hv_vapic", the guests cannot boot.

Comment 1 huiqingding 2014-04-28 06:42:56 UTC
Created attachment 890345 [details]
guest screenshot when booting failed

Comment 2 huiqingding 2014-04-28 06:52:30 UTC
The cpu info of IvyBridge as following:
# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                120
On-line CPU(s) list:   0-119
Thread(s) per core:    2
Core(s) per socket:    15
Socket(s):             4
NUMA node(s):          4
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 62
Model name:            Intel(R) Xeon(R) CPU E7-4890 v2 @ 2.80GHz
Stepping:              7
CPU MHz:               3232.468
BogoMIPS:              5595.32
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              38400K
NUMA node0 CPU(s):     0-14,60-74
NUMA node1 CPU(s):     15-29,75-89
NUMA node2 CPU(s):     30-44,90-104
NUMA node3 CPU(s):     45-59,105-119

The info of /proc/cpuinfo of IvyBridge as following:
# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 62
model name      : Intel(R) Xeon(R) CPU E7-4890 v2 @ 2.80GHz
stepping        : 7
microcode       : 0x700
cpu MHz         : 3199.984
cache size      : 38400 KB
physical id     : 0
siblings        : 30
core id         : 0
cpu cores       : 15
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips        : 5586.19
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual

Comment 4 Vadim Rozenfeld 2014-04-29 11:32:12 UTC
> 2. on IvyBridge host, only with "-cpu Haswell/SandyBridge,+x2apic,hv_vapic",
> the guests cannot boot.

Can we try without specifying +x2apic flag for Haswell/SandyBridge, they both
should have it be default.

Thanks,
Vadim.

Comment 5 huiqingding 2014-04-29 12:08:50 UTC
(In reply to Vadim Rozenfeld from comment #4)
> > 2. on IvyBridge host, only with "-cpu Haswell/SandyBridge,+x2apic,hv_vapic",
> > the guests cannot boot.
> 
> Can we try without specifying +x2apic flag for Haswell/SandyBridge, they both
> should have it be default.
> 
> Thanks,
> Vadim.

Hi, Vadim,

On ivybridge host, I use "-cpu Haswell,hv_vapic" to boot a win2012r2 guest. It is also failed and the sceenshot is also as comment 1. The ivybridge host supports apicv:
# cat /sys/module/kvm_intel/parameters/enable_apicv
Y

Disable apicv on this host as following commands, use "-cpu Haswell,hv_vapic" to boot a win2012r2 guest, the guest can boot successfully.

# modprobe -r kvm_intel
# modprobe kvm_intel enable_apicv=N
# cat /sys/module/kvm_intel/parameters/enable_apicv
N


Thanks.
Huiqing

Comment 6 Paolo Bonzini 2014-05-09 13:53:12 UTC
I think I understand what's going on here.

When Hyper-V enlightenments are in effect, Windows prefers to issue an Hyper-V 
MSR write to issue an EOI rather than an x2apic MSR write.  The Hyper-V MSR 
write is not handled by the processor, and on EOI the processor will modify the 
highest in-service interrupt (SVI) field of the VMCS, as explained in section 
29.1.4 of the SDM.  We need to do the same on Hyper-V MSR writes.

Can I have a test environment where I can test a fix?

Comment 7 juzhang 2014-05-12 02:04:51 UTC
> Can I have a test environment where I can test a fix?

Hi Huding,

Can we provide the host which can be stably reproduced this issue?

Best Regards,
Junyi

Comment 9 Paolo Bonzini 2014-05-16 15:50:01 UTC
huiqingding, where can I find the Windows 2012 ISO?  Could you copy it for me to /root?  Thanks!

Comment 14 Paolo Bonzini 2014-05-20 11:20:53 UTC
Created attachment 897538 [details]
tested patch

Brew build id 7474234

Comment 19 FuXiangChun 2014-08-12 13:48:00 UTC
Reproduced this bug with <kernel 3.10.0-143.el7.x86_64.

result:
win2k8r2 guest black screen.

verified this bug with 3.10.0-143.el7.x86_64.

guest works well.

Comment 20 FuXiangChun 2014-08-12 13:51:19 UTC
*** Bug 1129225 has been marked as a duplicate of this bug. ***

Comment 21 huiqingding 2014-08-27 09:41:45 UTC
Test this bug on an Ivybridge host using the following version:
kernel-3.10.0-146.el7.x86_64
qemu-kvm-rhev-2.1.0-2.el7.x86_64

The host supports apicv:
# cat /sys/module/kvm_intel/parameters/enable_apicv
Y

Steps to test:
1. boot a win2k8r2 guest with "+x2apic,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_time"
# /usr/libexec/qemu-kvm -M pc -cpu Haswell,+x2apic,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_time -enable-kvm -m 4096 -smp 4,sockets=2,cores=2,threads=1,maxcpus=160 -name rhel7 -rtc base=localtime,clock=host,driftfix=slew  -device virtio-serial-pci,id=virtio-serial0 -chardev socket,id=channel1,path=/tmp/test1,server,nowait -device virtserialport,chardev=channel1,name=com.redhat.com,bus=virtio-serial0.0,id=port1  -drive file=/home/win2k8r2.raw,if=none,id=drive-virtio-disk0,format=raw,serial=40c061dd-5d60-4fc5-865f-55db700407f0,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,drive=drive-virtio-disk0,id=virtio-disk -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -serial unix:/tmp/ttyS0,server,nowait -spice port=5900,disable-ticketing -vga qxl -global qxl-vga.vram_size=67108864 -monitor stdio -qmp tcp:0:4445,server,nowait -nodefaults -boot menu=on -cdrom 7601.17514.101119-1850_x64fre_server_eval_en-us-GRMSXEVAL_EN_DVD.iso -drive file=/home/driver.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=1,drive=drive-ide0-1-0,id=ide0-1-0,logical_block_size=512,physical_block_size=512,min_io_size=32,opt_io_size=64,discard_granularity=512,unit=1,ver=fuxc-ver-cdrom,bus=ide.0,unit=1

Actual results:
after step1, the guest can boot normally.

Comment 23 Miroslav Rezanina 2014-09-02 02:57:00 UTC
Moving to correct component

Comment 26 huiqingding 2014-11-05 06:03:21 UTC
Reproduce this bug using the following version:
kernel-3.10.0-142.el7.x86_64
qemu-kvm-1.5.3-77.el7.x86_64

The host supports apicv:
# cat /sys/module/kvm_intel/parameters/enable_apicv
Y

Steps to Reproduce:
1. boot a win2012r2 guest with "+x2apic,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_time"
# /usr/libexec/qemu-kvm -M pc -cpu Haswell,+x2apic,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_time -enable-kvm -m 4096 -smp 4,sockets=2,cores=2,threads=1,maxcpus=160 -name rhel7 -rtc base=localtime,clock=host,driftfix=slew  -device virtio-serial-pci,id=virtio-serial0 -chardev socket,id=channel1,path=/tmp/test1,server,nowait -device virtserialport,chardev=channel1,name=com.redhat.com,bus=virtio-serial0.0,id=port1  -drive file=/home/win2012r2-blk.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,serial=40c061dd-5d60-4fc5-865f-55db700407f0,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,drive=drive-virtio-disk0,id=virtio-disk -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -serial unix:/tmp/ttyS0,server,nowait -spice port=5900,disable-ticketing -vga qxl -global qxl-vga.vram_size=67108864 -monitor stdio -qmp tcp:0:4445,server,nowait -nodefaults -boot menu=on -cdrom 9600.17050.WINBLUE_REFRESH.140317-1640_X64FRE_SERVER_EVAL_EN-US-IR3_SSS_X64FREE_EN-US_DV9.ISO -drive file=/home/driver.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=1,drive=drive-ide0-1-0,id=ide0-1-0,logical_block_size=512,physical_block_size=512,min_io_size=32,opt_io_size=64,discard_granularity=512,unit=1,ver=fuxc-ver-cdrom,bus=ide.0,unit=1

Actual results:
after step1, the guest cannot boot sucessfully.

Comment 27 huiqingding 2014-11-05 06:33:08 UTC
Test this issue using the folliwng version:
kernel-3.10.0-196.el7.x86_64
qemu-kvm-1.5.3-77.el7.x86_64

Use the same command line of comment 26 to boot a win2012r2 guest and the result is ok: the guest can boot normally.

Comment 28 huiqingding 2014-11-05 06:35:24 UTC
Test this issue using the folliwng version:
kernel-3.10.0-196.el7.x86_64
qemu-kvm-rhev-2.1.2-6.el7.x86_64

Use the same command line of comment 26 to boot a win2012r2 guest and the result is ok: the guest can boot normally.

Comment 29 juzhang 2014-11-11 04:07:08 UTC
According to comment26-comment28, set this issue as verified.

Comment 32 errata-xmlrpc 2015-03-05 12:03:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0290.html


Note You need to log in before you can comment on or make changes to this bug.