Bug 863877 - non-Red Hat Linux guests fail to boot with -cpu host and more than 2 vcpus on Intel Xeon E5-2620
non-Red Hat Linux guests fail to boot with -cpu host and more than 2 vcpus on...
Status: CLOSED WORKSFORME
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm (Show other bugs)
6.2
x86_64 Linux
low Severity unspecified
: rc
: ---
Assigned To: Eduardo Habkost
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-10-07 18:49 EDT by Justin Lott
Modified: 2014-04-17 15:25 EDT (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-04-16 15:52:27 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Justin Lott 2012-10-07 18:49:37 EDT
Description of problem:
When starting guests that do not use Red Hat kernels in combination with the '-cpu host' option to qemu-kvm and when having > 2 vcpus assigned, the guest will stop booting during hardware initialization.


Version-Release number of selected component (if applicable):
Host:
qemu-kvm-0.12.1.2-2.209.el6_2.1.x86_64
kernel-2.6.32-220.2.1.el6.x86_64

Guest (Debian 6 that wont boot):
linux-image-2.6.32-5-amd64

Guest (CentOS 6.3 that boots fine):
kernel-2.6.32-279.9.1.el6.x86_64


How reproducible:
Always


Steps to Reproduce:
1. Install a non-Red Hat guest with < 3 vcpus and '-cpu host'
2. Change number of vcpus to < 2
3. Guest will freeze during boot up hardware initialization
4. Disable use of '-cpu host' and the guest will boot properly
  

Actual results:
Guest stops booting during hardware initialization


Expected results:
Guest should boot properly


Additional info:
I have seen this problem with Debian 6, Ubuntu 10, Ubuntu 11, and Ubuntu 12. The vast majority of information below was gleaned with Debian 6. There are reports from another Engineer that the same problem exists on AMD Opteron 6272 CPUs as well. I have not tested on that hardware.

Command line to reproduce:
/usr/libexec/qemu-kvm -cpu host -S -M rhel6.0.0 -enable-kvm -m 1024 -smp 24,sockets=24,cores=1,threads=1 -name debian -uuid bcf06a0e-fb57-d82e-32d3-b89e6dd394f7 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/debian.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -drive file=/dev/LVM/debian,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/debian-live-6.0.5-amd64-rescue.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -device virtio-net-pci,vlan=0,id=net0,mac=54:52:00:11:78:9b,bus=pci.0,addr=0x3 -net tap,fd=22,vlan=0,name=hostnet0 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -vnc 127.0.0.1:0 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5

When using -smp 2,sockets=2,cores=1,threads=1 the guest will boot properly. When using more than 2 vcpus, turning on as much kernel verbosity as possible shows the guest is locking up at the same point every time:

[    1.543404] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14
[    1.543404] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14
[    1.543407] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15
[    1.543407] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15
[    1.543604]  vda1 vda2 < vda5 >
[    1.543604]  vda1 vda2 < vda5 >
*** FREEZE HERE WITH -CPU HOST ***
[    1.701761] uhci_hcd: USB Universal Host Controller Interface driver
[    1.701761] uhci_hcd: USB Universal Host Controller Interface driver
[    1.703274] uhci_hcd 0000:00:01.2: PCI->APIC IRQ transform: INT D -> IRQ 11
[    1.703274] uhci_hcd 0000:00:01.2: PCI->APIC IRQ transform: INT D -> IRQ 11
[    1.705167] uhci_hcd 0000:00:01.2: setting latency timer to 64
[    1.705167] uhci_hcd 0000:00:01.2: setting latency timer to 64

Interestingly, using a Red Hat kernel in the Debian guest (2.6.32-279.9.1.el6.x86_64 in this case, with selinux=0) allows the guest to boot properly.

Upgrading the host kernel to a compiled 2.6.32 from source, using /boot/config-2.6.32-220.2.1.el6.x86_64, also allows all guests to boot properly. This holds true for various upstream kernel versions up to and including 3.6.0.

I've also noticed differences in the CPU flags that the host exposes to the guest, depending on the guest kernel. When booting either guest with 2 vcpu's (since thats the only way to get the Debian guest to boot with a Debian kernel) I see this:

Hosts /proc/cpuinfo:
-----------------------------
processor       : 23
vendor_id       : GenuineIntel
cpu family      : 6
model           : 45
model name      : Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
stepping        : 7
cpu MHz         : 2000.014
cache size      : 15360 KB
physical id     : 1
siblings        : 12
core id         : 5
cpu cores       : 6
apicid          : 43
initial apicid  : 43
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 x2apic popcnt aes xsave avx lahf_lm ida arat epb xsaveopt pln pts dts tpr_shadow vnmi flexpriority ept vpid
bogomips        : 3999.57
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

CentOS 6.3 guest w/ 2 vcpus & -cpu host (2.6.32-279.9.1.el6.x86_64):
--------------------------------------------------------------------
fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx lm constant_tsc unfair_spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes xsave avx hypervisor lahf_lm xsaveopt

debian 6.0 guest w/ 2 vcpus & -cpu host (2.6.32-5-amd64):
---------------------------------------------------------
fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx lm constant_tsc rep_good pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes xsave avx hypervisor lahf_lm

--- debian      2012-10-07 21:22:34.180495602 +0000
+++ centos      2012-10-07 21:22:48.960487676 +0000
@@ -27 +27 @@
-rep_good
+unfair_spinlock
@@ -40,0 +41 @@
+xsaveopt

It seems possible that might have something to do with it... in which case this may be a kernel bug and not in qemu-kvm. I did try upgrading the host to the latest 6.3 kernel available (2.6.32-279.9.1) and the problem still exists. I have also tried the latest qemu-kvm from git (latest commit to master as of 10/4/2012) with the same results.
Comment 1 Justin Lott 2012-10-07 18:59:55 EDT
I should add, in case it's not clear, that in a cluster of ~2500 hosts running qemu-kvm, we have only seen this problem on the AMD and Intel CPU's noted above. It does not appear to happen on any other CPU's in the cluster.
Comment 3 Ademar Reis 2012-10-09 09:44:30 EDT
Thank you for taking the time to enter a bug report with us. We appreciate the feedback and look to use reports such as this to guide our efforts at improving our products. That being said, this bug tracking system is not a mechanism for requesting support, and we are not able to  guarantee the timeliness or suitability of a resolution.

If this issue is critical or in any way time sensitive, please raise a ticket through your regular Red Hat support channels to make certain  it receives the proper attention and prioritization to assure a timely resolution. 

For information on how to contact the Red Hat production support team, please visit: https://www.redhat.com/support/process/production/#howto
Comment 4 Justin Lott 2012-10-10 16:14:09 EDT
Using "-cpu host,-xsave" works around this problem on both Intel Xeon E5-2620 and AMD Opteron 6272 processors.

vendor_id       : GenuineIntel
cpu family      : 6
model           : 45
model name      : Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
stepping        : 7
cpu MHz         : 2000.365
cache size      : 15360 KB
physical id     : 1
siblings        : 12
core id         : 5
cpu cores       : 6
apicid          : 43
initial apicid  : 43
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 x2apic popcnt aes xsave avx lahf_lm ida arat epb xsaveopt pln pts dts tpr_shadow vnmi flexpriority ept vpid
bogomips        : 3999.51
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

vendor_id	: AuthenticAMD
cpu family	: 21
model		: 1
model name	: AMD Opteron(TM) Processor 6272                 
stepping	: 2
cpu MHz		: 2100.074
cache size	: 2048 KB
physical id	: 3
siblings	: 16
core id		: 7
cpu cores	: 8
apicid		: 143
initial apicid	: 111
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nonstop_tsc extd_apicid amd_dcm aperfmperf pni pclmulqdq monitor ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 nodeid_msr topoext perfctr_core cpb npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
bogomips	: 4199.43
TLB size	: 1536 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate [9]
Comment 9 Eduardo Habkost 2014-04-16 15:52:27 EDT
Tried reproducing the bug in RHEL-6.4 on exactly the same host CPU model (Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz), using exactly the same Debian Live ISO and command-line (except for the network part), and couldn't reproduce it. It may have been fixed somewhere between 6.2 and 6.4.

If more information is available and the bug can be reproduced in RHEL-6.5, feel free to reopen the bug.
Comment 10 Eduardo Habkost 2014-04-17 15:25:45 EDT
I finally could reproduce it after installing a 6.2 host (kernel-2.6.32-220.el6.x86_64). The bug was gone after I have installed kernel-2.6.32-431.el6.x86_64.

Note You need to log in before you can comment on or make changes to this bug.