Bug 594559

Summary: Host Kernel panic at AMD machine when start a vm
Product: Red Hat Enterprise Linux 6 Reporter: Joy Pu <ypu>
Component: kernelAssignee: Gleb Natapov <gleb>
Status: CLOSED NEXTRELEASE QA Contact: Red Hat Kernel QE team <kernel-qe>
Severity: medium Docs Contact:
Priority: low    
Version: 6.0CC: aarcange, gleb, knoel, tburke
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-05-27 05:46:09 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Joy Pu 2010-05-21 02:46:53 UTC
Description:
Start a vm in RHEL6 at an AMD machine and get a kernel panic in host. According to the comment 7 of Bug #592311, open this one.

Version-Release number of selected component (if applicable):
host kernel: 2.6.32-19.el6.x86_64
guest kernel: 2.6.32-19.el6.x86_64
# rpm -qa |grep qemu
gpxe-roms-qemu-0.9.7-6.3.el6.noarch
qemu-kvm-debuginfo-0.12.1.2-2.42.el6.x86_64
qemu-kvm-0.12.1.2-2.53.el6.x86_64
qemu-kvm-tools-0.12.1.2-2.42.el6.x86_64
qemu-img-0.12.1.2-2.53.el6.x86_64

How reproducible:
Once

Steps to Reproduce:
Boot up a smp RHEL-6.0 guest and connect to host with minicom.

Actual results:
Host kernel panic

Expected results:
Boot up a RHEL-6.0 vm successful

Additional info:
1. The command line:
/root/autotest/client/tests/kvm/qemu -name 'vm1' -monitor tcp:0:6001,server,nowait -drive file=/root/autotest/client/tests/kvm/images/RHEL-Server-6.0-64.raw,if=ide,cache=writethrough,boot=on -net nic,vlan=0,model=e1000,macaddr=00:30:0D:20:54:d2 -net tap,vlan=0,ifname=e1000_0_6001,script=/root/autotest/client/tests/kvm/scripts/qemu-ifup-switch,downscript=no -m 2048 -smp 2 -soundhw ac97 -usbdevice tablet -rtc-td-hack -no-hpet -cpu qemu64,+sse2 -no-kvm-pit-reinjection -redir tcp:5000::22 -vnc :0 -serial unix:/tmp/serial-20100519-173032-HCL6,server,nowait 

2.Host cpuinfo
processor       : 2
vendor_id       : AuthenticAMD
cpu family      : 16
model           : 2
model name      : AMD Phenom(tm) 8750 Triple-Core Processor
stepping        : 3
cpu MHz         : 2400.000
cache size      : 512 KB
physical id     : 0
siblings        : 3
core id         : 2
cpu cores       : 3
apicid          : 2
initial apicid  : 2
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nonstop_tsc extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs
bogomips        : 4809.89
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate

3. Kernel panic information:
HARDWARE ERROR
CPU 0: Machine Check Exception:                4 Bank 0: f600000000010015
TSC 1731363033491 ADDR 88007c027000 
PROCESSOR 2:100f23 TIME 1274261453 SOCKET 0 APIC 0
MC0_STATUS: Uncorrected error, report: yes, MiscV: invalid, CPU context
corrupt: yes
 Data Cache Error: L1 TLB multimatch.
This is not a software problem!
Machine check: Processor context corrupt
Kernel panic - not syncing: Fatal machine check on current CPU
Pid: 1745, comm: qemu Tainted: G   M       2.6.32-19.el6.x86_64 #1
Call Trace:
 <#MC>  [<ffffffff814bfd69>] panic+0x78/0x137
 [<ffffffff81024b92>] mce_panic+0x1e2/0x210
 [<ffffffff81025c13>] do_machine_check+0x7f3/0xa20
 [<ffffffff81103f94>] ? filemap_fault+0x14/0x520
 [<ffffffff814c308c>] machine_check+0x1c/0x30
 [<ffffffff81103f94>] ? filemap_fault+0x14/0x520
 [<ffffffffa025c9b0>] ? mc_interception+0x0/0x20 [kvm_amd]
 [<ffffffffa025c9bb>] ? mc_interception+0xb/0x20 [kvm_amd]
 <<EOE>>  [<ffffffffa025efd7>] handle_exit+0x127/0x370 [kvm_amd]
 [<ffffffffa02e4728>] ? kvm_apic_has_interrupt+0x48/0x90 [kvm]
 [<ffffffffa02d3561>] kvm_arch_vcpu_ioctl_run+0x3d1/0xd50 [kvm]
 [<ffffffffa02bf43a>] kvm_vcpu_ioctl+0x52a/0x7d0 [kvm]
 [<ffffffff81170622>] vfs_ioctl+0x22/0xa0
 [<ffffffff81170b64>] do_vfs_ioctl+0x84/0x580
 [<ffffffff811710e1>] sys_ioctl+0x81/0xa0
 [<ffffffff81013172>] system_call_fastpath+0x16/0x1b
[drm:drm_fb_helper_panic] *ERROR* panic occurred, switching back to text
console
[drm] nouveau 0000:00:05.0: Setting dpms mode 0 on vga encoder (output 0)

Comment 2 Gleb Natapov 2010-05-22 19:19:16 UTC
This looks like AMD erratum 383 to me:
1. This is L1 TLB Multimatch error.
2. MC0_STATUS equals f600000000010015 (this is  B6000000_00010015h + 62bit
set).
The question is why rhel6 guest triggers it?

Comment 3 Gleb Natapov 2010-05-23 08:47:33 UTC
Do you know if transparent huge pages were used on the host and/or guest?

Comment 4 Joy Pu 2010-05-24 01:24:54 UTC
(In reply to comment #3)
> Do you know if transparent huge pages were used on the host and/or guest?    

I think so. Because I also run some test about transparent hugepage in the same machine at that day. So the transparent hugepage is setted to always in the host. So both host and guest were using it at that time.

Comment 5 RHEL Program Management 2010-05-25 14:37:01 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.

Comment 6 Andrea Arcangeli 2010-05-25 15:00:40 UTC
corrupt: yes
 Data Cache Error: L1 TLB multimatch.
This is not a software problem!
------

looks buggy cpu to me...

Could be THP trigger hardware bugs.

However THP code you were using is from older kernel, please try to install the latest -29 codebase on host and guest, first the anon-vma-chain version which is a unmodified -29 rhel6 version:

http://brewweb.devel.redhat.com/brew/taskinfo?taskID=2468180    

and if it's not fixing it, you can also try the -29 rhel6 with the backout-anon-vma version with some additional debug code enabled:

http://brewweb.devel.redhat.com/brew/taskinfo?taskID=2468172

It's mostly important for host side, because guest in theory could be malicious and they can do anything there and still in theory host shouldn't machinecheck...

I wonder if this could be an issue in the CPU, I've never seen this on my amd phenom x4 cpus with ntp enabled.

Thanks!

Comment 7 Andrea Arcangeli 2010-05-25 15:11:55 UTC
The two rpm I posted (or any -29 or more recent rhel6 build), includes a workaround for an amd errata that could trigger exactly when using THP on both host and guest. So there's a chance the CPU bug is worked around by upgrading host kernel to -29 release I posted.

Comment 8 Gleb Natapov 2010-05-25 15:26:32 UTC
Please try to reproduce with -29.

Comment 9 Joy Pu 2010-05-26 10:12:33 UTC
(In reply to comment #8)
> Please try to reproduce with -29.
I just meet this problem once before. It is not easy to reproduce.

And with the patch in host, I try to boot up vm 100 times using commands:
for i in `seq 100`; do boot_up_cmd ; done
for i in `seq 100`; do sleep 120 && kill -9 `pidof qemu` && echo "kill number" $i ;done

And boot up a vm run tsc test which trigger this problem at the first time 100 times and it seems the system still OK. No kernel panic and warning in host.

Comment 10 Gleb Natapov 2010-05-26 10:23:04 UTC
(In reply to comment #9)
> (In reply to comment #8)
> > Please try to reproduce with -29.
> I just meet this problem once before. It is not easy to reproduce.
> 
> And with the patch in host, I try to boot up vm 100 times using commands:
> for i in `seq 100`; do boot_up_cmd ; done
> for i in `seq 100`; do sleep 120 && kill -9 `pidof qemu` && echo "kill number"
> $i ;done
> 
> And boot up a vm run tsc test which trigger this problem at the first time 100
> times and it seems the system still OK. No kernel panic and warning in host.    
Have you set transparent huge page to "always" in the host and guest before testing? If no please retest.
If yes then lets close the bug.

Comment 11 Joy Pu 2010-05-27 01:08:58 UTC
(In reply to comment #10)
> (In reply to comment #9)
> > (In reply to comment #8)
> > > Please try to reproduce with -29.
> > I just meet this problem once before. It is not easy to reproduce.
> > 
> > And with the patch in host, I try to boot up vm 100 times using commands:
> > for i in `seq 100`; do boot_up_cmd ; done
> > for i in `seq 100`; do sleep 120 && kill -9 `pidof qemu` && echo "kill number"
> > $i ;done
> > 
> > And boot up a vm run tsc test which trigger this problem at the first time 100
> > times and it seems the system still OK. No kernel panic and warning in host.    
> Have you set transparent huge page to "always" in the host and guest before
> testing? If no please retest.
> If yes then lets close the bug.    

Yes. It already set to "always" and the scan_sleep_millisecs, alloc_sleep_millisecs is set to "0", defrag is set to "yes".

Comment 13 Gleb Natapov 2010-05-27 05:46:09 UTC
Then I am closing this bug. If the problem reappears it can be reopened.