Bug 654542

Summary: rhel56 guest hangs during boot up on intel-w3520
Product: Red Hat Enterprise Linux 5 Reporter: Amos Kong <akong>
Component: kernelAssignee: Marcelo Tosatti <mtosatti>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 5.6CC: ailan, syeghiay, tburke
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-12-09 15:31:49 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 580949, 640580    

Description Amos Kong 2010-11-18 07:09:52 UTC
Description of problem:
I installed RHEL56-20101110.0 to intel-w3520 host,
when I boot up a rhel56 guest, it hangs(attached a screendump), then host it's too slow.
But rhel48, rhel55 guest can boot up successfully.
This bug can only be reproduced on intel-w3520 machine.

Version-Release number of selected component (if applicable):
tree: RHEL5.6-Server-20101110.0
kernel: 2.6.18-231.el5
kmod-kvm-83-207.el5
etherboot-zroms-kvm-5.4.4-13.el5
etherboot-roms-kvm-5.4.4-13.el5
kvm-debuginfo-83-207.el5
kvm-83-207.el5
kvm-qemu-img-83-207.el5
kvm-tools-83-207.el5
kmod-kvm-debug-83-207.el5

How reproducible:
100% on intel-w3520

Steps to Reproduce:
1. install RHEL56-20101110.0 to a intel-w3520 machine
2. boot up a rhel56 guest on it
  
Actual results:
rhel56 guest hangs during boot up

Expected results:
rhel56 guest can boot up succesfully

Additional info:
# ps aux |grep qemu
root     16802  0.7  0.0 685288  3200 ?        D    22:48   0:01 /usr/bin/qemu-kvm -name vm1 -monitor unix:/tmp/monitor-humanmonitor1-20101115-153315-vgVU,server,nowait -serial unix:/tmp/serial-20101115-153315-vgVU,server,nowait -drive file=/home/kvm_autotest_root/images/RHEL-Server-5.6-64-virtio.qcow2,index=0,if=ide,cache=none -net nic,vlan=0,model=virtio,macaddr=9a:50:1e:5a:aa:e8 -net tap,vlan=0,ifname=t0-153315-vgVU,script=/etc/qemu-ifup-switch,downscript=no -net nic,vlan=1,model=virtio,macaddr=9a:50:1e:5a:b8:7d -net tap,vlan=1,ifname=t1-153315-vgVU,script=/etc/qemu-ifup-switch,downscript=no -net nic,vlan=2,model=virtio,macaddr=9a:50:1e:5a:e8:ff -net tap,vlan=2,ifname=t2-153315-vgVU,script=/etc/qemu-ifup-switch,downscript=no -net nic,vlan=3,model=virtio,macaddr=9a:50:1e:5a:15:ae -net tap,vlan=3,ifname=t3-153315-vgVU,script=/etc/qemu-ifup-switch,downscript=no -m 512 -smp 2 -redir tcp:5000::22 -vnc :0

=> gdb
(gdb) info threads
  5 Thread 0x41694940 (LWP 17044)  0x000000347b831744 in do_sigwaitinfo () from /lib64/libc.so.6
  4 Thread 0x42448940 (LWP 17118)  0x000000347b8cc9f7 in ioctl () from /lib64/libc.so.6
  3 Thread 0x42e49940 (LWP 17119)  0x000000347b8cc9f7 in ioctl () from /lib64/libc.so.6
  2 Thread 0x4384a940 (LWP 17120)  0x000000347b831744 in do_sigwaitinfo () from /lib64/libc.so.6
* 1 Thread 0x2b7463365150 (LWP 17016)  0x000000347b8cd372 in select () from /lib64/libc.so.6
(gdb) bt
#0  0x000000347b8cd372 in select () from /lib64/libc.so.6
#1  0x0000000000409501 in qemu_select (timeout=0) at /usr/src/debug/kvm-83-maint-snapshot-20090205/qemu/vl.c:3938
#2  main_loop_wait (timeout=0) at /usr/src/debug/kvm-83-maint-snapshot-20090205/qemu/vl.c:4037
#3  0x000000000050023a in kvm_main_loop () at /usr/src/debug/kvm-83-maint-snapshot-20090205/qemu/qemu-kvm.c:596
#4  0x000000000040e6b5 in main_loop (argc=33, argv=0x7fff30441278, envp=<value optimized out>) at /usr/src/debug/kvm-83-maint-snapshot-20090205/qemu/vl.c:4100
#5  main (argc=33, argv=0x7fff30441278, envp=<value optimized out>) at /usr/src/debug/kvm-83-maint-snapshot-20090205/qemu/vl.c:6536
(gdb) thread 2
[Switching to thread 2 (Thread 0x4384a940 (LWP 17120))]#0  0x000000347b831744 in do_sigwaitinfo () from /lib64/libc.so.6
(gdb) bt
#0  0x000000347b831744 in do_sigwaitinfo () from /lib64/libc.so.6
#1  0x000000347b8317fd in sigwaitinfo () from /lib64/libc.so.6
#2  0x000000000041a8a1 in sigwait_compat (opaque=<value optimized out>) at compatfd.c:38
#3  0x000000347c40673d in start_thread () from /lib64/libpthread.so.0
#4  0x000000347b8d40cd in clone () from /lib64/libc.so.6
(gdb) thread 3
[Switching to thread 3 (Thread 0x42e49940 (LWP 17119))]#0  0x000000347b8cc9f7 in ioctl () from /lib64/libc.so.6
(gdb) bt
#0  0x000000347b8cc9f7 in ioctl () from /lib64/libc.so.6
#1  0x000000000052b9ca in kvm_run (kvm=0xe7c6c90, vcpu=1, env=0xe881010) at libkvm.c:911
#2  0x0000000000500399 in kvm_cpu_exec (env=0x0) at /usr/src/debug/kvm-83-maint-snapshot-20090205/qemu/qemu-kvm.c:206
#3  0x0000000000500623 in kvm_main_loop_cpu (_env=<value optimized out>) at /usr/src/debug/kvm-83-maint-snapshot-20090205/qemu/qemu-kvm.c:402
#4  ap_main_loop (_env=<value optimized out>) at /usr/src/debug/kvm-83-maint-snapshot-20090205/qemu/qemu-kvm.c:443
#5  0x000000347c40673d in start_thread () from /lib64/libpthread.so.0
#6  0x000000347b8d40cd in clone () from /lib64/libc.so.6
(gdb) thread 4
[Switching to thread 4 (Thread 0x42448940 (LWP 17118))]#0  0x000000347b8cc9f7 in ioctl () from /lib64/libc.so.6
(gdb) bt
#0  0x000000347b8cc9f7 in ioctl () from /lib64/libc.so.6
#1  0x000000000052b9ca in kvm_run (kvm=0xe7c6c90, vcpu=0, env=0xe8676f0) at libkvm.c:911
#2  0x0000000000500399 in kvm_cpu_exec (env=0x0) at /usr/src/debug/kvm-83-maint-snapshot-20090205/qemu/qemu-kvm.c:206
#3  0x0000000000500623 in kvm_main_loop_cpu (_env=<value optimized out>) at /usr/src/debug/kvm-83-maint-snapshot-20090205/qemu/qemu-kvm.c:402
#4  ap_main_loop (_env=<value optimized out>) at /usr/src/debug/kvm-83-maint-snapshot-20090205/qemu/qemu-kvm.c:443
#5  0x000000347c40673d in start_thread () from /lib64/libpthread.so.0
#6  0x000000347b8d40cd in clone () from /lib64/libc.so.6
(gdb) thread 5
[Switching to thread 5 (Thread 0x41694940 (LWP 17044))]#0  0x000000347b831744 in do_sigwaitinfo () from /lib64/libc.so.6
(gdb) bt
#0  0x000000347b831744 in do_sigwaitinfo () from /lib64/libc.so.6
#1  0x000000347b8317fd in sigwaitinfo () from /lib64/libc.so.6
#2  0x000000000041a8a1 in sigwait_compat (opaque=<value optimized out>) at compatfd.c:38
#3  0x000000347c40673d in start_thread () from /lib64/libpthread.so.0
#4  0x000000347b8d40cd in clone () from /lib64/libc.so.6
(gdb) thread 1
[Switching to thread 1 (Thread 0x2b7463365150 (LWP 17016))]#0  0x000000347b8cd372 in select () from /lib64/libc.so.6
(gdb) bt
#0  0x000000347b8cd372 in select () from /lib64/libc.so.6
#1  0x0000000000409501 in qemu_select (timeout=0) at /usr/src/debug/kvm-83-maint-snapshot-20090205/qemu/vl.c:3938
#2  main_loop_wait (timeout=0) at /usr/src/debug/kvm-83-maint-snapshot-20090205/qemu/vl.c:4037
#3  0x000000000050023a in kvm_main_loop () at /usr/src/debug/kvm-83-maint-snapshot-20090205/qemu/qemu-kvm.c:596
#4  0x000000000040e6b5 in main_loop (argc=33, argv=0x7fff30441278, envp=<value optimized out>) at /usr/src/debug/kvm-83-maint-snapshot-20090205/qemu/vl.c:4100
#5  main (argc=33, argv=0x7fff30441278, envp=<value optimized out>) at /usr/src/debug/kvm-83-maint-snapshot-20090205/qemu/vl.c:6536

=> kvm_stat
efer_reload                    0         0
exits                   54202953    901627
fpu_reload                101299        54
halt_exits                    56         0
halt_wakeup                    1         0
host_state_reload         106040        54
hypercalls                     0         0
insn_emulation             21299         0
insn_emulation_fail            0         0
invlpg                         0         0
io_exits                  106641        18
irq_exits                  32980       495
irq_injections              4123        18
irq_window                  3770        14
kvm_request_irq                0         0
largepages                     0         0
mmio_exits                   166         0
mmu_cache_miss                68         0
mmu_flooded                    0         0
mmu_pde_zapped                 0         0
mmu_pte_updated                0         0
mmu_pte_write                  0         0
mmu_recycled                   0         0
mmu_shadow_zapped             79         0
mmu_unsync                     0         0
mmu_unsync_global              0         0
nmi_injections                 0         0
nmi_window                     0         0
pf_fixed                     150         0
pf_guest                       0         0
remote_tlb_flush               2         0
request_nmi                    0         0
signal_exits                   3         0
tlb_flush                     43         0

=> top info


top - 22:54:12 up 16:41,  3 users,  load average: 5.10, 4.89, 3.58
Tasks: 134 total,   1 running, 132 sleeping,   1 stopped,   0 zombie
Cpu(s):  0.1%us,  2.6%sy,  0.0%ni, 43.4%id, 52.4%wa,  0.1%hi,  1.5%si,  0.0%st
Mem:   8164612k total,  4658552k used,  3506060k free,     7688k buffers
Swap: 10223608k total,        0k used, 10223608k free,  4431568k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                     
 2903 root      10  -5     0    0    0 S  5.0  0.0   2:42.81 kondemand/2                                                                                                                  
16802 root      18   0  669m 3200 2132 D  2.7  0.0   0:03.23 qemu-kvm                                                                                                                     
   16 root      10  -5     0    0    0 S  1.0  0.0   0:10.83 events/2                                                                                                                     
   96 root      10  -5     0    0    0 S  0.7  0.0   0:27.97 kblockd/2                                                                                                                    
13334 root      15   0 77992 2636  844 S  0.3  0.0   0:02.65 screen                                                                                                                       
    1 root      15   0 10372  636  544 S  0.0  0.0   0:00.63 init                                                                                                                         
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 migration/0                                                                                                                  
    3 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/0                                                                                                                  
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/0                                                                                                                   
    5 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 migration/1                                                                                                                  
    6 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/1                                                                                                                  
    7 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/1                                                                                                                   
    8 root      RT  -5     0    0    0 S  0.0  0.0   0:00.91 migration/2                                                                                                                  
    9 root      34  19     0    0    0 S  0.0  0.0   0:00.01 ksoftirqd/2                                                                                                                  
   10 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/2                                                                                                                   
   11 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 migration/3                                                                                                                  
   12 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/3                                                                                                                  
   13 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/3                                                                                                                   
   14 root      10  -5     0    0    0 S  0.0  0.0   0:00.45 events/0    

=> cpuinfo:
processor       : 3
vendor_id       : GenuineIntel
cpu family      : 6
model           : 26
model name      : Intel(R) Xeon(R) CPU           W3520  @ 2.67GHz
stepping        : 5
cpu MHz         : 1596.000
cache size      : 8192 KB
physical id     : 0
siblings        : 4
core id         : 3
cpu cores       : 4
apicid          : 6
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips        : 5333.25
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management: [8]

=> meminfo 
MemTotal:      8164612 kB
MemFree:       3502732 kB
Buffers:          4992 kB
Cached:        4443960 kB
SwapCached:          0 kB
Active:          57656 kB
Inactive:      4426828 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      8164612 kB
LowFree:       3502732 kB
SwapTotal:    10223608 kB
SwapFree:     10223608 kB
Dirty:          467044 kB
Writeback:           0 kB
AnonPages:       35304 kB
Mapped:          16472 kB
Slab:           144312 kB
PageTables:       4180 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  14305912 kB
Committed_AS:   181760 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    267752 kB
VmallocChunk: 34359470511 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Comment 7 Bill Burns 2010-12-09 15:31:49 UTC
Closing this as it has not been reproduced. Please reopen if it can reproduced.