Bug 619306

Summary: CentOS 5.5/x86_64 hangs when -smp is enabled. Reproducible on two different machines.
Product: [Fedora] Fedora Reporter: Gilboa Davara <gilboad>
Component: qemuAssignee: Justin M. Forbes <jforbes>
Status: CLOSED WONTFIX QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: low    
Version: 13CC: amit.shah, berrange, clalance, dwmw2, ehabkost, extras-orphan, gcosta, itamar, jaswinder, jforbes, knoel, markmc, notting, ondrejj, quintela, scottt.tw, virt-maint
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-06-29 08:49:50 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:

Description Gilboa Davara 2010-07-29 04:49:19 EDT
I'm running the same SMP CentOS 5.5 / x86_64 guest on a couple of my machines.
All the machines are KVM capable (Xeon L5410, Xeon L5530, AMD Phenom II X3 425 and AMD Phenom II X4 635) 

When I start the guest in normal mode, it more-or-less behaves the same on both the AMD and Intel machines.
When I start the guest in snapshot mode, it usually (~50%) hangs when I start loading it with I/O requests.

Stack (Phenom II X4 635)
$ pstack $(pgrep qemu)
Thread 3 (Thread 0x7f2e88cfa710 (LWP 18185)):
#0  0x00000031eb4d95a7 in ioctl () from /lib64/libc.so.6
#1  0x0000000000429a40 in kvm_run ()
#2  0x0000000000429e99 in kvm_cpu_exec ()
#3  0x000000000042aae1 in ap_main_loop ()
#4  0x00000031ec007761 in start_thread () from /lib64/libpthread.so.0
#5  0x00000031eb4e14ed in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f2e83fff710 (LWP 18186)):
#0  0x00000031eb4d95a7 in ioctl () from /lib64/libc.so.6
#1  0x0000000000429a40 in kvm_run ()
#2  0x0000000000429e99 in kvm_cpu_exec ()
#3  0x000000000042aae1 in ap_main_loop ()
#4  0x00000031ec007761 in start_thread () from /lib64/libpthread.so.0
#5  0x00000031eb4e14ed in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f2e892ff740 (LWP 18055)):
#0  0x00000031eb4da043 in select () from /lib64/libc.so.6
#1  0x000000000040ba60 in main_loop_wait ()
#2  0x0000000000427aca in kvm_main_loop ()
#3  0x000000000040ea76 in main ()

Guest command line:
qemu-kvm -cdrom/dev/sr0
         -serial telnet::9007,server,nowait
         -netnic,vlan=1,macaddr=00:E0:81:B0:EB:07,model=e1000
         -nettap,vlan=1,ifname=tap70
         -netnic,vlan=2,macaddr=00:E0:81:B0:EB:17,model=e1000
         -nettap,vlan=2,ifname=tap71
         -netnic,vlan=3,macaddr=CE:5D:1F:B1:48:27,model=e1000
         -nettap,vlan=3,ifname=tap72
         -m3072
         -name lisa-vmw-demo.com
         -hda/usr/drives/kvm/lisa-vmw-demo.com/lisa-vmw-demo.com.img
         -smp2
         -usb
         -usbdevicetablet
         -vga std
         -vnc:7
         -snapshot-bootc
Comment 1 Gilboa Davara 2010-07-29 04:52:06 EDT
Wrong wording:

When I start the guest in normal mode, it more-or-less behaves the same on both
the AMD and Intel machines.
When I start the guest in snapshot mode on the AMD machine, it usually (~50%) hangs when I start loading it with I/O requests. When I do the same on Intel machines, the guest works as expected. (Beyond the usual performance degradation)
Comment 2 Gilboa Davara 2010-07-29 04:58:31 EDT
I should add the I'm using telnet to connect the machine's serial console. (In the guest: console=tty console=ttyS0,57600) and I'm not getting any type of OOPs (or any other error). The guest simply becomes unresponsive and the host qemu-kvm process stays at 0% CPU.

Here's the gdb output (w/ debuginfo package installed).

$ gdb /usr/bin/qemu-kvm -p $(pgrep qemu-kvm)
...
(gdb) thread apply all bt
Thread 3 (Thread 0x7f2e88cfa710 (LWP 18185)):
#0  0x00000031eb4d95a7 in ioctl () from /lib64/libc.so.6
#1  0x0000000000429a40 in kvm_run (env=0x2629c10) at /usr/src/debug/qemu-kvm-0.12.3/qemu-kvm.c:917
#2  0x0000000000429e99 in kvm_cpu_exec (env=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.3/qemu-kvm.c:1647
#3  0x000000000042aae1 in kvm_main_loop_cpu (_env=0x2629c10)
    at /usr/src/debug/qemu-kvm-0.12.3/qemu-kvm.c:1889
#4  ap_main_loop (_env=0x2629c10) at /usr/src/debug/qemu-kvm-0.12.3/qemu-kvm.c:1939
#5  0x00000031ec007761 in start_thread () from /lib64/libpthread.so.0
#6  0x00000031eb4e14ed in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f2e83fff710 (LWP 18186)):
#0  0x00000031eb4d95a7 in ioctl () from /lib64/libc.so.6
#1  0x0000000000429a40 in kvm_run (env=0x2643010) at /usr/src/debug/qemu-kvm-0.12.3/qemu-kvm.c:917
#2  0x0000000000429e99 in kvm_cpu_exec (env=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.3/qemu-kvm.c:1647
#3  0x000000000042aae1 in kvm_main_loop_cpu (_env=0x2643010)
    at /usr/src/debug/qemu-kvm-0.12.3/qemu-kvm.c:1889
#4  ap_main_loop (_env=0x2643010) at /usr/src/debug/qemu-kvm-0.12.3/qemu-kvm.c:1939
#5  0x00000031ec007761 in start_thread () from /lib64/libpthread.so.0
#6  0x00000031eb4e14ed in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f2e892ff740 (LWP 18055)):
#0  0x00000031eb4da043 in select () from /lib64/libc.so.6
#1  0x000000000040ba60 in main_loop_wait (timeout=1000) at /usr/src/debug/qemu-kvm-0.12.3/vl.c:3994
#2  0x0000000000427aca in kvm_main_loop () at /usr/src/debug/qemu-kvm-0.12.3/qemu-kvm.c:2122
#3  0x000000000040ea76 in main_loop (argc=35, argv=<value optimized out>, envp=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.3/vl.c:4212
#4  main (argc=35, argv=<value optimized out>, envp=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.3/vl.c:6246
(gdb) quit
A debugging session is active.

        Inferior 1 [process 18055] will be detached.
- Gilboa
Comment 3 Gilboa Davara 2010-07-29 05:01:39 EDT
I managed to reproduce it again (took me 30 seconds).
Seems that I can -easily- reproduce it by scp'ing files into the client. (bridged network, NM disabled)

- Gilboa
Comment 4 Gilboa Davara 2010-07-29 07:07:35 EDT
OK. Managed to reproduce it again, this time in non-snapshot.
It seems the the client doesn't really hang, it simply stops responding to I/O (be that serial console or network) requests.
I can connect to the redirected VGA (over VNC) and the client -seems- responsive, but any attempt to start any type of application (su, top, etc) hangs forever.
vmstat does work, and even though the client should be idle, its has fairly large number of pending IO (bo/bi are at 300-600) requests and I/O wait is at ~3-4%.
At the same time, the host's I/O is -completely- idle.

I should point out that a copy of the client (-exact- copy using the same HD image file, minus different IP, MAC and hostname) works flawlessly on two Xeon machines...

- Gilboa
Comment 5 Gilboa Davara 2010-07-29 08:27:54 EDT
Strike that. Managed to reproduce it on an Intel machine. (Took me a couple of hours).
Switching back to an older CentOS kernel has zero effect.
CentOS kernel bug?

- Gilboa
Comment 6 Gilboa Davara 2010-07-29 10:24:25 EDT
Started enabling / disabling parameters in my qemu automation scripts.
Once I disabled SMP, the guests on both machines started working again, no matter how much I/O load (both network and disk) I placed on them.

One more thing: by default, the guests boot with noapic (to prevent clock drift), removing it only seemed to reduce the time required to kill the guest.

I should point out that the change is very recent. I've been using these guest (in the current SMP configuration) for a -very- long time.
Looking at the yum logs on the CentOS guest, I can't say that I see anything major that changed in the last couple of weeks (which tends to point the finger back to the host)

- Gilboa
Comment 7 Justin M. Forbes 2010-08-12 12:00:57 EDT
can you tell me the versions of kernel, qemu, and seabios used on these machines?
Comment 8 Gilboa Davara 2010-08-12 12:25:32 EDT
$ rpm -qa | egrep -e '^seabios|^qemu' | sort
qemu-0.12.3-8.fc13.x86_64
qemu-common-0.12.3-8.fc13.x86_64
qemu-debuginfo-0.12.3-8.fc13.x86_64
qemu-img-0.12.3-8.fc13.x86_64
qemu-kvm-0.12.3-8.fc13.x86_64
qemu-kvm-tools-0.12.3-8.fc13.x86_64
qemu-launcher-1.7.4-7.fc12.noarch
qemu-system-arm-0.12.3-8.fc13.x86_64
qemu-system-cris-0.12.3-8.fc13.x86_64
qemu-system-m68k-0.12.3-8.fc13.x86_64
qemu-system-mips-0.12.3-8.fc13.x86_64
qemu-system-ppc-0.12.3-8.fc13.x86_64
qemu-system-sh4-0.12.3-8.fc13.x86_64
qemu-system-sparc-0.12.3-8.fc13.x86_64
qemu-system-x86-0.12.3-8.fc13.x86_64
qemu-user-0.12.3-8.fc13.x86_64
seabios-bin-0.5.1-3.fc13.noarch

$ uname -r
2.6.33.6-147.fc13.x86_64
Comment 9 Gilboa Davara 2010-08-12 12:30:05 EDT
Should I test w/ 2.6.34 from -testing?

- Gilboa
Comment 10 Gilboa Davara 2010-08-23 06:27:28 EDT
2.6.34 doesn't help.
I managed to reproduce it on -all- my F13 hosts. (Ranging from Athlon X3 to Xeon 5600's).

Looks as if a CentOS upgrade triggered this issue.
... Though, if I report it to bugzilla@centos, they will most likely point a finger at Fedora/kvm.

Thus far, we haven't experience this issue on our RHEL guest on RHEL host production machines, so it looks like a Fedora host problem.

- Gilboa
Comment 11 Justin M. Forbes 2010-09-02 12:42:16 EDT
If you could test with the latest qemu update and let me know if this fixes the issue?  You should be using qemu-0.12.5 packages and seabios-bin-0.6
Comment 12 Gilboa Davara 2010-09-03 08:57:14 EDT
Thing are somewhat better, but things are still problematic.

Let me try and explain.
For the test I'm using a heavy system-installation-and-configuration bash process that spawns ~8 sub-process which are more-or-less I/O bound.

- On a real 8 core Xeon machine w/ HT enabled, the host see ~8 bash processes eating 20-30% CPU (top) and ~8MB/s of I/O (iotop).

- On a guest with a single vCPU (running on the machine above, in snapshot mode), the host sees the qemu process eating 100% CPU time (top) and generating ~1 MB/s of I/O (iotop). iotop also sees ~6-10MB/s of unaccounted background I/O that ends the second the bash process ends on the guest.

- On a guest with 4 vCPUs (again in snapshot mode), the hosts sees the qemu process eating 20% CPU time (top, out of possible 1600%!) and generation ~300KB/s of I/O (itop, lower than single CPU guest). iotop also sees ~8-10MB/s of unaccounted background I/O that -doesn't- end when the bash process ends on the guest. The only way to stop this background I/O is to shutdown the guest. (Which took around >5 minutes - 8 times the time it took on non-SMP guest).

I tried identifying who's responsible for this unaccounted disk I/O, but neither vmstat nor mpstat give any indication as for the reason for this mysterious I/O.

I should add that the guests are using a raw image file, sitting on an ext4 file-system on top of LVM. (The LVM itself either sits on a single disk, or software RAID5 with 3-4 drives)

Will comparing the output of kvm_stat help?

- Gilboa
Comment 13 Bug Zapper 2011-06-01 08:35:26 EDT
This message is a reminder that Fedora 13 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 13.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '13'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 13's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 13 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 14 Bug Zapper 2011-06-29 08:49:50 EDT
Fedora 13 changed to end-of-life (EOL) status on 2011-06-25. Fedora 13 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.