Red Hat Bugzilla – Bug 494295
Fedora 11 slows down and stucks as KVM guest
Last modified: 2009-05-07 14:08:36 EDT
Description of problem: When trying to run F11/x64 guest on KVM (kernel-based virtual machine), it starts installation very slowly, and after a while VM stucks. Host: Fedora 7/x64, KVM-84, Intel Core 2 CPU. (also tried with KVM-83, with same result) Guest: Fedora 11 BETA/x64. Version-Release number of selected component (if applicable): How reproducible: Seems always, but I need more tests to be sure. Steps to Reproduce: 1. Start F11/x64 VM 2. Try to install it. 3. It will start installing packages _extremely slowly_, taking many hours 4. In the end the process dies. Actual results: Slow performance, and dead VM Expected results: F11 must install quickly (say 25 minutes at most). Additional info: I do not know whenever the problem is in Fedora 11, KVM, or something in between, but I would like to find out. Problem is - due to short release cycle of Fedora 11, we need to act quickly, otherwise we might end up with non-KVM compatible Fedora release. Dmesg output: loaded kvm module (kvm-84) BUG: warning at kernel/softirq.c:138/local_bh_enable() (Not tainted) Call Trace: [<ffffffff80229e7b>] local_bh_enable+0x42/0x98 [<ffffffff8025c008>] cond_resched_softirq+0x35/0x4b [<ffffffff8022e9f5>] release_sock+0x59/0xaa [<ffffffff8021c0a4>] tcp_recvmsg+0x9ca/0xadf [<ffffffff802819a2>] default_wake_function+0x0/0xe [<ffffffff8022f84e>] sock_common_recvmsg+0x30/0x45 [<ffffffff803e75b9>] sock_aio_read+0x10c/0x124 [<ffffffff8020c716>] do_sync_read+0xc9/0x10c [<ffffffff80293107>] autoremove_wake_function+0x0/0x2e [<ffffffff8020af1d>] vfs_read+0xde/0x173 [<ffffffff80210606>] sys_read+0x45/0x6e [<ffffffff8025729c>] tracesys+0xdc/0xe1
GDB Output: (after the VM stucked) (gdb) c Continuing. [Thread 1126189392 (LWP 2974) exited] [Thread 1115699536 (LWP 2968) exited] [New Thread 1126189392 (LWP 3169)] [New Thread 1115699536 (LWP 3170)] Couldn't write debug register: Input/output error. (gdb) bt #0 0x0000003a02605711 in __nptl_create_event () from /lib64/libpthread.so.0 #1 0x0000003a02606de7 in pthread_create@@GLIBC_2.2.5 () from /lib64/libpthread.so.0 #2 0x0000000000417eac in qemu_paio_submit (aiocb=0x11600d0, is_write=<value optimized out>) at posix-aio-compat.c:117 #3 0x0000000000418f6c in raw_aio_read (bs=0xc09d00, sector_num=8262296, buf=0x2aaacc959200 "(", nb_sectors=4, cb=<value optimized out>, opaque=<value optimized out>) at block-raw-posix.c:638 #4 0x000000000046472c in bdrv_aio_read (bs=0x3d0f00, sector_num=1094716976, buf=0x0, nb_sectors=4, cb=0x42803950, opaque=0x42803950) at block.c:1333 #5 0x000000000042fc04 in ide_atapi_cmd_read_dma_cb (opaque=0xea5ea8, ret=<value optimized out>) at /root/Linstall/kvm-84/qemu/hw/ide.c:1470 #6 0x000000000042e699 in bmdma_cmd_writeb (opaque=0x3d0f00, addr=<value optimized out>, val=9) at /root/Linstall/kvm-84/qemu/hw/ide.c:2994 #7 0x000000000050f0d8 in kvm_outb (opaque=<value optimized out>, addr=3632, data=0 '\0') at /root/Linstall/kvm-84/qemu/qemu-kvm.c:684 #8 0x00000000005398a7 in handle_io (kvm=0xbdb040, run=0x2aaaaaad0000, vcpu=0) at libkvm.c:722 #9 0x000000000053a1d3 in kvm_run (kvm=0xbdb040, vcpu=0, env=0xc0b9f0) at libkvm.c:951 #10 0x000000000050f839 in kvm_cpu_exec (env=0x0) at /root/Linstall/kvm-84/qemu/qemu-kvm.c:205 #11 0x000000000050fad0 in ap_main_loop (_env=<value optimized out>) at /root/Linstall/kvm-84/qemu/qemu-kvm.c:414 #12 0x0000003a026062f7 in start_thread () from /lib64/libpthread.so.0 #13 0x0000003a016d0fbd in clone () from /lib64/libc.so.6
Qemu/KVM run command: /usr/local/bin/qemu-system-x86_64 -cdrom /isos/linux/Fedora-11-Beta-x86_64-DVD.iso -hda /vm/f11-64.qcow2 -m 512 -name Fedora-11-Beta-x86_64
I think you're hitting the kvmclock problem, which requires a fix to the host kernel. Fedora 7 is not supported so it didn't get the fix. Fedora 9 and 10 should work as hosts. [commit abe6655dd699069b53bcccbc65b2717f60203b12 upstream]
Can you confirm that Fedora 9 and/or 10 work as hosts, or that applying that fix to older kernels fixes the problem?
<finger pointing> The backtrace seems to indicate an NPTL bug. </finger pointing> Alexey, please update the host as suggested.
Yes, with F10 host, Qemu survived. I.e. no crash of KVM userspace. But it is plagued by other errors. 1. slowness before crash, just like F7/x64 hosts. 2. Install finally failed, with Fedora 11 internal error message. (BTW - this only happens when installing from F11/x64 DVD ISO, does not happens with Live CD install) -Alexey, 12.4.2009.
Alexey, You say "slowness before crash". Can you gather more information on the crash? What version of Fedora 11 (link to the ISO/md5sum)? Will try to reproduce here.
Crash only occur on old kernels (2.6.21). On newer kernels (2.6.27) there is no crash. Slowness still exists however. I used: Fedora-11-Beta-x86_64-DVD.iso I won't be available in the next two days. Can you try to install it using default settings (ext4, standard package-set, etc..) on new Qcow2 image, please ? -Alexey, 13.4.2009.
OK, Anaconda crashed (screenshot and crashdump attached) at the end of installation, after all packages had been installed. I'd guess this is not a problem related to KVM, so will assign this bug to Anaconda. There was no noticeable slowdown as you mention, though (AMD host, with 2.6.18 kernel, qcow2). Default installer settings.
Created attachment 339743 [details] fc11 install crash (after package installation)
Created attachment 339744 [details] screenshot containing part of backtrace
# md5sum /root/Fedora-11-Beta-x86_64-DVD.iso 9dc917523132a3d269c59c91b647525a /root/Fedora-11-Beta-x86_64-DVD.iso
*** This bug has been marked as a duplicate of bug 471717 ***
Obviously not a duplicate of bug 471717
Alexey, Please confirm that the error you see is the same as the one in the screenshot. Also please provide more details on the slowdown.
Yes, on newer kernels (2.6.27 from F10), yes.
So the original error is not a KVM problem, but an installer one. Regarding the slowdown, need more details, since as reported before F11 Beta as a guest has been successfully installed on a RHEL5.3 host (with 2.6.18 kernel), without any problems. There is a Fedora/KVM testing day scheduled for 05/May, and any problems should be detected there. I would consider this bug marked as closed.
Confirmed this is not a KVM problem. Anaconda noticeably slows down at the of installation (more especifically the last 20% packages or so. And the "Finishing upgrade procedure, this will take a little while" step takes ages. strace shows all (?) packages verified or something.