Bug 494295

Summary: Fedora 11 slows down and stucks as KVM guest
Product: [Fedora] Fedora Reporter: Alexey Eromenko <al4321>
Component: anacondaAssignee: Anaconda Maintenance Team <anaconda-maint-list>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: low    
Version: rawhideCC: anaconda-maint-list, itamar, kernel-maint, mtosatti, pjones, quintela, rjones, rmaximo, vanmeeuwen+fedora
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-05-07 18:08:36 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
fc11 install crash (after package installation)
none
screenshot containing part of backtrace none

Description Alexey Eromenko 2009-04-06 08:48:44 UTC
Description of problem:
When trying to run F11/x64 guest on KVM (kernel-based virtual machine), it starts installation very slowly, and after a while VM stucks.

Host: Fedora 7/x64, KVM-84, Intel Core 2 CPU. (also tried with KVM-83, with same result)
Guest: Fedora 11 BETA/x64.

Version-Release number of selected component (if applicable):


How reproducible:
Seems always, but I need more tests to be sure.

Steps to Reproduce:
1. Start F11/x64 VM
2. Try to install it.
3. It will start installing packages _extremely slowly_, taking many hours
4. In the end the process dies.
  
Actual results:
Slow performance, and dead VM

Expected results:
F11 must install quickly (say 25 minutes at most).

Additional info:
I do not know whenever the problem is in Fedora 11, KVM, or something in between, but I would like to find out.

Problem is - due to short release cycle of Fedora 11, we need to act quickly, otherwise we might end up with non-KVM compatible Fedora release.

Dmesg output:
loaded kvm module (kvm-84)
BUG: warning at kernel/softirq.c:138/local_bh_enable() (Not tainted)

Call Trace:
 [<ffffffff80229e7b>] local_bh_enable+0x42/0x98
 [<ffffffff8025c008>] cond_resched_softirq+0x35/0x4b
 [<ffffffff8022e9f5>] release_sock+0x59/0xaa
 [<ffffffff8021c0a4>] tcp_recvmsg+0x9ca/0xadf
 [<ffffffff802819a2>] default_wake_function+0x0/0xe
 [<ffffffff8022f84e>] sock_common_recvmsg+0x30/0x45
 [<ffffffff803e75b9>] sock_aio_read+0x10c/0x124
 [<ffffffff8020c716>] do_sync_read+0xc9/0x10c
 [<ffffffff80293107>] autoremove_wake_function+0x0/0x2e
 [<ffffffff8020af1d>] vfs_read+0xde/0x173
 [<ffffffff80210606>] sys_read+0x45/0x6e
 [<ffffffff8025729c>] tracesys+0xdc/0xe1

Comment 1 Alexey Eromenko 2009-04-06 09:18:12 UTC
GDB Output: (after the VM stucked)

(gdb) c
Continuing.
[Thread 1126189392 (LWP 2974) exited]
[Thread 1115699536 (LWP 2968) exited]
[New Thread 1126189392 (LWP 3169)]
[New Thread 1115699536 (LWP 3170)]
Couldn't write debug register: Input/output error.
(gdb) bt
#0  0x0000003a02605711 in __nptl_create_event () from /lib64/libpthread.so.0
#1  0x0000003a02606de7 in pthread_create@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#2  0x0000000000417eac in qemu_paio_submit (aiocb=0x11600d0,
    is_write=<value optimized out>) at posix-aio-compat.c:117
#3  0x0000000000418f6c in raw_aio_read (bs=0xc09d00, sector_num=8262296,
    buf=0x2aaacc959200 "(", nb_sectors=4, cb=<value optimized out>,
    opaque=<value optimized out>) at block-raw-posix.c:638
#4  0x000000000046472c in bdrv_aio_read (bs=0x3d0f00, sector_num=1094716976, buf=0x0,
    nb_sectors=4, cb=0x42803950, opaque=0x42803950) at block.c:1333
#5  0x000000000042fc04 in ide_atapi_cmd_read_dma_cb (opaque=0xea5ea8,
    ret=<value optimized out>) at /root/Linstall/kvm-84/qemu/hw/ide.c:1470
#6  0x000000000042e699 in bmdma_cmd_writeb (opaque=0x3d0f00, addr=<value optimized out>,
    val=9) at /root/Linstall/kvm-84/qemu/hw/ide.c:2994
#7  0x000000000050f0d8 in kvm_outb (opaque=<value optimized out>, addr=3632, data=0 '\0')
    at /root/Linstall/kvm-84/qemu/qemu-kvm.c:684
#8  0x00000000005398a7 in handle_io (kvm=0xbdb040, run=0x2aaaaaad0000, vcpu=0)
    at libkvm.c:722
#9  0x000000000053a1d3 in kvm_run (kvm=0xbdb040, vcpu=0, env=0xc0b9f0) at libkvm.c:951
#10 0x000000000050f839 in kvm_cpu_exec (env=0x0)
    at /root/Linstall/kvm-84/qemu/qemu-kvm.c:205
#11 0x000000000050fad0 in ap_main_loop (_env=<value optimized out>)
    at /root/Linstall/kvm-84/qemu/qemu-kvm.c:414
#12 0x0000003a026062f7 in start_thread () from /lib64/libpthread.so.0
#13 0x0000003a016d0fbd in clone () from /lib64/libc.so.6

Comment 2 Alexey Eromenko 2009-04-06 14:19:54 UTC
Qemu/KVM run command:
/usr/local/bin/qemu-system-x86_64 -cdrom /isos/linux/Fedora-11-Beta-x86_64-DVD.iso  -hda /vm/f11-64.qcow2 -m 512 -name Fedora-11-Beta-x86_64

Comment 3 Chuck Ebbert 2009-04-07 19:21:39 UTC
I think you're hitting the kvmclock problem, which requires a fix to the host kernel. Fedora 7 is not supported so it didn't get the fix. Fedora 9 and 10 should work as hosts. [commit abe6655dd699069b53bcccbc65b2717f60203b12 upstream]

Comment 4 Chuck Ebbert 2009-04-08 14:32:47 UTC
Can you confirm that Fedora 9 and/or 10 work as hosts, or that applying that fix to older kernels fixes the problem?

Comment 5 Marcelo Tosatti 2009-04-09 09:29:37 UTC
<finger pointing>
The backtrace seems to indicate an NPTL bug.
</finger pointing>

Alexey, please update the host as suggested.

Comment 6 Alexey Eromenko 2009-04-12 16:01:52 UTC
Yes, with F10 host, Qemu survived. I.e. no crash of KVM userspace.

But it is plagued by other errors.

1. slowness before crash, just like F7/x64 hosts.
2. Install finally failed, with Fedora 11 internal error message.
(BTW - this only happens when installing from F11/x64 DVD ISO, does not happens with Live CD install)

-Alexey, 12.4.2009.

Comment 7 Marcelo Tosatti 2009-04-12 17:45:12 UTC
Alexey,

You say "slowness before crash". Can you gather more information on the crash?

What version of Fedora 11 (link to the ISO/md5sum)? Will try to reproduce here.

Comment 8 Alexey Eromenko 2009-04-13 08:12:34 UTC
Crash only occur on old kernels (2.6.21).

On newer kernels (2.6.27) there is no crash. 

Slowness still exists however.

I used:
Fedora-11-Beta-x86_64-DVD.iso

I won't be available in the next two days.

Can you try to install it using default settings (ext4, standard package-set, etc..) on new Qcow2 image, please ?

-Alexey, 13.4.2009.

Comment 9 Marcelo Tosatti 2009-04-15 20:47:57 UTC
OK, Anaconda crashed (screenshot and crashdump attached) at the end of installation, after all packages had been installed.

I'd guess this is not a problem related to KVM, so will assign this bug to Anaconda.

There was no noticeable slowdown as you mention, though (AMD host, with 2.6.18 kernel, qcow2). Default installer settings.

Comment 10 Marcelo Tosatti 2009-04-15 20:49:26 UTC
Created attachment 339743 [details]
fc11 install crash (after package installation)

Comment 11 Marcelo Tosatti 2009-04-15 20:50:30 UTC
Created attachment 339744 [details]
screenshot containing part of backtrace

Comment 12 Marcelo Tosatti 2009-04-15 20:52:29 UTC
# md5sum /root/Fedora-11-Beta-x86_64-DVD.iso 
9dc917523132a3d269c59c91b647525a  /root/Fedora-11-Beta-x86_64-DVD.iso

Comment 13 Chris Lumens 2009-04-15 21:12:12 UTC

*** This bug has been marked as a duplicate of bug 471717 ***

Comment 14 Chuck Ebbert 2009-04-15 22:17:22 UTC
Obviously not a duplicate of bug 471717

Comment 15 Marcelo Tosatti 2009-04-16 04:24:30 UTC
Alexey,

Please confirm that the error you see is the same as the one in the screenshot.

Also please provide more details on the slowdown.

Comment 16 Alexey Eromenko 2009-04-16 12:48:57 UTC
Yes, on newer kernels (2.6.27 from F10), yes.

Comment 17 Marcelo Tosatti 2009-04-30 21:58:52 UTC
So the original error is not a KVM problem, but an installer one.

Regarding the slowdown, need more details, since as reported before F11 Beta as a guest has been successfully installed on a RHEL5.3 host (with 2.6.18 kernel), without any problems.

There is a Fedora/KVM testing day scheduled for 05/May, and any problems should be detected there. I would consider this bug marked as closed.

Comment 18 Marcelo Tosatti 2009-05-07 18:08:36 UTC
Confirmed this is not a KVM problem. Anaconda noticeably slows down at the
of installation (more especifically the last 20% packages or so.

And the "Finishing upgrade procedure, this will take a little while" step takes ages. strace shows all (?) packages verified or something.