Bug 494295 - Fedora 11 slows down and stucks as KVM guest
Fedora 11 slows down and stucks as KVM guest
Status: CLOSED NOTABUG
Product: Fedora
Classification: Fedora
Component: anaconda (Show other bugs)
rawhide
All Linux
low Severity high
: ---
: ---
Assigned To: Anaconda Maintenance Team
Fedora Extras Quality Assurance
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-04-06 04:48 EDT by Alexey Eromenko
Modified: 2009-05-07 14:08 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-05-07 14:08:36 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
fc11 install crash (after package installation) (176.26 KB, text/plain)
2009-04-15 16:49 EDT, Marcelo Tosatti
no flags Details
screenshot containing part of backtrace (116.28 KB, image/png)
2009-04-15 16:50 EDT, Marcelo Tosatti
no flags Details

  None (edit)
Description Alexey Eromenko 2009-04-06 04:48:44 EDT
Description of problem:
When trying to run F11/x64 guest on KVM (kernel-based virtual machine), it starts installation very slowly, and after a while VM stucks.

Host: Fedora 7/x64, KVM-84, Intel Core 2 CPU. (also tried with KVM-83, with same result)
Guest: Fedora 11 BETA/x64.

Version-Release number of selected component (if applicable):


How reproducible:
Seems always, but I need more tests to be sure.

Steps to Reproduce:
1. Start F11/x64 VM
2. Try to install it.
3. It will start installing packages _extremely slowly_, taking many hours
4. In the end the process dies.
  
Actual results:
Slow performance, and dead VM

Expected results:
F11 must install quickly (say 25 minutes at most).

Additional info:
I do not know whenever the problem is in Fedora 11, KVM, or something in between, but I would like to find out.

Problem is - due to short release cycle of Fedora 11, we need to act quickly, otherwise we might end up with non-KVM compatible Fedora release.

Dmesg output:
loaded kvm module (kvm-84)
BUG: warning at kernel/softirq.c:138/local_bh_enable() (Not tainted)

Call Trace:
 [<ffffffff80229e7b>] local_bh_enable+0x42/0x98
 [<ffffffff8025c008>] cond_resched_softirq+0x35/0x4b
 [<ffffffff8022e9f5>] release_sock+0x59/0xaa
 [<ffffffff8021c0a4>] tcp_recvmsg+0x9ca/0xadf
 [<ffffffff802819a2>] default_wake_function+0x0/0xe
 [<ffffffff8022f84e>] sock_common_recvmsg+0x30/0x45
 [<ffffffff803e75b9>] sock_aio_read+0x10c/0x124
 [<ffffffff8020c716>] do_sync_read+0xc9/0x10c
 [<ffffffff80293107>] autoremove_wake_function+0x0/0x2e
 [<ffffffff8020af1d>] vfs_read+0xde/0x173
 [<ffffffff80210606>] sys_read+0x45/0x6e
 [<ffffffff8025729c>] tracesys+0xdc/0xe1
Comment 1 Alexey Eromenko 2009-04-06 05:18:12 EDT
GDB Output: (after the VM stucked)

(gdb) c
Continuing.
[Thread 1126189392 (LWP 2974) exited]
[Thread 1115699536 (LWP 2968) exited]
[New Thread 1126189392 (LWP 3169)]
[New Thread 1115699536 (LWP 3170)]
Couldn't write debug register: Input/output error.
(gdb) bt
#0  0x0000003a02605711 in __nptl_create_event () from /lib64/libpthread.so.0
#1  0x0000003a02606de7 in pthread_create@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#2  0x0000000000417eac in qemu_paio_submit (aiocb=0x11600d0,
    is_write=<value optimized out>) at posix-aio-compat.c:117
#3  0x0000000000418f6c in raw_aio_read (bs=0xc09d00, sector_num=8262296,
    buf=0x2aaacc959200 "(", nb_sectors=4, cb=<value optimized out>,
    opaque=<value optimized out>) at block-raw-posix.c:638
#4  0x000000000046472c in bdrv_aio_read (bs=0x3d0f00, sector_num=1094716976, buf=0x0,
    nb_sectors=4, cb=0x42803950, opaque=0x42803950) at block.c:1333
#5  0x000000000042fc04 in ide_atapi_cmd_read_dma_cb (opaque=0xea5ea8,
    ret=<value optimized out>) at /root/Linstall/kvm-84/qemu/hw/ide.c:1470
#6  0x000000000042e699 in bmdma_cmd_writeb (opaque=0x3d0f00, addr=<value optimized out>,
    val=9) at /root/Linstall/kvm-84/qemu/hw/ide.c:2994
#7  0x000000000050f0d8 in kvm_outb (opaque=<value optimized out>, addr=3632, data=0 '\0')
    at /root/Linstall/kvm-84/qemu/qemu-kvm.c:684
#8  0x00000000005398a7 in handle_io (kvm=0xbdb040, run=0x2aaaaaad0000, vcpu=0)
    at libkvm.c:722
#9  0x000000000053a1d3 in kvm_run (kvm=0xbdb040, vcpu=0, env=0xc0b9f0) at libkvm.c:951
#10 0x000000000050f839 in kvm_cpu_exec (env=0x0)
    at /root/Linstall/kvm-84/qemu/qemu-kvm.c:205
#11 0x000000000050fad0 in ap_main_loop (_env=<value optimized out>)
    at /root/Linstall/kvm-84/qemu/qemu-kvm.c:414
#12 0x0000003a026062f7 in start_thread () from /lib64/libpthread.so.0
#13 0x0000003a016d0fbd in clone () from /lib64/libc.so.6
Comment 2 Alexey Eromenko 2009-04-06 10:19:54 EDT
Qemu/KVM run command:
/usr/local/bin/qemu-system-x86_64 -cdrom /isos/linux/Fedora-11-Beta-x86_64-DVD.iso  -hda /vm/f11-64.qcow2 -m 512 -name Fedora-11-Beta-x86_64
Comment 3 Chuck Ebbert 2009-04-07 15:21:39 EDT
I think you're hitting the kvmclock problem, which requires a fix to the host kernel. Fedora 7 is not supported so it didn't get the fix. Fedora 9 and 10 should work as hosts. [commit abe6655dd699069b53bcccbc65b2717f60203b12 upstream]
Comment 4 Chuck Ebbert 2009-04-08 10:32:47 EDT
Can you confirm that Fedora 9 and/or 10 work as hosts, or that applying that fix to older kernels fixes the problem?
Comment 5 Marcelo Tosatti 2009-04-09 05:29:37 EDT
<finger pointing>
The backtrace seems to indicate an NPTL bug.
</finger pointing>

Alexey, please update the host as suggested.
Comment 6 Alexey Eromenko 2009-04-12 12:01:52 EDT
Yes, with F10 host, Qemu survived. I.e. no crash of KVM userspace.

But it is plagued by other errors.

1. slowness before crash, just like F7/x64 hosts.
2. Install finally failed, with Fedora 11 internal error message.
(BTW - this only happens when installing from F11/x64 DVD ISO, does not happens with Live CD install)

-Alexey, 12.4.2009.
Comment 7 Marcelo Tosatti 2009-04-12 13:45:12 EDT
Alexey,

You say "slowness before crash". Can you gather more information on the crash?

What version of Fedora 11 (link to the ISO/md5sum)? Will try to reproduce here.
Comment 8 Alexey Eromenko 2009-04-13 04:12:34 EDT
Crash only occur on old kernels (2.6.21).

On newer kernels (2.6.27) there is no crash. 

Slowness still exists however.

I used:
Fedora-11-Beta-x86_64-DVD.iso

I won't be available in the next two days.

Can you try to install it using default settings (ext4, standard package-set, etc..) on new Qcow2 image, please ?

-Alexey, 13.4.2009.
Comment 9 Marcelo Tosatti 2009-04-15 16:47:57 EDT
OK, Anaconda crashed (screenshot and crashdump attached) at the end of installation, after all packages had been installed.

I'd guess this is not a problem related to KVM, so will assign this bug to Anaconda.

There was no noticeable slowdown as you mention, though (AMD host, with 2.6.18 kernel, qcow2). Default installer settings.
Comment 10 Marcelo Tosatti 2009-04-15 16:49:26 EDT
Created attachment 339743 [details]
fc11 install crash (after package installation)
Comment 11 Marcelo Tosatti 2009-04-15 16:50:30 EDT
Created attachment 339744 [details]
screenshot containing part of backtrace
Comment 12 Marcelo Tosatti 2009-04-15 16:52:29 EDT
# md5sum /root/Fedora-11-Beta-x86_64-DVD.iso 
9dc917523132a3d269c59c91b647525a  /root/Fedora-11-Beta-x86_64-DVD.iso
Comment 13 Chris Lumens 2009-04-15 17:12:12 EDT

*** This bug has been marked as a duplicate of bug 471717 ***
Comment 14 Chuck Ebbert 2009-04-15 18:17:22 EDT
Obviously not a duplicate of bug 471717
Comment 15 Marcelo Tosatti 2009-04-16 00:24:30 EDT
Alexey,

Please confirm that the error you see is the same as the one in the screenshot.

Also please provide more details on the slowdown.
Comment 16 Alexey Eromenko 2009-04-16 08:48:57 EDT
Yes, on newer kernels (2.6.27 from F10), yes.
Comment 17 Marcelo Tosatti 2009-04-30 17:58:52 EDT
So the original error is not a KVM problem, but an installer one.

Regarding the slowdown, need more details, since as reported before F11 Beta as a guest has been successfully installed on a RHEL5.3 host (with 2.6.18 kernel), without any problems.

There is a Fedora/KVM testing day scheduled for 05/May, and any problems should be detected there. I would consider this bug marked as closed.
Comment 18 Marcelo Tosatti 2009-05-07 14:08:36 EDT
Confirmed this is not a KVM problem. Anaconda noticeably slows down at the
of installation (more especifically the last 20% packages or so.

And the "Finishing upgrade procedure, this will take a little while" step takes ages. strace shows all (?) packages verified or something.

Note You need to log in before you can comment on or make changes to this bug.