I have a (mostly) F10 machine configured with a few KVM guests. Running recent-ish fedora releases (or rawhide) under KVM is very unstable. 2.6.27* kernels seem to regularly hang hard, particularly under substantial I/O. yum update tends to make them hang. 2.6.28 kernels usually crash under heavy I/O. Other guests on the same machine (windows 2k8 and recent opensolaris) don't seem to be affected by this. They seem to run without any problem. I'm also able to run similar guests (rawhide and such) under Xen without any problems. This leads me to believe that this is due to a problem with KVM. I've been running the kvm package version that currently ships with F10 (-74-6.f10). Today I also tried the latest F11 package (-79-1.f11). I get similar results with both. I'm currently running my rawhide guest with virtio drivers, but I get similar results when I use the "normal" drivers as well. Host OS is running kernel-2.6.27.5-117.fc10.x86_64 Let me know if you need other info, or want me to test patches.
Created attachment 326378 [details] set of oops messages from rawhide guest running 2.6.28-ish kernel Set of oops messages from rawhide guest. The problems under 2.6.28-ish kernels seem to manifest themselves as oopses there. There doesn't seem to be any consistency in how the oopses look, however. They're generally all over the place. 2.6.27-ish kernels usually just hang hard With either guest kernel, after the problem occurs the guest usually goes into 100% CPU utilization.
Chris Lalancette suggested changing the clocksource. It looks like the kernel is defaulting to clock-kvm. The first oops in the attached log though seems to be coming from that code. I changed the timer to acpi_pm and it seems to be behaving much more stably under 2.6.28. I'm going to plan to do some more testing to be sure, but no oopses yet even after a yum update and other package installation and removal. Usually that would make it oops quickly.
So far so good on 2.6.27 when I boot with clocksource=acpi_pm... On 2.6.28-ish kernels, the lvm volume group fails to activate when I boot with that option. I can boot without that option and if it survives long enough for me to log in and change the clocksource under /sys, then it seems to be reasonably stable as well. I think that more or less implicates kvm-clock as the source of the problems I was experiencing.
I can confirm this. Host is F10 x86_64 on Athlon X2, guests are F10 i686, only happens with kvm-clock. It seems to happen under high load in the guest, that might be helpful for reproducing this.
I'm seeing very similar problems for my new Fedora 10 x86_64 kvm host and guest, using the latest Fedora 10 kernel (2.6.27.9-159) for both the host and the guest. While I can't get the guest to hang with yum et. al., I've twice now had to hard reset the guest because it hung. When the guest hangs, the kvm process on the host consumes 100% of the CPU. The guest has yet to achive >12 hours of uptime without hanging. I suspect this bug is also the source of Fedora 10 kvm guest installer woes; see bug 474116. I've switched the clocksource of the guest from kvm-clock to acpi_pm (via grub.conf, as the 2.6.27 kernel doesn't seem to have any LVM issues when booting with that clocksource). I'll report back on my experiences.
My kvm virtual guest now has 36+ hours of uptime with clocksource=acpi_pm, so I'm fairly confident that clocksource=kvm-clock was the problem. I'd be happy to help test patches...
Another clue: This might be triggered only on hosts with CPU frequency scaling enabled. Can others confirm that or rule that out? This looks like the same bug: http://lkml.indiana.edu/hypermail/linux/kernel/0812.2/index.html#02381
I'm seeing what seems to be this problem, and I'm using frequency scaling.
*** Bug 479407 has been marked as a duplicate of this bug. ***
I'm seeing this problem, and I'm using CPU frequency scaling. When I have a chance, I'll test the combination of (no CPU frequency scaling on the host, clocksource=clock-kvm on the guest) and see if I can reproduce a hang.
I disabled CPU freq scaling on my host (basically, just set the governor to "performance"), and it seems to be a lot more stable with kvm-clock. I'm patching the guest with a big yum update now. In the past, that's generally made it hang after a while, but no lockup so far today.
Can the people seeing the kvm-clock hangs please give some details on the host hardware? Specifically about the CPU? Does it happen with UP hosts? Or SMP hosts only? Multicore CPUs? Can you place /proc/cpuinfo content (processor 0 is enougth) here? thanks.
Oh, and the complete boot messages for the oops log (attachment #1 [details]) would be great too in case it is available ...
(In reply to comment #13) > Oh, and the complete boot messages for the oops log (attachment #1 [details]) would be > great too in case it is available ... Unfortunately, I don't have the boot messages from that particular panic. Reproducing panics with rawhide guest kernels from mid-december timeframe is fairly easy, but as I mentioned before they seem to show up in all sorts of code. It was pure luck that that one showed a panic in kvm-clock. Most of the time they show up elsewhere.
Created attachment 329453 [details] Another /proc/cpuinfo Here's from my single slot dual core system. Also an AMD Athlon system, although with a different CPU model.
Created attachment 329472 [details] test patch Ok, seems we are in trouble in case cpus / cores on a SMP machine run on different cpu (and tsc) frequencies, because a single global variable keeps the current tsc frequency, which simply isn't going to work then ...
Thanks Gerd, I'll try to build it and give it a test later today (unless you are planning to build a kernel in koji with this patch soon in which case I'll just wait for that). Do I need to apply this patch to the host kernel, the guest kernel, or both?
Hmm... if it uses tsc, does it properly cope with unstable and unsynchronised tsc:s? The host kernel complains about this fact on these CPUs.
Re #18: It is a host kernel patch. Re #19: In theory it should, in practice it doesn't correctly on some SMP boxes.
*lpj = cpufreq_scale(loops_per_jiffy_ref, ref_freq, freq->new); tsc_khz = cpufreq_scale(tsc_khz_ref, ref_freq, freq->new); + __get_cpu_var(cpu_tsc_khz) = tsc_khz; if (!(freq->flags & CPUFREQ_CONST_LOOPS)) mark_tsc_unstable("cpufreq changes"); } there's no guarantee that the handler will run in the cpu that got the frequency change. We have to update it to freq->cpu, not current cpu. Right ?
*** Bug 474116 has been marked as a duplicate of this bug. ***
Created attachment 329614 [details] And another cpuinfo I'm seeing this too, here is my cpuinfo.
Created attachment 329660 [details] Yet another cpuinfo
I tested the patch in comment #17 but it didn't seem to help. The F10 installer still eventually hung. I redirected the console output to the serial port but there was no oops message and unfortunately I was unable to get any sort of sysrq info.
For the record, I applied the patch to the 2.6.28.1-17 from koji.
I saw some traffic on LKML mentioning that KVM guests are hanging under 2.6.28, so I can't be sure that my testing was actually good. If you can provide a patch that's backported to 2.6.27-ish kernels I'll be happy to test it. This one wouldn't apply when I tried it.
Created attachment 329812 [details] Patch that makes my Opteron rev F work This patch works for me. Basically it saves tsc_khz by cpu instead of using a global variable. It works with my Dual socket Opteron revF, that should have the same issues that the Athlons that appear in the cpuinfo variables. Could people give it a try?
Created attachment 329834 [details] /proc/cpuinfo (Intel(R) Xeon(R) CPU E5420) > Could people give it a try? I'm consistently experiencing flakiness while installing rawhide/x86_64 as a kvm guest on my F10/x86_64 dual quad-core system. I'll be happy to give a kernel a spin.
Okay, building a rawhide kernel with a workaround patch to disable kvmclock on hosts which don't have a synchronised TSC See here: http://www.redhat.com/archives/fedora-kernel-list/2009-January/msg00109.html Should be in kernel-2.6.29-0.66.rc3.fc11 * Thu Jan 29 2009 Mark McLoughlin <markmc> - Temporarily disable kvmclock, fix for upstream in progress (#475598) Note: keeping this bug open for F10
(In reply to comment #30) > Okay, building a rawhide kernel with a workaround patch to disable kvmclock on > hosts which don't have a synchronised TSC Available here: https://koji.fedoraproject.org/koji/buildinfo?buildID=80794
(In reply to comment #30) > Okay, building a rawhide kernel with a workaround patch to disable kvmclock on > hosts which don't have a synchronised TSC Looks like there's a 2.6.29-rc3 build with this fix in dist-f10-updates-candidate now: https://koji.fedoraproject.org/koji/buildinfo?buildID=80840 * Thu Jan 29 2009 Chuck Ebbert <cebbert> 2.6.29-0.4.rc3 - Copy firewire updates, at76 driver, a compile fix and a KVM clock fix from rawhide. - Temporarily disable CONFIG_DRM_RADEON_KMS Not sure what's going on there.
Created attachment 330621 [details] patch -- unsync tsc workaround patch It looks like the recent f10 updates candidate kernels contain this patch. It appears to just disable kvm-clock in guests -- not sure, I'm not that familiar with KVM's innards. This seems like it would be appropriate for f10-ish kernels until this gets straightened out upstream.
Created attachment 330630 [details] Test patch Test kernel (still building right now). http://koji.fedoraproject.org/koji/taskinfo?taskID=1099354
This F10 2.6.27 update will contain the workaround: * Sat Feb 07 2009 Chuck Ebbert <cebbert> 2.6.27.15-170.2.18.rc1 - Copy kvmclock fix from our 2.6.29 kernel.
Is there a way to install Fedora 10 in KVM now? (without patching kernels?) Maybe add some kernel boot parameter via GRUB in guest ?
I was able to install F10 by adding clocksource=acpi_pm to the guest's kernel command line. With the kernel mentioned in comment #35 running on the host, you might also be able to install the guest with no special options. I haven't tested and install, but I haven't seen any lockups on my existing guests since booting the dom0 to that kernel.
The F10 update is just waiting to be pushed: https://admin.fedoraproject.org/updates/kernel-2.6.27.15-170.2.19.fc10 For reference, Avi has queued up the workaround for 2.6.29 here: http://git.kernel.org/?p=linux/kernel/git/avi/kvm.git;a=commit;h=f160ca75 and the real fix is queued for 2.6.30 here: http://git.kernel.org/?p=linux/kernel/git/avi/kvm.git;a=commit;h=48aa74b8
unfortunately this doesn't seems to solve the problem:-( while 2.6.27.5-117.fc10.i686 with clocksource=acpi_pm working while 2.6.27.15-170.2.19.fc10.i686 without it also hang randomly.
(In reply to comment #39) > unfortunately this doesn't seems to solve the problem:-( > while 2.6.27.5-117.fc10.i686 with clocksource=acpi_pm working while > 2.6.27.15-170.2.19.fc10.i686 without it also hang randomly. Could you attach the contents of your /proc/cpuinfo ?
Created attachment 331670 [details] here is my cpuinfo
(In reply to comment #39) > unfortunately this doesn't seems to solve the problem:-( > while 2.6.27.5-117.fc10.i686 with clocksource=acpi_pm working while > 2.6.27.15-170.2.19.fc10.i686 without it also hang randomly. You have constant_tsc, so it sounds like this may not be the same issue. You're absolutely certain that you are seeing random hangs that are resolved by booting the guest with clocksource=acpi_pm? (Yes, I know I'm asking you to repeat yourself - sorry - but we need as much further details on the bug you're seeing as possible)
i'm absolutely certain that two guests (i386 and x86_64) working without problem under load with: ---------------------------------- kernel /vmlinuz-2.6.27.5-117.fc10.i686 ro root=/dev/VolGroup00/LogVol00 clocksource=acpi_pm kernel /vmlinuz-2.6.27.5-117.fc10.x86_64 ro root=/dev/VolGroup00/LogVol00 clocksource=acpi_pm ---------------------------------- and at the same time if i switch any of the guest to the mentioned and downloaded from koji kernel (without: clocksource=acpi_pm) ---------------------------------- kernel /vmlinuz-2.6.27.15-170.2.19.fc10.i686 ro root=/dev/VolGroup00/LogVol00 kernel /vmlinuz-2.6.27.15-170.2.19.fc10.x86_64 ro root=/dev/VolGroup00/LogVol00 ---------------------------------- then both guests hang within minutes or many be during boot.
Created attachment 332332 [details] cpuinfo Linux zeus.themaw.net 2.6.27.12-78.2.8.fc9.i686 #1 SMP Mon Jan 19 20:14:35 EST 2009 i686 i686 i386 GNU/Linux kvm-65-15.fc9.i386 I also see a hang (and very slow response) when trying to install Rawhide x86_64.
(In reply to comment #44) > Created an attachment (id=332332) [details] > cpuinfo > > Linux zeus.themaw.net 2.6.27.12-78.2.8.fc9.i686 #1 SMP Mon Jan 19 20:14:35 EST > 2009 i686 i686 i386 GNU/Linux > > kvm-65-15.fc9.i386 Okay, this is F-9, which doesn't currently have the workaround patch. But you have constant_tsc anyway, so it shouldn't make a difference. > I also see a hang (and very slow response) when trying to install > Rawhide x86_64. Does this problem go away when booting the guest with clocksource=acpi_pm?
(In reply to comment #45) > (In reply to comment #44) > > Created an attachment (id=332332) [details] [details] > > cpuinfo > > > > Linux zeus.themaw.net 2.6.27.12-78.2.8.fc9.i686 #1 SMP Mon Jan 19 20:14:35 EST > > 2009 i686 i686 i386 GNU/Linux > > > > kvm-65-15.fc9.i386 > > Okay, this is F-9, which doesn't currently have the workaround patch. But you > have constant_tsc anyway, so it shouldn't make a difference. > > > I also see a hang (and very slow response) when trying to install > > Rawhide x86_64. > > Does this problem go away when booting the guest with clocksource=acpi_pm? Nope, but I've gone to F-10, x86_64 just now and will see how things go.
(In reply to comment #46) > (In reply to comment #45) > > Does this problem go away when booting the guest with clocksource=acpi_pm? > > Nope Okay, if changing the clocksource doesn't fix your problem then it probably isn't kvmclock related. Please file another bug if the problem persists.
Maybe we need to re-open BZ# 474116? I have the following symptoms: * Fedora 10/x86-64 fails to install as KVM guest. Hangs at "installing bootloader" step. * Fedora 10/x86-64 with current updates is the host OS. This behavior appears with both the Fedora stock kernel or a custom 2.6.29-rc5 kernel running on the host. I tried "clocksource=acpi_pm" for the Fedora 10 guest, without success. Hardware: Intel ICH10 /proc/cpuinfo selected lines: ... vendor_id : GenuineIntel cpu family : 6 model : 26 model name : Genuine Intel(R) CPU 000 @ 3.20GHz stepping : 4 ... flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
(In reply to comment #46) > (In reply to comment #45) > > (In reply to comment #44) > > > Created an attachment (id=332332) [details] [details] [details] > > > cpuinfo > > > > > > Linux zeus.themaw.net 2.6.27.12-78.2.8.fc9.i686 #1 SMP Mon Jan 19 20:14:35 EST > > > 2009 i686 i686 i386 GNU/Linux > > > > > > kvm-65-15.fc9.i386 > > > > Okay, this is F-9, which doesn't currently have the workaround patch. But you > > have constant_tsc anyway, so it shouldn't make a difference. > > > > > I also see a hang (and very slow response) when trying to install > > > Rawhide x86_64. > > > > Does this problem go away when booting the guest with clocksource=acpi_pm? > > Nope, but I've gone to F-10, x86_64 just now and will see how things go. Host: 2.6.27.15-170.2.24.fc10.x86_64 #1 SMP Wed Feb 11 23:14:31 EST 2009 x86_64 x86_64 x86_64 GNU/Linux F-10, i386 install hung at installing bootloader. F-10, x86_64 install hung at Running anaconda.
Okay, this bug covers the case where guests using kvmclock hang on a host machine does not have constant_tsc This is fixed by disabling kvmclock on host machines which don't have constant_tsc: http://git.kernel.org/?p=linux/kernel/git/avi/kvm.git;a=commitdiff;h=f160ca75 This patch has been in F10 since kernel-2.6.27.15-170.2.19.fc10 Closing this bug - any other guest hangs should be filed as separate bugs.