This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours
Bug 475598 - recent fedora guests in KVM are unstable with kvm-clock and !constant_tsc
recent fedora guests in KVM are unstable with kvm-clock and !constant_tsc
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: kvm (Show other bugs)
10
x86_64 Linux
medium Severity medium
: ---
: ---
Assigned To: Glauber Costa
Fedora Extras Quality Assurance
:
: 479407 (view as bug list)
Depends On:
Blocks: F11VirtTarget
  Show dependency treegraph
 
Reported: 2008-12-09 13:39 EST by Jeff Layton
Modified: 2014-06-18 03:38 EDT (History)
23 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-03-20 12:15:23 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
set of oops messages from rawhide guest running 2.6.28-ish kernel (37.34 KB, text/plain)
2008-12-09 13:44 EST, Jeff Layton
no flags Details
/proc/cpuinfo from host OS (1.42 KB, text/plain)
2009-01-20 06:34 EST, Jeff Layton
no flags Details
Another /proc/cpuinfo (1.47 KB, text/plain)
2009-01-20 07:39 EST, Pierre Ossman
no flags Details
test patch (2.09 KB, patch)
2009-01-20 10:09 EST, Gerd Hoffmann
no flags Details | Diff
And another cpuinfo (1.45 KB, text/plain)
2009-01-21 10:29 EST, Hans de Goede
no flags Details
Yet another cpuinfo (1.42 KB, application/octet-stream)
2009-01-21 16:23 EST, David Lutterkort
no flags Details
Patch that makes my Opteron rev F work (2.60 KB, patch)
2009-01-23 07:23 EST, Juan Quintela
no flags Details | Diff
/proc/cpuinfo (Intel(R) Xeon(R) CPU E5420) (5.55 KB, text/plain)
2009-01-23 08:32 EST, James Laska
no flags Details
patch -- unsync tsc workaround patch (1.44 KB, patch)
2009-02-02 07:16 EST, Jeff Layton
no flags Details | Diff
Test patch (5.78 KB, patch)
2009-02-02 10:07 EST, Gerd Hoffmann
no flags Details | Diff
here is my cpuinfo (2.46 KB, text/plain)
2009-02-12 04:44 EST, Levente Farkas
no flags Details
cpuinfo (1.31 KB, text/plain)
2009-02-17 21:53 EST, Ian Kent
no flags Details

  None (edit)
Description Jeff Layton 2008-12-09 13:39:53 EST
I have a (mostly) F10 machine configured with a few KVM guests. Running recent-ish fedora releases (or rawhide) under KVM is very unstable. 2.6.27* kernels seem to regularly hang hard, particularly under substantial I/O. yum update tends to make them hang. 2.6.28 kernels usually crash under heavy I/O.

Other guests on the same machine (windows 2k8 and recent opensolaris) don't seem to be affected by this. They seem to run without any problem. I'm also able to run similar guests (rawhide and such) under Xen without any problems. This leads me to believe that this is due to a problem with KVM.

I've been running the kvm package version that currently ships with F10 (-74-6.f10). Today I also tried the latest F11 package (-79-1.f11). I get similar results with both.

I'm currently running my rawhide guest with virtio drivers, but I get similar results when I use the "normal" drivers as well.

Host OS is running kernel-2.6.27.5-117.fc10.x86_64

Let me know if you need other info, or want me to test patches.
Comment 1 Jeff Layton 2008-12-09 13:44:28 EST
Created attachment 326378 [details]
set of oops messages from rawhide guest running 2.6.28-ish kernel

Set of oops messages from rawhide guest. The problems under 2.6.28-ish kernels seem to manifest themselves as oopses there. There doesn't seem to be any consistency in how the oopses look, however. They're generally all over the place.

2.6.27-ish kernels usually just hang hard

With either guest kernel, after the problem occurs the guest usually goes into 100% CPU utilization.
Comment 2 Jeff Layton 2008-12-11 08:57:44 EST
Chris Lalancette suggested changing the clocksource. It looks like the kernel is defaulting to clock-kvm. The first oops in the attached log though seems to be coming from that code.

I changed the timer to acpi_pm and it seems to be behaving much more stably under 2.6.28. I'm going to plan to do some more testing to be sure, but no oopses yet even after a yum update and other package installation and removal. Usually that would make it oops quickly.
Comment 3 Jeff Layton 2008-12-12 08:50:52 EST
So far so good on 2.6.27 when I boot with clocksource=acpi_pm...

On 2.6.28-ish kernels, the lvm volume group fails to activate when I boot with that option. I can boot without that option and if it survives long enough for me to log in and change the clocksource under /sys, then it seems to be reasonably stable as well.

I think that more or less implicates kvm-clock as the source of the problems I was experiencing.
Comment 4 Alexander Boström 2008-12-23 11:23:05 EST
I can confirm this. Host is F10 x86_64 on Athlon X2, guests are F10 i686, only happens with kvm-clock.

It seems to happen under high load in the guest, that might be helpful for reproducing this.
Comment 5 James Ralston 2009-01-03 21:55:24 EST
I'm seeing very similar problems for my new Fedora 10 x86_64 kvm host and guest, using the latest Fedora 10 kernel (2.6.27.9-159) for both the host and the guest.

While I can't get the guest to hang with yum et. al., I've twice now had to hard reset the guest because it hung.  When the guest hangs, the kvm process on the host consumes 100% of the CPU. The guest has yet to achive >12 hours of uptime without hanging.

I suspect this bug is also the source of Fedora 10 kvm guest installer woes; see bug 474116.

I've switched the clocksource of the guest from kvm-clock to acpi_pm (via grub.conf, as the 2.6.27 kernel doesn't seem to have any LVM issues when booting with that clocksource). I'll report back on my experiences.
Comment 6 James Ralston 2009-01-05 12:37:04 EST
My kvm virtual guest now has 36+ hours of uptime with clocksource=acpi_pm, so I'm fairly confident that clocksource=kvm-clock was the problem.

I'd be happy to help test patches...
Comment 7 Alexander Boström 2009-01-11 22:32:49 EST
Another clue: This might be triggered only on hosts with CPU frequency scaling enabled. Can others confirm that or rule that out?

This looks like the same bug:
http://lkml.indiana.edu/hypermail/linux/kernel/0812.2/index.html#02381
Comment 8 Pierre Ossman 2009-01-12 07:46:48 EST
I'm seeing what seems to be this problem, and I'm using frequency scaling.
Comment 9 Chris Lalancette 2009-01-12 07:47:24 EST
*** Bug 479407 has been marked as a duplicate of this bug. ***
Comment 10 James Ralston 2009-01-12 11:19:03 EST
I'm seeing this problem, and I'm using CPU frequency scaling.

When I have a chance, I'll test the combination of (no CPU frequency scaling on the host, clocksource=clock-kvm on the guest) and see if I can reproduce a hang.
Comment 11 Jeff Layton 2009-01-12 11:58:46 EST
I disabled CPU freq scaling on my host (basically, just set the governor to "performance"), and it seems to be a lot more stable with kvm-clock. I'm patching the guest with a big yum update now. In the past, that's generally made it hang after a while, but no lockup so far today.
Comment 12 Gerd Hoffmann 2009-01-20 03:41:33 EST
Can the people seeing the kvm-clock hangs please give some details on the host hardware?  Specifically about the CPU?  Does it happen with UP hosts?  Or SMP hosts only?  Multicore CPUs?  Can you place /proc/cpuinfo content (processor 0 is enougth) here?
thanks.
Comment 13 Gerd Hoffmann 2009-01-20 04:35:17 EST
Oh, and the complete boot messages for the oops log (attachment #1 [details]) would be great too in case it is available ...
Comment 15 Jeff Layton 2009-01-20 06:35:30 EST
(In reply to comment #13)
> Oh, and the complete boot messages for the oops log (attachment #1 [details]) would be
> great too in case it is available ...


Unfortunately, I don't have the boot messages from that particular panic. Reproducing panics with rawhide guest kernels from mid-december timeframe is fairly easy, but as I mentioned before they seem to show up in all sorts of code. It was pure luck that that one showed a panic in kvm-clock. Most of the time they show up elsewhere.
Comment 16 Pierre Ossman 2009-01-20 07:39:38 EST
Created attachment 329453 [details]
Another /proc/cpuinfo

Here's from my single slot dual core system. Also an AMD Athlon system, although with a different CPU model.
Comment 17 Gerd Hoffmann 2009-01-20 10:09:06 EST
Created attachment 329472 [details]
test patch

Ok, seems we are in trouble in case cpus / cores on a SMP machine run on different cpu (and tsc) frequencies, because a single global variable keeps the current tsc frequency, which simply isn't going to work then ...
Comment 18 Jeff Layton 2009-01-20 10:26:59 EST
Thanks Gerd, I'll try to build it and give it a test later today (unless you are planning to build a kernel in koji with this patch soon in which case I'll just wait for that).

Do I need to apply this patch to the host kernel, the guest kernel, or both?
Comment 19 Pierre Ossman 2009-01-20 10:31:45 EST
Hmm... if it uses tsc, does it properly cope with unstable and unsynchronised tsc:s? The host kernel complains about this fact on these CPUs.
Comment 20 Gerd Hoffmann 2009-01-20 11:21:19 EST
Re #18: It is a host kernel patch.
Re #19: In theory it should, in practice it doesn't correctly on some SMP boxes.
Comment 21 Glauber Costa 2009-01-20 11:41:55 EST
 		*lpj = 	cpufreq_scale(loops_per_jiffy_ref, ref_freq, freq->new);
 		tsc_khz = cpufreq_scale(tsc_khz_ref, ref_freq, freq->new);
+		__get_cpu_var(cpu_tsc_khz) = tsc_khz;
 		if (!(freq->flags & CPUFREQ_CONST_LOOPS))
 			mark_tsc_unstable("cpufreq changes");
 	}

there's no guarantee that the handler will run in the cpu that got the frequency change. We have to update it to freq->cpu, not current cpu. Right ?
Comment 22 Chris Lalancette 2009-01-21 07:59:35 EST
*** Bug 474116 has been marked as a duplicate of this bug. ***
Comment 23 Hans de Goede 2009-01-21 10:29:02 EST
Created attachment 329614 [details]
And another cpuinfo

I'm seeing this too, here is my cpuinfo.
Comment 24 David Lutterkort 2009-01-21 16:23:49 EST
Created attachment 329660 [details]
Yet another cpuinfo
Comment 25 Jeff Layton 2009-01-21 21:47:32 EST
I tested the patch in comment #17 but it didn't seem to help. The F10 installer still eventually hung. I redirected the console output to the serial port but there was no oops message and unfortunately I was unable to get any sort of sysrq info.
Comment 26 Jeff Layton 2009-01-21 21:52:40 EST
For the record, I applied the patch to the 2.6.28.1-17 from koji.
Comment 27 Jeff Layton 2009-01-23 07:16:26 EST
I saw some traffic on LKML mentioning that KVM guests are hanging under 2.6.28, so I can't be sure that my testing was actually good. If you can provide a patch that's backported to 2.6.27-ish kernels I'll be happy to test it. This one wouldn't apply when I tried it.
Comment 28 Juan Quintela 2009-01-23 07:23:03 EST
Created attachment 329812 [details]
Patch that makes my Opteron rev F work

This patch works for me.  Basically it saves tsc_khz by cpu instead of using a global variable.  It works with my Dual socket Opteron revF, that should have the same issues that the Athlons that appear in the cpuinfo variables.

Could people give it a try?
Comment 29 James Laska 2009-01-23 08:32:55 EST
Created attachment 329834 [details]
/proc/cpuinfo (Intel(R) Xeon(R) CPU E5420)

> Could people give it a try?

I'm consistently experiencing flakiness while installing rawhide/x86_64 as a kvm guest on my F10/x86_64 dual quad-core system.  I'll be happy to give a kernel a spin.
Comment 30 Mark McLoughlin 2009-01-29 14:38:29 EST
Okay, building a rawhide kernel with a workaround patch to disable kvmclock on hosts which don't have a synchronised TSC

See here:

  http://www.redhat.com/archives/fedora-kernel-list/2009-January/msg00109.html

Should be in kernel-2.6.29-0.66.rc3.fc11

* Thu Jan 29 2009 Mark McLoughlin <markmc@redhat.com>
- Temporarily disable kvmclock, fix for upstream in progress (#475598)

Note: keeping this bug open for F10
Comment 31 Mark McLoughlin 2009-01-30 02:40:31 EST
(In reply to comment #30)
> Okay, building a rawhide kernel with a workaround patch to disable kvmclock on
> hosts which don't have a synchronised TSC

Available here:

  https://koji.fedoraproject.org/koji/buildinfo?buildID=80794
Comment 32 Mark McLoughlin 2009-01-30 02:59:32 EST
(In reply to comment #30)
> Okay, building a rawhide kernel with a workaround patch to disable kvmclock on
> hosts which don't have a synchronised TSC

Looks like there's a 2.6.29-rc3 build with this fix in dist-f10-updates-candidate now:

  https://koji.fedoraproject.org/koji/buildinfo?buildID=80840

  * Thu Jan 29 2009 Chuck Ebbert <cebbert@redhat.com> 2.6.29-0.4.rc3
  - Copy firewire updates, at76 driver, a compile fix and a KVM clock fix from 
    rawhide.
  - Temporarily disable CONFIG_DRM_RADEON_KMS

Not sure what's going on there.
Comment 33 Jeff Layton 2009-02-02 07:16:16 EST
Created attachment 330621 [details]
patch -- unsync tsc workaround patch

It looks like the recent f10 updates candidate kernels contain this patch. It appears to just disable kvm-clock in guests -- not sure, I'm not that familiar with KVM's innards.

This seems like it would be appropriate for f10-ish kernels until this gets straightened out upstream.
Comment 34 Gerd Hoffmann 2009-02-02 10:07:42 EST
Created attachment 330630 [details]
Test patch

Test kernel (still building right now).
http://koji.fedoraproject.org/koji/taskinfo?taskID=1099354
Comment 35 Mark McLoughlin 2009-02-08 04:53:18 EST
This F10 2.6.27 update will contain the workaround:

* Sat Feb 07 2009 Chuck Ebbert <cebbert@redhat.com> 2.6.27.15-170.2.18.rc1
- Copy kvmclock fix from our 2.6.29 kernel.
Comment 36 Alexey Eromenko 2009-02-09 09:01:52 EST
Is there a way to install Fedora 10 in KVM now? (without patching kernels?)

Maybe add some kernel boot parameter via GRUB in guest ?
Comment 37 Jeff Layton 2009-02-09 09:13:22 EST
I was able to install F10 by adding clocksource=acpi_pm to the guest's
kernel command line.

With the kernel mentioned in comment #35 running on the host, you might
also be able to install the guest with no special options. I haven't
tested and install, but I haven't seen any lockups on my existing guests
since booting the dom0 to that kernel.
Comment 38 Mark McLoughlin 2009-02-11 15:05:52 EST
The F10 update is just waiting to be pushed:

  https://admin.fedoraproject.org/updates/kernel-2.6.27.15-170.2.19.fc10

For reference, Avi has queued up the workaround for 2.6.29 here:

  http://git.kernel.org/?p=linux/kernel/git/avi/kvm.git;a=commit;h=f160ca75

and the real fix is queued for 2.6.30 here:

  http://git.kernel.org/?p=linux/kernel/git/avi/kvm.git;a=commit;h=48aa74b8
Comment 39 Levente Farkas 2009-02-12 04:12:56 EST
unfortunately this doesn't seems to solve the problem:-(
while 2.6.27.5-117.fc10.i686 with clocksource=acpi_pm working while 2.6.27.15-170.2.19.fc10.i686 without it also hang randomly.
Comment 40 Mark McLoughlin 2009-02-12 04:23:37 EST
(In reply to comment #39)
> unfortunately this doesn't seems to solve the problem:-(
> while 2.6.27.5-117.fc10.i686 with clocksource=acpi_pm working while
> 2.6.27.15-170.2.19.fc10.i686 without it also hang randomly.

Could you attach the contents of your /proc/cpuinfo ?
Comment 41 Levente Farkas 2009-02-12 04:44:55 EST
Created attachment 331670 [details]
here is my cpuinfo
Comment 42 Mark McLoughlin 2009-02-12 09:45:17 EST
(In reply to comment #39)
> unfortunately this doesn't seems to solve the problem:-(
> while 2.6.27.5-117.fc10.i686 with clocksource=acpi_pm working while
> 2.6.27.15-170.2.19.fc10.i686 without it also hang randomly.

You have constant_tsc, so it sounds like this may not be the same issue.

You're absolutely certain that you are seeing random hangs that are resolved by booting the guest with clocksource=acpi_pm?

(Yes, I know I'm asking you to repeat yourself - sorry - but we need as much further details on the bug you're seeing as possible)
Comment 43 Levente Farkas 2009-02-12 10:31:28 EST
i'm absolutely certain that two guests (i386 and x86_64) working without problem under load with:
----------------------------------
kernel /vmlinuz-2.6.27.5-117.fc10.i686 ro root=/dev/VolGroup00/LogVol00 clocksource=acpi_pm

kernel /vmlinuz-2.6.27.5-117.fc10.x86_64 ro root=/dev/VolGroup00/LogVol00 clocksource=acpi_pm
----------------------------------
and at the same time if i switch any of the guest to the mentioned and downloaded from koji kernel (without: clocksource=acpi_pm)
----------------------------------
kernel /vmlinuz-2.6.27.15-170.2.19.fc10.i686 ro root=/dev/VolGroup00/LogVol00
kernel /vmlinuz-2.6.27.15-170.2.19.fc10.x86_64 ro root=/dev/VolGroup00/LogVol00
----------------------------------
then both guests hang within minutes or many be during boot.
Comment 44 Ian Kent 2009-02-17 21:53:23 EST
Created attachment 332332 [details]
cpuinfo

Linux zeus.themaw.net 2.6.27.12-78.2.8.fc9.i686 #1 SMP Mon Jan 19 20:14:35 EST 2009 i686 i686 i386 GNU/Linux

kvm-65-15.fc9.i386

I also see a hang (and very slow response) when trying to install
Rawhide x86_64.
Comment 45 Mark McLoughlin 2009-02-18 04:46:06 EST
(In reply to comment #44)
> Created an attachment (id=332332) [details]
> cpuinfo
> 
> Linux zeus.themaw.net 2.6.27.12-78.2.8.fc9.i686 #1 SMP Mon Jan 19 20:14:35 EST
> 2009 i686 i686 i386 GNU/Linux
> 
> kvm-65-15.fc9.i386

Okay, this is F-9, which doesn't currently have the workaround patch. But you have constant_tsc anyway, so it shouldn't make a difference.

> I also see a hang (and very slow response) when trying to install
> Rawhide x86_64.

Does this problem go away when booting the guest with clocksource=acpi_pm?
Comment 46 Ian Kent 2009-02-18 07:33:27 EST
(In reply to comment #45)
> (In reply to comment #44)
> > Created an attachment (id=332332) [details] [details]
> > cpuinfo
> > 
> > Linux zeus.themaw.net 2.6.27.12-78.2.8.fc9.i686 #1 SMP Mon Jan 19 20:14:35 EST
> > 2009 i686 i686 i386 GNU/Linux
> > 
> > kvm-65-15.fc9.i386
> 
> Okay, this is F-9, which doesn't currently have the workaround patch. But you
> have constant_tsc anyway, so it shouldn't make a difference.
> 
> > I also see a hang (and very slow response) when trying to install
> > Rawhide x86_64.
> 
> Does this problem go away when booting the guest with clocksource=acpi_pm?

Nope, but I've gone to F-10, x86_64 just now and will see how things go.
Comment 47 Mark McLoughlin 2009-02-20 13:58:28 EST
(In reply to comment #46)
> (In reply to comment #45)

> > Does this problem go away when booting the guest with clocksource=acpi_pm?
> 
> Nope

Okay, if changing the clocksource doesn't fix your problem then it probably isn't kvmclock related.

Please file another bug if the problem persists.
Comment 48 Jeff Garzik 2009-02-21 00:55:01 EST
Maybe we need to re-open BZ# 474116?

I have the following symptoms:

* Fedora 10/x86-64 fails to install as KVM guest.  Hangs at "installing bootloader" step.
* Fedora 10/x86-64 with current updates is the host OS.

This behavior appears with both the Fedora stock kernel or a custom 2.6.29-rc5 kernel running on the host.

I tried "clocksource=acpi_pm" for the Fedora 10 guest, without success.

Hardware:  Intel ICH10

/proc/cpuinfo selected lines:
...
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Genuine Intel(R) CPU             000  @ 3.20GHz
stepping	: 4
...
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
Comment 49 Ian Kent 2009-02-22 21:48:00 EST
(In reply to comment #46)
> (In reply to comment #45)
> > (In reply to comment #44)
> > > Created an attachment (id=332332) [details] [details] [details]
> > > cpuinfo
> > > 
> > > Linux zeus.themaw.net 2.6.27.12-78.2.8.fc9.i686 #1 SMP Mon Jan 19 20:14:35 EST
> > > 2009 i686 i686 i386 GNU/Linux
> > > 
> > > kvm-65-15.fc9.i386
> > 
> > Okay, this is F-9, which doesn't currently have the workaround patch. But you
> > have constant_tsc anyway, so it shouldn't make a difference.
> > 
> > > I also see a hang (and very slow response) when trying to install
> > > Rawhide x86_64.
> > 
> > Does this problem go away when booting the guest with clocksource=acpi_pm?
> 
> Nope, but I've gone to F-10, x86_64 just now and will see how things go.

Host:
2.6.27.15-170.2.24.fc10.x86_64 #1 SMP Wed Feb 11 23:14:31 EST 2009 x86_64 x86_64 x86_64 GNU/Linux

F-10, i386 install hung at installing bootloader.
F-10, x86_64 install hung at Running anaconda.
Comment 50 Mark McLoughlin 2009-03-20 12:15:23 EDT
Okay, this bug covers the case where guests using kvmclock hang on a host machine does not have constant_tsc

This is fixed by disabling kvmclock on host machines which don't have constant_tsc:

  http://git.kernel.org/?p=linux/kernel/git/avi/kvm.git;a=commitdiff;h=f160ca75

This patch has been in F10 since kernel-2.6.27.15-170.2.19.fc10

Closing this bug - any other guest hangs should be filed as separate bugs.

Note You need to log in before you can comment on or make changes to this bug.