Bug 584310 - non-smp guests become unresponsive and use 100% cpu with clock source kvm-clock
non-smp guests become unresponsive and use 100% cpu with clock source kvm-clock
Status: CLOSED DUPLICATE of bug 570824
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kvm (Show other bugs)
5.4
All Linux
low Severity high
: rc
: ---
Assigned To: Glauber Costa
Virtualization Bugs
:
Depends On:
Blocks: Rhel5KvmTier1
  Show dependency treegraph
 
Reported: 2010-04-21 06:52 EDT by Need Real Name
Modified: 2010-11-09 08:14 EST (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-06-28 09:07:43 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
strace -s 250 -f -p MADGUESTPID (19.40 KB, text/plain)
2010-04-21 09:34 EDT, Need Real Name
no flags Details
kvmtrace -o blah -w 5 (blah.kvmtrace.0) (3.33 MB, application/octet-stream)
2010-04-21 09:40 EDT, Need Real Name
no flags Details
kvmtrace -o blah -w 5 (blah.kvmtrace.1) (337.30 KB, application/octet-stream)
2010-04-21 09:40 EDT, Need Real Name
no flags Details
kvmtrace -o blah -w 5 (blah.kvmtrace.2) (2.68 MB, application/octet-stream)
2010-04-21 09:42 EDT, Need Real Name
no flags Details
kvmtrace -o blah -w 5 (blah.kvmtrace.3) (794.70 KB, application/octet-stream)
2010-04-21 09:43 EDT, Need Real Name
no flags Details
/proc/cpuinfo on host (2.73 KB, text/plain)
2010-06-23 14:54 EDT, Need Real Name
no flags Details

  None (edit)
Description Need Real Name 2010-04-21 06:52:31 EDT
This is actually occuring under centos 5.4 x86_64 (fully updated) with centos 5.4 32-bit guests. Reporting here since you are upstream.

I have five guests running on this machine. At least once a day one or more (normally more) guests will hit 100% cpu and become unresponsive.

Setting kernel.panic=10 in the guest does not reboot the guest. The only solution is to destroy the guest and start it again.

# rpm -qa|grep kvm
kmod-kvm-83-105.el5_4.28
etherboot-zroms-kvm-5.4.4-10.el5.centos
kvm-tools-83-105.el5_4.28
kvm-qemu-img-83-105.el5_4.28
kvm-83-105.el5_4.28

kernel 2.6.18-164.15.1.el5

The guests were using the virtio_blk disk device vda but I have switched them back to hda. The network cards use virtio. The disks are qcow2. All other settings are standard.
Comment 1 Need Real Name 2010-04-21 09:34:20 EDT
Created attachment 408066 [details]
strace -s 250 -f -p MADGUESTPID

Maybe this strace will help.
Comment 2 Need Real Name 2010-04-21 09:40:40 EDT
Created attachment 408069 [details]
kvmtrace -o blah -w 5 (blah.kvmtrace.0)
Comment 3 Need Real Name 2010-04-21 09:40:52 EDT
Created attachment 408070 [details]
kvmtrace -o blah -w 5 (blah.kvmtrace.1)
Comment 4 Need Real Name 2010-04-21 09:42:20 EDT
Created attachment 408071 [details]
kvmtrace -o blah -w 5 (blah.kvmtrace.2)
Comment 5 Need Real Name 2010-04-21 09:43:27 EDT
Created attachment 408072 [details]
kvmtrace -o blah -w 5 (blah.kvmtrace.3)
Comment 6 Need Real Name 2010-04-21 09:44:35 EDT
Trace files from kvmtrace and strace attached.

Please could you mark these files as sensitive/confidential.
Comment 7 Need Real Name 2010-04-21 10:16:27 EDT
Setting severity to high since this is a hard crash.
Comment 8 Need Real Name 2010-04-21 10:34:16 EDT
Using hangcheck has no effect at all. It never fires.
Using kernel.panic=10 doesn't help either.
Nothing in the hosts logs. No console messages in the guest.
Comment 9 Need Real Name 2010-04-21 14:44:43 EDT
To get this into a "supported" state, so that you can worry about it (!), I converted the three least busy machines to
i) not use virtio_blk
ii) not use the virtio network device
ii) not use the qcow2 disk format

One of these newly supported machines just did it again: 100% cpu and not responding.

What can I try please?
Comment 10 Amit Shah 2010-04-21 15:07:03 EDT
(In reply to comment #1)
> Created an attachment (id=408066) [details]
> strace -s 250 -f -p MADGUESTPID

Could be a time drift issue. Do you see clock skews in the guests? Are you using kvmclock in the guest?
Comment 11 Need Real Name 2010-04-21 15:23:20 EDT
I have noticed a few seconds difference. I am using kvm-clock:

# cat /sys/devices/system/clocksource/clocksource0/current_clocksource 
kvm-clock 

The guests with problems are almost idle.

Time difference is about 1 second: the guests are behind by a second.

Do you recommend installing ntpd in the guests?
Comment 12 Need Real Name 2010-04-21 16:23:50 EDT
I installed ntpd in the guests along with step tickers. I just had the guest crash again. Same strace. ARGH! :(
Comment 13 Need Real Name 2010-04-22 05:15:05 EDT
Three crashes so far today. If I can put kvm into debug mode and you want me to post that, please let me know.
Comment 14 Need Real Name 2010-04-22 15:38:21 EDT
I switched from clocksource=kvm-clock to clocksource=acpi_pm, keeping ntpd and it seems a lot more stable. No crashes since the change.

I will watch it for another 24 hours with ntpd now off to see if it crashes again and if the time drifts.

Perhaps important is that the machines which are loaded never or almost never crash.
Comment 15 Need Real Name 2010-04-23 10:35:19 EDT
24 hours is up: not one single extra crash.

I've changed all guests away from using kvm-clock. Is this a know problem with 5.4?
Comment 16 Need Real Name 2010-04-25 10:20:59 EDT
Still no more crashes!

Interesting is how the cpu flags on the host and guest compare:

HOST: tsc constant_tsc nonstop_tsc
GUEST: tsc

i.e. no constant_tsc, which means it might be linked to bug 475598 - but it's not clear to me in that bug if there is a missing constant_tsc in the guests or on the host.
Comment 17 XinSun 2010-05-31 23:49:37 EDT
I meet this problem on rhel5u5 (32-bit server) guest, I use rhevm(sm70) to crate these guests (with virtio disk driver and virtio network driver) on rhev-hypervisor-5.5-2.2.1

These rhel5u5 (32-bit server) guests will be hang after some running time.
Comment 18 Glauber Costa 2010-06-01 08:07:13 EDT
Ok, In fact there is a know kvmclock problem with rhel5.5

The know bug, so far, is known to bite SMP. Are you doing SMP in the guest? 

For the record, this is the bug:
https://bugzilla.redhat.com/show_bug.cgi?id=570824

RPMs for it are likely to arrive soon.
Comment 19 philippe.plouffe 2010-06-01 09:53:41 EDT
(In reply to comment #18)
> Ok, In fact there is a know kvmclock problem with rhel5.5
> 
> The know bug, so far, is known to bite SMP. Are you doing SMP in the guest? 
> 
> For the record, this is the bug:
> https://bugzilla.redhat.com/show_bug.cgi?id=570824
> 
> RPMs for it are likely to arrive soon.    

I'm having the same issue (5.5 32 bit guests) and yes they all have multiple CPU assigned.
Comment 20 Glauber Costa 2010-06-01 11:43:19 EDT
Please do test single-cpu guest to make sure it does go away.
Comment 21 Need Real Name 2010-06-01 12:20:25 EDT
I was seeing this on single cpu guests.
Comment 22 Glauber Costa 2010-06-07 10:26:35 EDT
Note added to all kvmclock bugs:

Please retest with kernel-2.6.18-202.el5 (RHEL5) or kernel-2.6.32-33.el6 (RHEL6) in your guest kernel. In case it works, please close as a DUP of bugs 570824 (RHEL5) or 569603 (RHEL6)
Comment 23 Glauber Costa 2010-06-22 09:25:05 EDT
Did any of you re-tested this ?
Comment 24 philippe.plouffe 2010-06-22 10:01:01 EDT
kernel 202 is not available yet. I don't find a place to get the kernel-2.6.18-202.el5 you are refering too. The closest thing I found looking at referencing bugs id was version 203 found at http://people.redhat.com/jwilson/el5 ( url comes from https://bugzilla.redhat.com/show_bug.cgi?id=570824 )

Where can I get it ?
Comment 25 Glauber Costa 2010-06-22 15:24:45 EDT
203 will do.
Comment 26 Zachary Amsden 2010-06-22 16:59:30 EDT
This sounds exactly like the bug I was hitting running RHEL6 kvmclock guests.

The problem only happens for SMP guests, as described here, and resulted in hangs.  Switching away from kvmclock or switching to UP guest fixed the problem.

Glauber's patches to the guest kernel to make the kvmclock not go backwards fixed the problem.  I'm highly suspicious this is a dup.
Comment 27 Need Real Name 2010-06-22 17:12:53 EDT
> as described here

Please could everyone stop stealing my bug.

This bug is for single cpu guests.
Comment 28 Glauber Costa 2010-06-23 09:23:33 EDT
Ok, let's start from the supposition this is not the same bug.

Would be good to give the new kernel a test anyway, so we can start from a
fresher base. Many kvmclock patches went in, so maybe your problem is fixed by it.

In case it is not, please report, together with the info present in your /proc/cpuinfo (host)
Comment 29 Need Real Name 2010-06-23 14:54:33 EDT
Created attachment 426358 [details]
/proc/cpuinfo on host
Comment 30 Need Real Name 2010-06-23 14:56:45 EDT
kernel-2.6.18-203.el5.i686.rpm installed in guest and rebooted.
Comment 31 Need Real Name 2010-06-25 02:40:51 EDT
So far, no 100% craziness.
Comment 32 Need Real Name 2010-06-26 11:10:36 EDT
Still working well. It's never lasted anywhere near this long before. Timekeeping is also fine.

Do you want me to do anything else?
Comment 33 Glauber Costa 2010-06-28 09:06:57 EDT
if it is working for you, I'll close it as a dup.
Comment 34 Glauber Costa 2010-06-28 09:07:43 EDT

*** This bug has been marked as a duplicate of bug 570824 ***

Note You need to log in before you can comment on or make changes to this bug.