Bug 1034171 - Responsiveness has become worse since about beginning of the year
Summary: Responsiveness has become worse since about beginning of the year
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm
Version: 6.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Radim Krčmář
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-11-25 11:47 UTC by Albert Flügel
Modified: 2015-03-13 17:42 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-03-13 17:42:08 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Albert Flügel 2013-11-25 11:47:59 UTC
Description of problem:
I have 6 virtual instances of different Redhat OSes on a Sun X2200 with
KVM and QEMU. Until the last updates everything was really nice. Now
The virtual machines are hanging for seconds several times every minute.
I'm on the virtual machines via ssh and every now and then (let's say
2 - 4 times a minute) it hangs for 1 .. 5 seconds. pinging the ip
address of the virtual machines shows no replies during these hangs either,
so it seems, the entire instance is hanging. When stracing the qemu-kvm
process it's busy all the time, so it's not the underlaying OS hanging.
Furthermore it does not happen with all 6 virtual machines at the same time.
The host system is a Redhat-6 with
kernel-2.6.32-358.18.1.el6.x86_64
qemu-kvm-0.12.1.2-2.415.el6_5.3.x86_64

The virtual instances are:
CentOS-6.4
CentOS-5.9
Fedora-17 latest available patch level
CentOS-4.8
Redhat-EL 5.6 (2 instances)

All these are basiclly idling, in local top programs i see 99% idle
or more. On the host system i see typically:
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
 6334 qemu      20   0 16.6g  10g 5552 S 112.6 17.2  23:49.99 qemu-kvm          
 6869 qemu      20   0 16.6g 1.3g 5096 S 99.7  2.1  21:24.13 qemu-kvm           
 6595 qemu      20   0 16.6g 4.1g 5552 S 37.2  6.5  21:07.94 qemu-kvm           
 5661 qemu      20   0 39.9g 1.6g 5356 R 34.5  2.5  15:09.04 qemu-kvm           
 5264 qemu      20   0 30.1g 1.3g 5356 R 20.3  2.1   9:11.64 qemu-kvm           
 5987 qemu      20   0 10.6g 1.4g 5840 S  1.3  2.3   1:10.55 qemu-kvm           

though none of them is actually doing anything. The lowest one is the
Fedora instance. It seems to idle "most efficiently".

Version-Release number of selected component (if applicable):
kernel-2.6.32-358.18.1.el6.x86_64
qemu-kvm-0.12.1.2-2.415.el6_5.3.x86_64

How reproducible:
Configure KVM, with official network interfaces, i.e. no NAT, no local
DHCP and DNS, no spanning tree, just a virtual bridge connecting the
eth0 with the virtual interfaces. Run several virtual instances and
use them

Steps to Reproduce:
1. See above
2. Login to one of the virtual instances
3. Do something and may ping the instance at the same time from another host
4. Probably watch the CPU utilization on the hosting system

Actual results:
Connection or the entire virtual instance hangs for several seconds (typically
1 .. 5)

Expected results:
Nearly constant responsiveness. Ok it may vary due to real load on the host system or on the virtual instance, but not that hard. Without any load i'd want to expect nearly constant responsiveness.

Additional info:
It did not happen with
kernel-2.6.32-279.19.1.el6.x86_64
qemu-kvm-0.12.1.2-2.209.el6_2.4

Comment 1 Albert Flügel 2013-11-25 11:49:50 UTC
Additional Info: On a different host system (Also RHEL6 in same patch level) i have other virtual instances running and there it's just the same. It was all fine some months ago but now i see these pauses.

Comment 3 Albert Flügel 2013-11-27 09:01:31 UTC
In the meantime i played a bit with that and found:
Reducing the number of CPUs of the virtual CentOS-4 instance to 2 changed the situation significantly. Overview:

Host system: Sun X2200 with 4 CPU cores

6 Virtual systems:
# OS         #CPUs
1 CentOS-6.4 2
1 CentOS-5.9 2
1 Fedora-17  2
2 RHEL-5.6   4
1 CentOS-4.8 2 (now, had 4 before)

Now when none of the virtual hosts has notable load (what is true most of the time), any single node is very responsive. If any of the systems has a bit more load, e.g. one virtual CPU is 100% busy, the situation is like before: All of them are really slow and hardly usable.

Unlike a year ago, there seems to be a quite sharp line in the total load. If the load is higher than this limit, all virtual systems show very bad responsiveness.

I'd be interested in options to configure the scheduler that is managing the qemu processes, if there are any.

Comment 4 Ademar Reis 2013-11-28 17:04:19 UTC
Albert, thanks for taking the time to enter a bug report with us. We appreciate
the feedback and look to use reports such as this to guide our efforts at
improving our products. That being said, we're not able to guarantee the
timeliness or suitability of a resolution for issues entered here because this
is not a mechanism for requesting support.

If this issue is critical or in any way time sensitive, please raise a ticket
through your regular Red Hat support channels to make certain  it receives the
proper attention and prioritization to assure a timely resolution.

For information on how to contact the Red Hat production support team, please
visit: https://www.redhat.com/support/process/production/#howto

Comment 7 Radim Krčmář 2015-01-29 21:13:21 UTC
Albert, are you still experiencing these hangs?

Did old kernel work well with newer qemu-kvm?
(kernel-2.6.32-279.19.1.el6.x86_64 and qemu-kvm-0.12.1.2-2.415.el6_5.3.x86_64)

Thank you.

Comment 8 Ademar Reis 2015-03-12 20:15:13 UTC
(In reply to Radim Krčmář from comment #7)
> Albert, are you still experiencing these hangs?
> 
> Did old kernel work well with newer qemu-kvm?
> (kernel-2.6.32-279.19.1.el6.x86_64 and
> qemu-kvm-0.12.1.2-2.415.el6_5.3.x86_64)

Setting needinfo.

Comment 9 Albert Flügel 2015-03-13 17:19:51 UTC
Thanks for the inquiry ! Unfortunately
i have currently no possibility to test this.
So please feel free to close this bug for now.
As soon as i have the chance to check that again, i can reopen, if necessary.
Is that ok for you ?

Comment 10 Ademar Reis 2015-03-13 17:42:08 UTC
(In reply to Albert Flügel from comment #9)
> Thanks for the inquiry ! Unfortunately
> i have currently no possibility to test this.
> So please feel free to close this bug for now.
> As soon as i have the chance to check that again, i can reopen, if necessary.
> Is that ok for you ?

OK, thanks.


Note You need to log in before you can comment on or make changes to this bug.