Bug 736499 - kvm guests kills host IO performance
Summary: kvm guests kills host IO performance
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kernel
Version: 6.1
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Red Hat Kernel Manager
QA Contact: Red Hat Kernel QE team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-09-07 20:44 UTC by Orion Poplawski
Modified: 2012-06-05 14:15 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-06-05 14:15:29 UTC


Attachments (Terms of Use)

Description Orion Poplawski 2011-09-07 20:44:07 UTC
Description of problem:

I've had to disable raid-check because the check/resync process absolutely kills performance.  Server is used for backups and for multiple kvm guests so a fair amount of IO.

Version-Release number of selected component (if applicable):
2.6.32-131.12.1.el6.x86_64
mdadm-3.2.2-1.el6.2.x86_64

Comment 1 Orion Poplawski 2011-09-07 20:44:39 UTC
# cat /proc/mdstat
Personalities : [raid1] [raid10] 
md1 : active raid10 sdb2[0] sde1[7] sdf1[6] sdg1[5] sdh1[4] sdc1[3] sdd1[2] sda2[1]
      3906203648 blocks 256K chunks 2 near-copies [8/8] [UUUUUUUU]
      
md0 : active raid1 sdb1[0] sda1[1]
      200704 blocks [2/2] [UU]

Comment 3 RHEL Product and Program Management 2011-10-07 15:47:43 UTC
Since RHEL 6.2 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as
exception or blocker.

Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.

Comment 4 Orion Poplawski 2011-10-20 19:10:15 UTC
It seems to be the combination of kvm guests and the raid10 resync that kills IO performance for some reason.  Stopping all kvm guests during the resync (or stopping the resync if it is just a check) gets me back to >100MB/s read/write performance.

The kvm guests are using raw lvm volumes on a volume group on the raid10 array and using virtio.  One kvm guest in particular that triggers it is a 2 cpu Fedora 14 instance running zabbix-server - so a fairly steady cpu and io load.

2.6.32-131.17.1.el6.x86_64

Comment 5 Orion Poplawski 2011-11-18 18:20:27 UTC
I just noticed that a copy between two disks on the host was crawling (5-10MB/s).  I starting shutting down kvm guests (which were mainly idle) and now I'm up to 50-70MB/s.  Something about kvm guests is really killing the host IO performance.

Any chance that having hyperthreading enabled would have any effect on this?  I may disable just to see.

Comment 6 Orion Poplawski 2011-11-18 18:26:55 UTC
This is the controller I'm using:

07:00.0 SCSI storage controller: LSI Logic / Symbios Logic MegaRAID SAS 8208ELP/8208ELP (rev 08)

with mptsas driver.

Comment 7 Orion Poplawski 2012-02-17 06:05:04 UTC
This appears to be fixed in recent kernels.  Feel free to close.

Comment 8 Nerijus Baliūnas 2012-04-05 01:21:45 UTC
Still happens on F16 latest kernel BTW.

Comment 9 Orion Poplawski 2012-04-05 03:45:54 UTC
Is this a regular F16 system?  If so I would open a new bug against F16.  I'm not seeing any trouble now with a current 6.2 system.


Note You need to log in before you can comment on or make changes to this bug.