Bug 736499

Summary: kvm guests kills host IO performance
Product: Red Hat Enterprise Linux 6 Reporter: Orion Poplawski <orion>
Component: kernelAssignee: Red Hat Kernel Manager <kernel-mgr>
Status: CLOSED CURRENTRELEASE QA Contact: Red Hat Kernel QE team <kernel-qe>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.1CC: nerijus
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-06-05 14:15:29 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Orion Poplawski 2011-09-07 20:44:07 UTC
Description of problem:

I've had to disable raid-check because the check/resync process absolutely kills performance.  Server is used for backups and for multiple kvm guests so a fair amount of IO.

Version-Release number of selected component (if applicable):
2.6.32-131.12.1.el6.x86_64
mdadm-3.2.2-1.el6.2.x86_64

Comment 1 Orion Poplawski 2011-09-07 20:44:39 UTC
# cat /proc/mdstat
Personalities : [raid1] [raid10] 
md1 : active raid10 sdb2[0] sde1[7] sdf1[6] sdg1[5] sdh1[4] sdc1[3] sdd1[2] sda2[1]
      3906203648 blocks 256K chunks 2 near-copies [8/8] [UUUUUUUU]
      
md0 : active raid1 sdb1[0] sda1[1]
      200704 blocks [2/2] [UU]

Comment 3 RHEL Program Management 2011-10-07 15:47:43 UTC
Since RHEL 6.2 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as
exception or blocker.

Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.

Comment 4 Orion Poplawski 2011-10-20 19:10:15 UTC
It seems to be the combination of kvm guests and the raid10 resync that kills IO performance for some reason.  Stopping all kvm guests during the resync (or stopping the resync if it is just a check) gets me back to >100MB/s read/write performance.

The kvm guests are using raw lvm volumes on a volume group on the raid10 array and using virtio.  One kvm guest in particular that triggers it is a 2 cpu Fedora 14 instance running zabbix-server - so a fairly steady cpu and io load.

2.6.32-131.17.1.el6.x86_64

Comment 5 Orion Poplawski 2011-11-18 18:20:27 UTC
I just noticed that a copy between two disks on the host was crawling (5-10MB/s).  I starting shutting down kvm guests (which were mainly idle) and now I'm up to 50-70MB/s.  Something about kvm guests is really killing the host IO performance.

Any chance that having hyperthreading enabled would have any effect on this?  I may disable just to see.

Comment 6 Orion Poplawski 2011-11-18 18:26:55 UTC
This is the controller I'm using:

07:00.0 SCSI storage controller: LSI Logic / Symbios Logic MegaRAID SAS 8208ELP/8208ELP (rev 08)

with mptsas driver.

Comment 7 Orion Poplawski 2012-02-17 06:05:04 UTC
This appears to be fixed in recent kernels.  Feel free to close.

Comment 8 Nerijus Baliƫnas 2012-04-05 01:21:45 UTC
Still happens on F16 latest kernel BTW.

Comment 9 Orion Poplawski 2012-04-05 03:45:54 UTC
Is this a regular F16 system?  If so I would open a new bug against F16.  I'm not seeing any trouble now with a current 6.2 system.