Bug 518314 - RHEL5 kernels: performance degradation in relation to increase of CPUs
Summary: RHEL5 kernels: performance degradation in relation to increase of CPUs
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kernel
Version: 5.3
Hardware: i686
OS: Linux
low
medium
Target Milestone: ---
: ---
Assignee: Prarit Bhargava
QA Contact: Red Hat Kernel QE team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2009-08-19 20:10 UTC by Paul Mackiewicz
Modified: 2009-10-07 15:54 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2009-10-07 15:54:53 UTC


Attachments (Terms of Use)

Description Paul Mackiewicz 2009-08-19 20:10:58 UTC
Description of problem: Performance decreases between 40-60% when performing software builds on a 24-core system vs. performing the identical builds on identical hardware with less cores (for example 2-4 cores).  We speculate this to be a CPU scheduler issue.  The current mainline kernel compiled with identical options as Red Hat supplied kernel behaves as expected; the more cores you activate, the better the performance.


Version-Release number of selected component (if applicable):

Verified the following, but suspect all 2.6.18 Red Hat kernels:

2.6.18-128.1.1
2.6.18-128.2.1
2.6.18-128.4.1 
2.6.18-156

How reproducible:  Always.


Steps to Reproduce:
1.  Boot a machine with 4 logical CPUs, perform a build (run make -j4 on new kernel tree as an example, though any CPU intensive task will do if you populate all cores).
2.  Boot the same machine with 24 logical CPUs, perform the same build.  Note the performance degradation (usually 40-60%).
3.  Boot the same machine with 24 logical CPUs, with the latest mainline kernel - 2.6.30.4 - with the same config options as the Red Hat supplied kernels.  Perform the same build.  Note the performance increase (typically 40-60% gain).

Actual results:  Performance degrades as you add logical CPUs.

Example build times on the 8 core machine with all 8 cores active:

make option,real time,kernel
-j4, 4:40, 2.6.30.4
-j4, 4:42, 2.6.30.4
-j4, 4:55, 2.6.30.4
-j8, 4:18, 2.6.30.4
-j8, 4:33, 2.6.30.4

-j4, 7:41, 2.6.18.128
-j4, 7:40, 2.6.18.128
-j4, 7:38, 2.6.18.128
-j8, 7:35, 2.6.18.128
-j8, 7:34, 2.6.18.128

Expected results:  Performance increases as you add logical CPUs.


Additional info:  Machines used for testing: IBM X3850 M2 (24 core [4x6-core CPUS], 64GB memory), Dell 2950 M3 (8 core [2x4-core CPUS], 16GB memory).  Virtual machines on other hardware.

So far only tested on i686.

Comment 2 Prarit Bhargava 2009-08-20 14:07:13 UTC
Paul,

Can you do the following?  

1.  Upgrade your kernel to the latest RHEL5 kernel.  This can be found at

http://people.redhat.com/~dzickus/el5/

2.  Attach an sosreport from your system, and,

3.  There is a good kernel building benchmark utility called kernbench which is available for download at

http://freshmeat.net/projects/kernbench/

Could you run this without any options and send us the output?

Thanks,

P.

Comment 3 Prarit Bhargava 2009-08-26 18:22:00 UTC
Paul,

Your "cpu intensive" task, is it memory intensive too?

P.


Note You need to log in before you can comment on or make changes to this bug.