Description of problem: Performance decreases between 40-60% when performing software builds on a 24-core system vs. performing the identical builds on identical hardware with less cores (for example 2-4 cores).  We speculate this to be a CPU scheduler issue.  The current mainline kernel compiled with identical options as Red Hat supplied kernel behaves as expected; the more cores you activate, the better the performance.


Version-Release number of selected component (if applicable):

Verified the following, but suspect all 2.6.18 Red Hat kernels:

2.6.18-128.1.1
2.6.18-128.2.1
2.6.18-128.4.1 
2.6.18-156

How reproducible:  Always.


Steps to Reproduce:
1.  Boot a machine with 4 logical CPUs, perform a build (run make -j4 on new kernel tree as an example, though any CPU intensive task will do if you populate all cores).
2.  Boot the same machine with 24 logical CPUs, perform the same build.  Note the performance degradation (usually 40-60%).
3.  Boot the same machine with 24 logical CPUs, with the latest mainline kernel - 2.6.30.4 - with the same config options as the Red Hat supplied kernels.  Perform the same build.  Note the performance increase (typically 40-60% gain).

Actual results:  Performance degrades as you add logical CPUs.

Example build times on the 8 core machine with all 8 cores active:

make option,real time,kernel
-j4, 4:40, 2.6.30.4
-j4, 4:42, 2.6.30.4
-j4, 4:55, 2.6.30.4
-j8, 4:18, 2.6.30.4
-j8, 4:33, 2.6.30.4

-j4, 7:41, 2.6.18.128
-j4, 7:40, 2.6.18.128
-j4, 7:38, 2.6.18.128
-j8, 7:35, 2.6.18.128
-j8, 7:34, 2.6.18.128

Expected results:  Performance increases as you add logical CPUs.


Additional info:  Machines used for testing: IBM X3850 M2 (24 core [4x6-core CPUS], 64GB memory), Dell 2950 M3 (8 core [2x4-core CPUS], 16GB memory).  Virtual machines on other hardware.

So far only tested on i686.