Bug 456911 - RHEL4 scheduler optimizations for financial applications
Summary: RHEL4 scheduler optimizations for financial applications
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: kernel
Version: 4.8
Hardware: All
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: Larry Woodman
QA Contact: Red Hat Kernel QE team
URL:
Whiteboard:
Keywords:
Depends On:
Blocks: RHEL4u8_relnotes 461304
TreeView+ depends on / blocked
 
Reported: 2008-07-28 15:49 UTC by Goutham Kandiar
Modified: 2018-11-14 18:01 UTC (History)
11 users (show)

(edit)
A new allowable value has been added to the /proc/sys/kernel/wake_balance tunable parameter. Setting wake_balance to a value of 2 will instruct the scheduler to run the thread on any available CPU rather than scheduling it on the optimal CPU. Setting this kernel parameter to 2 will force the scheduler to reduce the overall latency even at the cost of total system throughput.
Clone Of:
(edit)
Last Closed: 2009-05-18 19:32:07 UTC


Attachments (Terms of Use)
Wombat Performance w/ 2-1Gbit nics on Intel quad-core (28.92 KB, application/octet-stream)
2008-08-04 20:57 UTC, John Shakshober
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2009:1024 normal SHIPPED_LIVE Important: Red Hat Enterprise Linux 4.8 kernel security and bug fix update 2009-05-18 14:57:26 UTC

Description Goutham Kandiar 2008-07-28 15:49:09 UTC
Description of problem:

A large financial customer is seeing poor performance from the RHEL4 scheduler
in terms of sub millisecond CPU response.  

The application is a high frequency financial trading application.

The customer kernel developer changed the code to allocate the job to free CPU's
and managed to reduce latency in their application from milliseconds  to
microseconds. 

Red Hat has identified the following steps:
 
1) Research kernel patches for scheduler optimization from customer and share
with Red Hat kernel team for review

2) Test kernel patches against real production applications and assess potential
benefits and/or drawbacks, including assessment of potential performance regressions

3) Leverage scheduler optimizations in future RHEL updates/versions as appropriate

4) Help push Red Hat-reviewed code changes upstream for Fedora 10/11 and Red Hat
Enterprise Linux 6

The customer has supplied Red Hat with their patch.

Comment 1 John Shakshober 2008-08-04 17:57:48 UTC
Attached is improved performance by Tom Tracy with Wombat, on a kernel from Larry Woodman.

Completed testing the scheduler patch using Wombat. Throughput increased from 44K to 74K. Latency is comparable to RHEL5.2 I have attached results showing the throughput and latency comparisons with the scheduler patch.

Comment 2 John Shakshober 2008-08-04 20:57:20 UTC
Created attachment 313402 [details]
Wombat Performance w/ 2-1Gbit nics on Intel quad-core

Comment 3 Chris Snook 2008-08-11 21:37:31 UTC
Looking at the patch that was posted to the list, we may want to differentiate between static (realtime) priority threads and dynamic priority threads for the default behavior of the migrate-on-clone code.  For realtime threads, it's safe to assume that latency is top priority, so we should probably enable it by default there.  For dynamic priority threads, there will often be a throughput benefit to avoiding the migration, due to cache effects, as well as a power saving benefit to keeping cores idle longer, so it should be disabled by default there.  Since we're going to have a sysctl tunable anyway, we might as well default to the settings that will make the most people happy.

Comment 4 RHEL Product and Program Management 2008-09-03 13:09:03 UTC
Updating PM score.

Comment 5 RHEL Product and Program Management 2008-12-17 20:19:53 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.

Comment 6 Vivek Goyal 2009-01-12 14:50:24 UTC
Committed in 78.27.EL . RPMS are available at http://people.redhat.com/vgoyal/rhel4/

Comment 7 Larry Woodman 2009-01-23 19:10:48 UTC
Release note added. If any revisions are required, please set the 
"requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.

New Contents:
A new allowable value was added to the /proc/sys/kernel/wake_balance tunable parameter.  
Setting wake_balance  to 2 will cause the scheduler to run the thread being awakened on any avaialble CPU rather than scheduling it on the optimal CPU based on a combination of cache footprint and idleness of the CPU in question.  This will cause the scheduler to reduce the overall latencey even at the cost of total system throughput.
 
Large financial applications that experience poor latencey performance from the RHEL4 scheduler
and would like to see sub millisecond CPU response  times should set /proc/sys/kernel/wake_balance = 2.

Comment 11 Ryan Lerch 2009-02-25 06:55:17 UTC
Release note updated. If any revisions are required, please set the 
"requires_release_notes"  flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.

Diffed Contents:
@@ -1,5 +1 @@
-A new allowable value was added to the /proc/sys/kernel/wake_balance tunable parameter.  
+A new allowable value has been added to the /proc/sys/kernel/wake_balance tunable parameter. Setting wake_balance to a value of 2 will instruct the scheduler to run the thread on any available CPU rather than scheduling it on the optimal CPU. Setting this kernel parameter to 2 will force the scheduler to reduce the overall latency even at the cost of total system throughput.-Setting wake_balance  to 2 will cause the scheduler to run the thread being awakened on any avaialble CPU rather than scheduling it on the optimal CPU based on a combination of cache footprint and idleness of the CPU in question.  This will cause the scheduler to reduce the overall latencey even at the cost of total system throughput.
- 
-Large financial applications that experience poor latencey performance from the RHEL4 scheduler
-and would like to see sub millisecond CPU response  times should set /proc/sys/kernel/wake_balance = 2.

Comment 14 Chris Ward 2009-05-05 13:57:23 UTC
Any updates here? Has this issue been resolved in the RHEL 4.8 Beta? later kernel?

Comment 15 John Shakshober 2009-05-05 14:08:29 UTC
We have tested this for performance in 4.8 and ack this for 4.8.
/proc/sys/kernel/wake_balance = 2.

Comment 16 Larry Woodman 2009-05-05 14:11:29 UTC
Chris, this change is in RHEL4-U8.  You enable it by setting proc/sys/kernel/wake_balance = 2.

Larry Woodman

Comment 17 Chris Ward 2009-05-05 14:22:08 UTC
Sorry for the confusion. I meant to ask whether this issue had been tested by QA, customer or partner and if so, whether or not it has been VERIFIED.

Comment 19 errata-xmlrpc 2009-05-18 19:32:07 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2009-1024.html


Note You need to log in before you can comment on or make changes to this bug.