Bug 858396 - latency issues on four-socket systems
Summary: latency issues on four-socket systems
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: realtime-kernel
Version: 2.2
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: 2.3.1
: ---
Assignee: John Kacur
QA Contact: David Sommerseth
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-09-18 21:07 UTC by Clark Williams
Modified: 2016-05-22 23:35 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: high contention on run-queue lock when load balancing before idle Consequence: latency spikes on high-core count systems Fix: use IPI to send notification to cores with pending work and have the core push the work rather than trying to pull it Result: reduced contention during load balance and no latency spikes
Clone Of:
Environment:
Last Closed: 2013-03-11 18:39:36 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Use IPIs to trigger pushing rt tasks instead of pulling them (7.35 KB, patch)
2013-02-11 19:50 UTC, Steven Rostedt
no flags Details | Diff


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2013:0622 0 normal SHIPPED_LIVE Important: kernel-rt security and bug fix update 2013-03-11 22:36:36 UTC

Comment 1 Steven Rostedt 2013-02-11 19:50:10 UTC
Created attachment 696258 [details]
Use IPIs to trigger pushing rt tasks instead of pulling them

I sent this patch to mainline for review, and although Thomas pretty much NAK'd it, we do not have any other option that can deal with boxes with a large number of CPUs.

Mike Galbraith tested this patch and it showed a HUGE impact on his box. His max latencies on his box went from 600 to 1000 microseconds down to just 50 (most were under 20)!  That makes it pretty darn convincing to apply the patch.

The description of the patch is attached to the patch itself. I will add here that the patch creates a sched_feature called RT_PUSH_IPI. By default, if the box has 16 or less CPUs, it continues to do the pull_rt_task. A box with 17 or more CPUs (and any box with > 16 CPUS like those with 16.5 CPUS), will default with the IPI to push RT tasks.

You can force to either use the feature or not via the debugfs directory:

to enable:
 echo RT_PUSH_IPI > /sys/kernel/debug/sched_features

to disable:
 echo NO_RT_PUSH_IPI > /sys/kernel/debug/sched_features

Comment 7 errata-xmlrpc 2013-03-11 18:39:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0622.html


Note You need to log in before you can comment on or make changes to this bug.