Bug 858396 - latency issues on four-socket systems
latency issues on four-socket systems
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: realtime-kernel (Show other bugs)
x86_64 Linux
unspecified Severity medium
: 2.3.1
: ---
Assigned To: John Kacur
David Sommerseth
Depends On:
  Show dependency treegraph
Reported: 2012-09-18 17:07 EDT by Clark Williams
Modified: 2016-05-22 19:35 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: high contention on run-queue lock when load balancing before idle Consequence: latency spikes on high-core count systems Fix: use IPI to send notification to cores with pending work and have the core push the work rather than trying to pull it Result: reduced contention during load balance and no latency spikes
Story Points: ---
Clone Of:
Last Closed: 2013-03-11 14:39:36 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
Use IPIs to trigger pushing rt tasks instead of pulling them (7.35 KB, patch)
2013-02-11 14:50 EST, Steven Rostedt
no flags Details | Diff

  None (edit)
Comment 1 Steven Rostedt 2013-02-11 14:50:10 EST
Created attachment 696258 [details]
Use IPIs to trigger pushing rt tasks instead of pulling them

I sent this patch to mainline for review, and although Thomas pretty much NAK'd it, we do not have any other option that can deal with boxes with a large number of CPUs.

Mike Galbraith tested this patch and it showed a HUGE impact on his box. His max latencies on his box went from 600 to 1000 microseconds down to just 50 (most were under 20)!  That makes it pretty darn convincing to apply the patch.

The description of the patch is attached to the patch itself. I will add here that the patch creates a sched_feature called RT_PUSH_IPI. By default, if the box has 16 or less CPUs, it continues to do the pull_rt_task. A box with 17 or more CPUs (and any box with > 16 CPUS like those with 16.5 CPUS), will default with the IPI to push RT tasks.

You can force to either use the feature or not via the debugfs directory:

to enable:
 echo RT_PUSH_IPI > /sys/kernel/debug/sched_features

to disable:
 echo NO_RT_PUSH_IPI > /sys/kernel/debug/sched_features
Comment 7 errata-xmlrpc 2013-03-11 14:39:36 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.