This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1247184 - scheduler plugin causes nohz_full to be de-activated
scheduler plugin causes nohz_full to be de-activated
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: tuned (Show other bugs)
7.2
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Jaroslav Škarvada
Tereza Cerna
:
Depends On:
Blocks: kvm-rt-tuned
  Show dependency treegraph
 
Reported: 2015-07-27 10:03 EDT by Luiz Capitulino
Modified: 2015-11-19 07:21 EST (History)
5 users (show)

See Also:
Fixed In Version: tuned-2.5.1-3.el7
Doc Type: Bug Fix
Doc Text:
Cause: Previously the new perf code for runtime tuning functionality of plugin_scheduler could cause nohz_full not to work correctly. Consequence: This may degrade performance for real-time workloads or HPC workloads. Fix: The perf code was improved to work correctly with nohz_full. Result: Now the runtime functionality of scheduler_plugin works correctly with nohz_full.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-11-19 07:21:59 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Luiz Capitulino 2015-07-27 10:03:10 EDT
Description of problem:

During KVM-RT testing, we've found that tuned's scheduler plugin causes nohz_full to be de-activated. This happens because the scheduler plugin sets up perf event watching on all cores during startup and nohz_full is currently incompatible with perf event watching (see reproducer below).

Version-Release number of selected component (if applicable): tuned-2.4.1-1.20150705git1da9f3cc.el7.noarch


How reproducible:


Steps to Reproduce:
1. Set up nohz_full on a core. Eg., use the following options to the kernel command-line:

isolcpus=X nohz_full=X

2. Install tuned and activate a profile that uses the scheduler plugin. I'm using the realtime-virtual-host profile

3. Run the following script, where Y is the CPU mask corresponding to the nohz_full core and "stress" is any application capable of taking 100% of the CPU

"""
/bin/bash

cd /sys/kernel/debug/tracing
echo Y > tracing_cpumask
echo tick_stop > set_event
echo nop > current_tracer
echo > set_ftrace_filter
echo > /tracing/trace
echo 1 > /tracing/tracing_on

cd
chrt -f 1 taskset -c 15 ./stress --cpu 1 &
sleep 2

grep tick_stop /tracing/trace
pkill -9 stress
"""

Actual results:

Lots of:

tick_stop: success=no msg=perf events running


Expected results:

Only one of the following:

stress-3236  [015] d...1..   292.142704: tick_stop: success=yes msg=


Additional info:
Comment 1 Luiz Capitulino 2015-07-27 10:07:26 EDT
I forgot to mention that it's possible to disable the scheduler plugin runtime feature as a workaround for this issue. Simply edit the tuned.conf file for the profile and add runtime=0 to the scheduler section, like:

[scheduler]
runtime=0
Comment 2 Jaroslav Škarvada 2015-07-31 10:32:06 EDT
Upstream commit fixing this problem:
https://git.fedorahosted.org/cgit/tuned.git/commit/?id=5024f00ef59de394c31668696eb2f311f2aebc85
Comment 4 Fedora Update System 2015-09-03 14:49:50 EDT
tuned-2.5.1-2.fc21 has been pushed to the Fedora 21 stable repository. If problems still persist, please make note of it in this bug report.
Comment 5 Fedora Update System 2015-09-03 14:50:46 EDT
tuned-2.5.1-2.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.
Comment 7 errata-xmlrpc 2015-11-19 07:21:59 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2375.html

Note You need to log in before you can comment on or make changes to this bug.