Bug 1686446 - Tuned-adm cannot change cpu governor
Summary: Tuned-adm cannot change cpu governor
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: tuned
Version: 7.6
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: rc
: ---
Assignee: Ondřej Lysoněk
QA Contact: Robin Hack
URL:
Whiteboard:
Depends On: 1679205
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-07 13:32 UTC by Jiří Vozár
Modified: 2019-08-06 13:05 UTC (History)
8 users (show)

Fixed In Version: tuned-2.11.0-1.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-08-06 13:04:46 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github redhat-performance tuned pull 165 None closed Fallback to the 'powersave' CPU scaling governor in the 'balanced' and 'powersave' profiles 2020-01-29 12:52:23 UTC
Red Hat Product Errata RHBA-2019:2201 None None None 2019-08-06 13:05:18 UTC

Comment 5 Jiri Hladky 2019-05-02 10:18:29 UTC
Hi Robin,

you need an Intel CPU with intel_pstate CPU Performance Scaling Driver

https://www.kernel.org/doc/html/v4.12/admin-guide/pm/intel_pstate.html

See the description as well: "On RHEL 7 this is bug reproducible only on Intel processors for workstations,
where the conservative option is missing."

I'm installing RHEL-7.6 Workstation on W541 notebook:
https://beaker.cluster-qe.lab.eng.brq.redhat.com/bkr/view/w541.tpb.lab.eng.brq.redhat.com#details

Once it's completed I will give you ssh access (via e-mail).

Thanks
Jirka

Comment 9 Jiri Hladky 2019-05-02 15:32:16 UTC
Hi Jirko,

could you please try to reproduce the problem? It should be broken on RHEL7.6 Workstation but I was not able to reproduce it on these systems:

t470s.tpb.lab.eng.brq.redhat.com
t460p.tpb.lab.eng.brq.redhat.com
p50.tpb.lab.eng.brq.redhat.com

The available governors are just performance powersave

$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors 
performance powersave

but the governor switching works just fine despite having 

governor=conservative

in /usr/lib/tuned/balanced/tuned.conf

I have done testing with tuned 2.10.0 which should be broken (fix appeared in 2.11). Without reproducing the problem first we cannot really verify the fix. 

Thanks!
Jirka

Comment 10 Jiří Vozár 2019-05-02 21:36:35 UTC
The tuned always sets the governor to 'powersave' before loading and applying the selected profile (see below). If I wanted to set any non-existing governor which is equivalent of 'performance', the problem would occur again. The log from the tuned-adm command still shows that the governor setting failed (tested on the T470s, tuned v2.10.0):

$ tuned-adm --loglevel info profile balanced
INFO     tuned.daemon.daemon: stopping tuning
INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'normal' on cpu 'cpu2'
INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'normal' on cpu 'cpu3'
INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'normal' on cpu 'cpu0'
INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'normal' on cpu 'cpu1'
INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu2'
INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu3'
INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu0'
INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu1'
INFO     tuned.profiles.loader: loading profile: balanced
INFO     tuned.daemon.daemon: starting tuning
INFO     tuned.plugins.base: instance cpu: assigning devices cpu2, cpu3, cpu0, cpu1
INFO     tuned.plugins.plugin_cpu: intel_pstate detected
INFO     tuned.plugins.base: instance audio: assigning devices snd_hda_intel
INFO     tuned.plugins.base: instance video: assigning devices card0
WARNING  tuned.plugins.base: instance disk: no matching devices available
INFO     tuned.plugins.base: instance scsi_host: assigning devices host0
INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu 'cpu2', it is not supported
INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu 'cpu3', it is not supported
INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu 'cpu0', it is not supported
INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu 'cpu1', it is not supported
INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'normal' on cpu 'cpu2'
INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'normal' on cpu 'cpu3'
INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'normal' on cpu 'cpu0'
INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'normal' on cpu 'cpu1'
ERROR    tuned.utils.commands: Error when reading file '/sys/class/drm/card0/device/power_method': '[Errno 2] No such file or directory: u'/sys/class/drm/card0/device/power_method''
WARNING  tuned.plugins.plugin_video: radeon_powersave is not supported on 'card0'
INFO     tuned.plugins.plugin_scsi_host: ALPM control file ('/sys/class/scsi_host/host0/link_power_management_policy') not found, skipping ALPM setting for 'host0'
INFO     tuned.daemon.daemon: static tuning from profile 'balanced' applied

Comment 11 Jiri Hladky 2019-05-02 21:59:41 UTC
Thanks for the update! I have verified that with tuned-2.11.0-2.el7:

* tuned.conf has both possible profiles (for example conservative OR powesave) properly listed[1]
* no more unsupported governor messages [2]

Problem is gone:-) 


@Robin - could please enhance your tests to use --loglevel info and look for messages like this one?
INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu 'cpu2', it is not supported
(when the unsupported governor is reported, the testcase will fail). 



[1] $ find /usr/lib/tuned -name tuned.conf -exec grep -H governor= {} \; 
/usr/lib/tuned/balanced/tuned.conf:governor=conservative|powersave 
/usr/lib/tuned/latency-performance/tuned.conf:governor=performance 
/usr/lib/tuned/powersave/tuned.conf:governor=ondemand|powersave 
/usr/lib/tuned/throughput-performance/tuned.conf:governor=performance

[2] $ tuned-adm --loglevel info profile powersave 
INFO     tuned.daemon.daemon: stopping tuning 
INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'normal' on cpu 'cpu6' 
INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'normal' on cpu 'cpu7' 
INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'normal' on cpu 'cpu4' 
INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'normal' on cpu 'cpu5' 
INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'normal' on cpu 'cpu2' 
INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'normal' on cpu 'cpu3' 
INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'normal' on cpu 'cpu0' 
INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'normal' on cpu 'cpu1' 
INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu6' 
INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu7' 
INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu4' 
INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu5' 
INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu2' 
INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu3' 
INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu0' 
INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu1' 
INFO     tuned.profiles.loader: loading profile: powersave 
INFO     tuned.daemon.daemon: starting tuning 
INFO     tuned.units.manager: skipping plugin 'eeepc_she', not supported on your system 
INFO     tuned.plugins.base: instance cpu: assigning devices cpu6, cpu7, cpu4, cpu5, cpu2, cpu3, cpu0, cpu1 
INFO     tuned.plugins.plugin_cpu: We are running on an x86 GenuineIntel platform 
INFO     tuned.plugins.plugin_cpu: intel_pstate detected 
INFO     tuned.plugins.base: instance audio: assigning devices snd_hda_intel 
INFO     tuned.plugins.base: instance video: assigning devices card1, card0 
INFO     tuned.plugins.base: instance disk: assigning devices sda 
INFO     tuned.plugins.base: instance net: assigning devices wlp3s0, enp0s31f6 
INFO     tuned.plugins.base: instance scsi_host: assigning devices host3, host2, host1, host0 
INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu6' 
INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu7' 
INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu4' 
INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu5' 
INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu2' 
INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu3' 
INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu0' 
INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu1' 
INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'powersave' on cpu 'cpu6' 
INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'powersave' on cpu 'cpu7' 
INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'powersave' on cpu 'cpu4' 
INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'powersave' on cpu 'cpu5' 
INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'powersave' on cpu 'cpu2' 
INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'powersave' on cpu 'cpu3' 
INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'powersave' on cpu 'cpu0' 
INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'powersave' on cpu 'cpu1' 
ERROR    tuned.utils.commands: Error when reading file '/sys/class/drm/card1/device/power_method': '[Errno 2] No such file or directory: u'/sys/class/drm/card1/device/power_method'' 
WARNING  tuned.plugins.plugin_video: radeon_powersave is not supported on 'card1' 
ERROR    tuned.utils.commands: Error when reading file '/sys/class/drm/card0/device/power_method': '[Errno 2] No such file or directory: u'/sys/class/drm/card0/device/power_method'' 
WARNING  tuned.plugins.plugin_video: radeon_powersave is not supported on 'card0' 
INFO     tuned.plugins.plugin_sysctl: reapplying system sysctl 
ERROR    tuned.plugins.plugin_sysctl: Failed to set sysctl parameter 'net.bridge.bridge-nf-call-ip6tables' to '0', the parameter does not exist 
ERROR    tuned.plugins.plugin_sysctl: Failed to set sysctl parameter 'net.bridge.bridge-nf-call-iptables' to '0', the parameter does not exist 
ERROR    tuned.plugins.plugin_sysctl: Failed to set sysctl parameter 'net.bridge.bridge-nf-call-arptables' to '0', the parameter does not exist 
INFO     tuned.plugins.plugin_script: calling script '/usr/lib/tuned/powersave/script.sh' with arguments '['start']' 
INFO     tuned.daemon.daemon: static tuning from profile 'powersave' applied

Comment 13 Ondřej Lysoněk 2019-05-03 09:23:40 UTC
(In reply to Jiri Hladky from comment #11)
> @Robin - could please enhance your tests to use --loglevel info and look for
> messages like this one?
> INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu
> 'cpu2', it is not supported
> (when the unsupported governor is reported, the testcase will fail). 

That seems unnecessary. Why do you think checking for that is important? The important thing is that Tuned sets the correct governor when a profile is applied and that it restores the previously set governor when the profile is unapplied. Nothing else matters.

Comment 14 Jiri Hladky 2019-05-03 09:34:01 UTC
> 
> That seems unnecessary. Why do you think checking for that is important? The
> important thing is that Tuned sets the correct governor when a profile is
> applied and that it restores the previously set governor when the profile is
> unapplied. Nothing else matters.

Well, in case of the unsupported governor (like this one) 

INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu 'cpu2', it is not supported

we should go ahead and fix the tuned.conf (adding another governor to the OR list). Without having this test you end up with some unpredictable governor. It may be the correct one but on some other CPU model, it could be a wrong one. Either way, IMHO it should be flagged by our tests so that we can fix tuned.conf. 

@Robin - on how many different CPU models do we run the tests? 

Thanks!
Jirka

Comment 19 errata-xmlrpc 2019-08-06 13:04:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2201


Note You need to log in before you can comment on or make changes to this bug.