Bug 1679205

Hm, so the machine only supports governors 'performance' and 'powersave'. Consequently, after activating the 'balanced' profile (which tries to set the conservative governor), the governor remains set to 'performance', as that is the kernel default.

The same thing happens with the powersave profile, which tries to set the 'ondemand' governor.

We should probably fallback to the powersave governor in the powersave profile, if 'ondemand' is not available. However I'm not sure if we should prefer the 'powersave' governor over 'performance' in the 'balanced' profile. Perhaps we should keep that as it is.

What I'm also not clear about at the moment is whether the list of supported governor has always been this limited on this hardware, or if it changed due to some kernel change.

I have checked the available governors on my laptop (T530) with i7-3520M CPU @ 2.90GHz and available governors are only "performance" and "powersave". With powersave the CPU frequency scaling works as expected. 

The same applies for Xeon CPUs running RHEL-8 kernel (tested on CPU E5-4607 0 @ 2.20GHz) - the available governors are only "performance" and "powersave".

According to https://www.kernel.org/doc/html/v4.12/admin-guide/pm/intel_pstate.html this is expected when intel_pstate CPU Performance Scaling Driver is used. 

On AMD the situation is different: AMD EPYC 7351
cat /sys/devices/system/cpu/cpu1/cpufreq/scaling_available_governors
conservative ondemand userspace powersave performance 

Since balanced tuned profile is intended for the Laptops I suggest to use the powersave governor if the conservative governor is not available. As far as I can tell, intel_pstate "powersave" governor is the closest to the conservative governor on other CPUs. 

Thanks a lot
Jirka

Thanks for the info.

Hi Ondrej,

any updated on that? I think that we should try to fix this as soon as possible as it breaks CPU governor setting on Intel. 

Thanks
Jirka

(In reply to Jiri Hladky from comment #5)
> Hi Ondrej,
> 
> any updated on that? I think that we should try to fix this as soon as
> possible as it breaks CPU governor setting on Intel. 

No update, sorry. I didn't treat it as a priority. So is this a regression compared to RHEL-7? As I understand it, the intel_pstate driver is used in RHEL-7 too, so there should be the exact same problem.

Hi Ondrej,

yes, this is a regression compared to RHEL-7. intel_pstate CPU Performance Scaling Driver behaves differently in RHEL-7 and in RHEL-8 (there are different profiles). For me, it's a blocker since tuned in RHEL-8 is not able to change the CPU governor on Intel CPUs. So even if you set "powersave" tuned profile, the CPU profile will remain in the performance state. 

Thanks!
Jirka

Setting Regression keyword based on comment#8.

Agreed on the blocker assessment. 

Thanks for looking into it! 

Just a small note - please keep in mind that while in RHEL-8 the profiles supported on Intel CPUs are only "performance" and "powersave", on other CPUs (namely on AMD) there are the same CPU governors available as in RHEL-7:

AMD EPYC 7351
cat /sys/devices/system/cpu/cpu1/cpufreq/scaling_available_governors
conservative ondemand userspace powersave performance 

So the tuned has to behave differently on Intel and on other CPUs. 

Thanks!
Jirka

Hi, I'm QE of tuned for the RHEL-8.0.0. I don't have capacity to do whole regression testing again, it takes e.g. 3 days. I can test a fix, but not to investigate regressions.

Info for QE:
============

You need to have available all necessary cpufreq governors and it should have drive intel_pstate:

# cpupower frequency-info | grep 'available cpufreq governors'
  available cpufreq governors: conservative ondemand userspace powersave performance
# cpupower frequency-info | grep driver
  driver: intel_pstate

Then, just run:

# cat /usr/lib/tuned/throughput-performance/tuned.conf | grep governor
governor=performance
# tuned-adm profile throughput-performance
# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
performance

# cat /usr/lib/tuned/balanced/tuned.conf | grep governor
governor=conservative
# tuned-adm profile balanced
# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
conservative

The last row is necessary for testing of the fix.

(In reply to Tereza Cerna from comment #12)
> You need to have available all necessary cpufreq governors and it should
> have drive intel_pstate:
> 
> # cpupower frequency-info | grep 'available cpufreq governors'
>   available cpufreq governors: conservative ondemand userspace powersave

No, this is not right. If these governors are available, you won't be able to reproduce the issue. The problem occurs only when the available cpufreq governors are "performance powersave".


> # cat /usr/lib/tuned/balanced/tuned.conf | grep governor
> governor=conservative

Note that once the bug is fixed, this value in the profile will likely be different. It'll probably contain "governor=conservative|powersave". I.e., tuned will set the 'conservative' governor, if it's available, otherwise it'll set 'powersave'.

> # tuned-adm profile balanced
> # cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
> conservative
>
> The last row is necessary for testing of the fix

No, you should get 'powersave' here once the bug is fixed.

Upstream pull request:
https://github.com/redhat-performance/tuned/pull/165

So I finally got a machine to take a look at how it behaves in RHEL-7 and I think this really cannot be considered a regression in Tuned. Setting the scaling governors 'conservative' and 'ondemand' doesn't work there either if the intel_pstate driver is active. The only available governors in RHEL-7 are 'powersave' and 'performance', same as in RHEL-8. The only difference is that in RHEL-7, the default governor is 'powersave', whereas in RHEL-8 the default is 'performance', so you don't notice the problem in RHEL-7. However this has nothing to do with Tuned.

# uname -a
Linux hp-skylake-z4-01.lab.eng.rdu.redhat.com 3.10.0-957.el7.x86_64 #1 SMP Thu Oct 4 20:48:51 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
# rpm -q tuned
tuned-2.10.0-6.el7.noarch
# cpupower frequency-info | grep 'driver\|available'
  driver: intel_pstate
  available cpufreq governors: performance powersave
# systemctl stop tuned
# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor | uniq
powersave
# > /var/log/tuned/tuned.log 
# tuned-adm profile balanced
Cannot talk to Tuned daemon via DBus. Is Tuned daemon running?
Trying to (re)start tuned...
Tuned (re)started, changes applied.
# grep governor /var/log/tuned/tuned.log 
2019-03-07 03:32:03,743 INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu 'cpu8', it is not supported
2019-03-07 03:32:03,743 INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu 'cpu9', it is not supported
2019-03-07 03:32:03,744 INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu 'cpu6', it is not supported
[...]

Thanks for the update! I will check it on RHEL-7 as well and let you know the outcome. 

Based on this I would suggest
1. Let's remove the blocker flag
2. We should fix tuned both in RHEL-7 and RHEL-8. Should we open a new BZ for RHEL-7? 

Jirka

(In reply to Jiri Hladky from comment #20)
> 2. We should fix tuned both in RHEL-7 and RHEL-8. Should we open a new BZ
> for RHEL-7? 

Yes, please.

Hi Ondrej,

on RHEL-7.6 the situation seems to be more complicated and it seems there are different lists of available CPU governors on different Intel CPUs:

RHEL-7.6 Server variant on 
  * gold-1s.tpb.lab.eng.brq.redhat.com - Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz
# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors
conservative userspace powersave ondemand performance

=> tuned can switch the CPU governors 

RHEL-7.6 Server variant on 
  * w541.tpb.lab.eng.brq.redhat.com - Intel(R) Core(TM) i7-4810MQ CPU @ 2.80GHz
[root@w541 ~]# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors
performance powersave

=> tuned is not able to switch the governors

  * t460p.tpb.lab.eng.brq.redhat.com - Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz
[root@t460p ~]# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors
performance powersave

=> tuned is not able to switch the governors

  * t470s.tpb.lab.eng.brq.redhat.com - Intel(R) Core(TM) i7-7600U CPU @ 2.80GHz
[root@t470s ~]# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors
performance powersave

So I would still argue that this is a regression since on RHEL-7, tuned is able to switch the governor on Server CPUs. I will double check that by installing RHEL-8 on these systems. 

Jirka Vozar will open a BZ for RHEL-7. 

Thanks!
Jirka

I have now tried to install RHEL-7.6 Workstation on servers and RHEL-7.6 Server on laptops (same HW as in the comment #22) and the results was exactly the same. So in RHEL-7, the available CPU governors on Intel CPUs will depend on CPU model. 

I will now try the same with RHEL-8. 

Jirka

(In reply to Jiri Hladky from comment #23)
> I have now tried to install RHEL-7.6 Workstation on servers and RHEL-7.6
> Server on laptops (same HW as in the comment #22) and the results was
> exactly the same. So in RHEL-7, the available CPU governors on Intel CPUs
> will depend on CPU model. 

More precisely, it depends on the scaling driver that is used (/sys/devices/system/cpu/cpu*/cpufreq/scaling_driver). With the intel_pstate driver you get the powersave and performance driver. If acpi-cpufreq is used instead, you'll get (I think) conservative, userspace, powersave, ondemand, performance. Nevertheless, the driver that gets used by default is dependent on the CPU model (you might encounter some differences in the "CPU model->default driver" mapping between different kernel versions).

> More precisely, it depends on the scaling driver that is used (/sys/devices/system/cpu/cpu*/cpufreq/scaling_driver).

Yes, you are right! The interesting point is that on RHEL-8, I will always get intel_pstate driver on Intel CPUs. Tested on:


gold-1s.tpb.lab.eng.brq.redhat.com - Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz
w541.tpb.lab.eng.brq.redhat.com - Intel(R) Core(TM) i7-4810MQ CPU @ 2.80GHz
t460p.tpb.lab.eng.brq.redhat.com - Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz
t470s.tpb.lab.eng.brq.redhat.com - Intel(R) Core(TM) i7-7600U CPU @ 2.80GHz

On all these systems with RHEL-8 (RHEL-8.0.0-20190228.1, kernel 4.18.0-74.el8.x86_64) the intel_pstate driver is used. This is the change to RHEL-7.6, where on laptops, acpi-cpufreq is used.

Fixed upstream by PR:
https://github.com/redhat-performance/tuned/pull/165/commits/52873fa9c8a85d6bb155401282c2c0c8f7bf4f27

(In reply to Jaroslav Škarvada from comment #29)
> Fixed upstream by PR:
> https://github.com/redhat-performance/tuned/pull/165/commits/
> 52873fa9c8a85d6bb155401282c2c0c8f7bf4f27

https://github.com/redhat-performance/tuned/pull/165

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3633