Bug 1264128

Summary: Convert tuned to use new /sys/devices/system/cpu features for isolated and nohz_full cores
Product: Red Hat Enterprise Linux 7 Reporter: Jeremy Eder <jeder>
Component: tunedAssignee: Jaroslav Škarvada <jskarvad>
Status: CLOSED ERRATA QA Contact: Tereza Cerna <tcerna>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 7.3CC: jeder, jskarvad, tcerna
Target Milestone: rcKeywords: Upstream
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: tuned-2.7.0-1.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-04 07:25:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Comment 2 Jaroslav Škarvada 2016-06-16 11:24:53 UTC
The mentioned kernel patches added support to show current isolcpus and nohz_full through sysfs. How could Tuned use this information? This sysfs doesn't seem to help before the machine is rebooted with the new kernel command line.

But the current situation may need improvement. It seems that kernel is quite restrictive about isolcpus and nohz_full settings. It seems it ignores the settings if there are non-valid or non-existent CPUs specified. I think this could be improved by adding Tuned built-in function, e.g. "function_cpulist_unpack_available" or similar to filter user specified CPUs through available CPUs. It is still not optimal solution, because the number of CPUs could change between reboots, but I think it would be better than the current state.

Comment 3 Jaroslav Škarvada 2016-06-16 15:18:49 UTC
I am going to filter the cpulist through:
/sys/devices/system/cpu/present - cpus that have been identified as being present in the system.

I will name the function cpulist_present.

Comment 4 Jeremy Eder 2016-06-16 15:27:33 UTC
The way tuned could use this is to "fail" the profile load if the variables.conf info does not match cpulist_present.  It's for error-checking.

Comment 5 Jaroslav Škarvada 2016-06-16 16:08:22 UTC
(In reply to Jeremy Eder from comment #4)
> The way tuned could use this is to "fail" the profile load if the
> variables.conf info does not match cpulist_present.  It's for error-checking.

Thanks for info.

I added the following functions, which could be useful:
cpulist_present - filters the cpulist through present CPUs
cpulist_pack - to pack the cpulist, e.g. converts 1,2,3 to 1-3

I initially wanted to filter ${isolated_cores} through the cpulist_present in the Tuned profiles, but I will just add the error check.

Comment 6 Jaroslav Škarvada 2016-06-16 16:59:09 UTC
I added 'assertion' function, usage:
dummy = ${f:assertion:error_text:check_val:curr_val}

it compares check_val with curr_val and if they don't match it logs error_text and aborts profile loading. I will add the assertion to realtime profiles.

Comment 7 Jeremy Eder 2016-06-16 17:18:42 UTC
Perfect, thanks Jaroslav!

Comment 9 Jaroslav Škarvada 2016-06-17 13:03:30 UTC
Example of error logged
2016-06-17 14:10:44,267 ERROR    tuned.profiles.functions.function_assertion: assertion 'isolated_cores contains present CPU(s)' failed: '1,2,3,4,5' != '1,2,3'

Comment 12 Tereza Cerna 2016-09-08 05:36:02 UTC
==============================================
Verified in:
    tuned-2.7.1-2.el7.noarch
    tuned-profiles-realtime-2.7.1-2.el7.noarch
    tuned-profiles-nfv-2.7.1-2.el7.noarch
PASS
==============================================

Check /usr/lib/tuned/$PROFILE/tuned.conf files for realtime, realtime-virtual-host, realtime-virtual-guest profiles and patches was applied on these files.

# cat /proc/cpuinfo | grep processor | wc -l
88
# cat /etc/tuned/realtime-variables.conf 
isolated_cores=80-90
# tuned-adm profile realtime
Cannot load profile 'realtime': Assertion 'isolated_cores contains present CPU(s)' failed.
# cat /var/log/tuned/tuned.log | grep ERROR
2016-09-06 07:45:54,884 ERROR    tuned.profiles.functions.function_assertion: assertion 'isolated_cores contains present CPU(s)' failed: '80,81,82,83,84,85,86,87,88,89,90' != '80,81,82,83,84,85,86,87'
# tuned-adm active
Current active profile: balanced

The same behavior also for realtime-virtual-host and realtime-virtual-guest profiles. Content of files isolated and nohz_full in directory /sys/devices/system/cpu.

==================================================
Reproduced in:
    tuned-2.5.1-4.el7_2.3.noarch
    tuned-profiles-realtime-2.5.1-4.el7_2.3.noarch
    tuned-profiles-nfv-2.5.1-4.el7_2.3.noarch
FAIL
==================================================

# cat /proc/cpuinfo | grep processor | wc -l
88
# cat /etc/tuned/realtime-variables.conf 
isolated_cores=80-90
# tuned-adm profile realtime
# cat /var/log/tuned/tuned.log | grep ERROR
# tuned-adm active
Current active profile: realtime

Comment 14 errata-xmlrpc 2016-11-04 07:25:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2479.html