Bug 1264128 - Convert tuned to use new /sys/devices/system/cpu features for isolated and nohz_full cores
Convert tuned to use new /sys/devices/system/cpu features for isolated and no...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: tuned (Show other bugs)
7.3
All Linux
unspecified Severity medium
: rc
: ---
Assigned To: Jaroslav Škarvada
Tereza Cerna
: Upstream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-09-17 10:41 EDT by Jeremy Eder
Modified: 2016-11-04 03:25 EDT (History)
3 users (show)

See Also:
Fixed In Version: tuned-2.7.0-1.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-11-04 03:25:22 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Comment 2 Jaroslav Škarvada 2016-06-16 07:24:53 EDT
The mentioned kernel patches added support to show current isolcpus and nohz_full through sysfs. How could Tuned use this information? This sysfs doesn't seem to help before the machine is rebooted with the new kernel command line.

But the current situation may need improvement. It seems that kernel is quite restrictive about isolcpus and nohz_full settings. It seems it ignores the settings if there are non-valid or non-existent CPUs specified. I think this could be improved by adding Tuned built-in function, e.g. "function_cpulist_unpack_available" or similar to filter user specified CPUs through available CPUs. It is still not optimal solution, because the number of CPUs could change between reboots, but I think it would be better than the current state.
Comment 3 Jaroslav Škarvada 2016-06-16 11:18:49 EDT
I am going to filter the cpulist through:
/sys/devices/system/cpu/present - cpus that have been identified as being present in the system.

I will name the function cpulist_present.
Comment 4 Jeremy Eder 2016-06-16 11:27:33 EDT
The way tuned could use this is to "fail" the profile load if the variables.conf info does not match cpulist_present.  It's for error-checking.
Comment 5 Jaroslav Škarvada 2016-06-16 12:08:22 EDT
(In reply to Jeremy Eder from comment #4)
> The way tuned could use this is to "fail" the profile load if the
> variables.conf info does not match cpulist_present.  It's for error-checking.

Thanks for info.

I added the following functions, which could be useful:
cpulist_present - filters the cpulist through present CPUs
cpulist_pack - to pack the cpulist, e.g. converts 1,2,3 to 1-3

I initially wanted to filter ${isolated_cores} through the cpulist_present in the Tuned profiles, but I will just add the error check.
Comment 6 Jaroslav Škarvada 2016-06-16 12:59:09 EDT
I added 'assertion' function, usage:
dummy = ${f:assertion:error_text:check_val:curr_val}

it compares check_val with curr_val and if they don't match it logs error_text and aborts profile loading. I will add the assertion to realtime profiles.
Comment 7 Jeremy Eder 2016-06-16 13:18:42 EDT
Perfect, thanks Jaroslav!
Comment 9 Jaroslav Škarvada 2016-06-17 09:03:30 EDT
Example of error logged
2016-06-17 14:10:44,267 ERROR    tuned.profiles.functions.function_assertion: assertion 'isolated_cores contains present CPU(s)' failed: '1,2,3,4,5' != '1,2,3'
Comment 12 Tereza Cerna 2016-09-08 01:36:02 EDT
==============================================
Verified in:
    tuned-2.7.1-2.el7.noarch
    tuned-profiles-realtime-2.7.1-2.el7.noarch
    tuned-profiles-nfv-2.7.1-2.el7.noarch
PASS
==============================================

Check /usr/lib/tuned/$PROFILE/tuned.conf files for realtime, realtime-virtual-host, realtime-virtual-guest profiles and patches was applied on these files.

# cat /proc/cpuinfo | grep processor | wc -l
88
# cat /etc/tuned/realtime-variables.conf 
isolated_cores=80-90
# tuned-adm profile realtime
Cannot load profile 'realtime': Assertion 'isolated_cores contains present CPU(s)' failed.
# cat /var/log/tuned/tuned.log | grep ERROR
2016-09-06 07:45:54,884 ERROR    tuned.profiles.functions.function_assertion: assertion 'isolated_cores contains present CPU(s)' failed: '80,81,82,83,84,85,86,87,88,89,90' != '80,81,82,83,84,85,86,87'
# tuned-adm active
Current active profile: balanced

The same behavior also for realtime-virtual-host and realtime-virtual-guest profiles. Content of files isolated and nohz_full in directory /sys/devices/system/cpu.

==================================================
Reproduced in:
    tuned-2.5.1-4.el7_2.3.noarch
    tuned-profiles-realtime-2.5.1-4.el7_2.3.noarch
    tuned-profiles-nfv-2.5.1-4.el7_2.3.noarch
FAIL
==================================================

# cat /proc/cpuinfo | grep processor | wc -l
88
# cat /etc/tuned/realtime-variables.conf 
isolated_cores=80-90
# tuned-adm profile realtime
# cat /var/log/tuned/tuned.log | grep ERROR
# tuned-adm active
Current active profile: realtime
Comment 14 errata-xmlrpc 2016-11-04 03:25:22 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2479.html

Note You need to log in before you can comment on or make changes to this bug.