Bug 1706171

Summary: tuned fails to apply profile with the provided core list
Product: Red Hat Enterprise Linux 7 Reporter: Nitesh Narayan Lal <nilal>
Component: tunedAssignee: Jaroslav Škarvada <jskarvad>
Status: CLOSED ERRATA QA Contact: Robin Hack <rhack>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 7.7CC: bhu, jeder, jianzzha, jskarvad, lcapitulino, olysonek, pezhang, psklenar, rhack
Target Milestone: rcKeywords: Patch, Regression, Upstream
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard:
Fixed In Version: tuned-2.11.0-3.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-06 13:04:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
patch none

Description Nitesh Narayan Lal 2019-05-03 17:33:16 UTC
Description of problem:
System with 2 NUMA Nodes and 36 cores.
lscpu | grep ^NUMA
NUMA node(s):          2
NUMA node0 CPU(s):     0-17
NUMA node1 CPU(s):     18-35

On isolating cores 19-35 tuned fails to activate the realtime-virtual-host profile. It results in the following error:
Cannot load profile(s) 'realtime-virtual-host': Assertion 'isolated_cores contains online CPU(s)' failed.


Version-Release number of selected component:
Kernel: 3.10.0-1040.rt56.999.el7.x86_64


How reproducible:
Every time for cores 19-35.

Steps to Reproduce:
[root@intel-skylake-loaner ~]# tuned-adm active
No current active profile.

[root@intel-skylake-loaner ~]# cat /etc/tuned/realtime-virtual-host-variables.conf
# Examples:
# isolated_cores=2,4-7
# isolated_cores=2-23
#
isolated_cores=32,33,34,35,19,20,21,22,23,24,25,26,27,28,29,30,31

[root@intel-skylake-loaner ~]# tuned-adm profile realtime-virtual-host
Cannot load profile(s) 'realtime-virtual-host': Assertion 'isolated_cores contains online CPU(s)' failed.

[root@intel-skylake-loaner ~]# cat /sys/devices/system/cpu/online
0-35


tuned.log:
2019-05-03 13:27:58,261 INFO     tuned.profiles.loader: loading profile: realtime-virtual-host
2019-05-03 13:27:58,266 ERROR    tuned.profiles.functions.function_assertion: assertion 'isolated_cores contains online CPU(s)' failed: '19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35' != '32,33,34,35,19,20,21,22,23,24,25,26,27,28,29,30,31'
2019-05-03 13:27:58,266 ERROR    tuned.daemon.controller: Failed to apply profile 'realtime-virtual-host'

Comment 2 Nitesh Narayan Lal 2019-05-07 14:23:37 UTC
The issue is reproducible on RT kernel(3.10.0-957.19.1.rt56.931.el7) and on non-RT kernel(3.10.0-1040.el7.x86_64) as well.
Tuned version: 2.10.0

Following are some of the logs showing the error with both above-stated kernels.

RT Kernel:
[root@intel-skylake-loaner ~]# uname -a
Linux intel-skylake-loaner.virt.lab.eng.bos.redhat.com 3.10.0-957.19.1.rt56.931.el7.x86_64 #1 SMP PREEMPT RT Mon Apr 29 10:09:25 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@intel-skylake-loaner ~]# uptime 
 10:17:55 up 1 min,  1 user,  load average: 0.35, 0.22, 0.08
[root@intel-skylake-loaner ~]# systemctl status tuned
● tuned.service - Dynamic System Tuning Daemon
   Loaded: loaded (/usr/lib/systemd/system/tuned.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-05-07 10:16:44 EDT; 2min 11s ago
     Docs: man:tuned(8)
           man:tuned.conf(5)
           man:tuned-adm(8)
 Main PID: 8061 (tuned)
    Tasks: 5
   CGroup: /system.slice/tuned.service
           └─8061 /usr/bin/python2 -Es /usr/sbin/tuned -l -P

May 07 10:16:43 intel-skylake-loaner.virt.lab.eng.bos.redhat.com systemd[1]: Starting Dynamic System Tuning Daemon...
May 07 10:16:44 intel-skylake-loaner.virt.lab.eng.bos.redhat.com systemd[1]: Started Dynamic System Tuning Daemon.
[root@intel-skylake-loaner ~]# tuned-adm active
Current active profile: balanced
[root@intel-skylake-loaner ~]# cat /etc/tuned/realtime-virtual-host-variables.conf
# Examples:
# isolated_cores=2,4-7
# isolated_cores=2-23
isolated_cores=19-35
[root@intel-skylake-loaner ~]# cat /sys/devices/system/cpu/online
0-35
[root@intel-skylake-loaner ~]# tuned-adm profile realtime-virtual-host
Cannot load profile(s) 'realtime-virtual-host': Assertion 'isolated_cores contains online CPU(s)' failed.
[root@intel-skylake-loaner ~]# 


Non-RT Kernel:

[root@intel-skylake-loaner ~]# uname -a
Linux intel-skylake-loaner.virt.lab.eng.bos.redhat.com 3.10.0-1040.el7.x86_64 #1 SMP Thu Apr 18 15:22:56 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@intel-skylake-loaner ~]# uptime 
 10:11:50 up 0 min,  1 user,  load average: 0.70, 0.21, 0.07
[root@intel-skylake-loaner ~]# tuned -v
tuned 2.10.0
[root@intel-skylake-loaner ~]# systemctl status tuned
● tuned.service - Dynamic System Tuning Daemon
   Loaded: loaded (/usr/lib/systemd/system/tuned.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-05-07 10:11:43 EDT; 17s ago
     Docs: man:tuned(8)
           man:tuned.conf(5)
           man:tuned-adm(8)
 Main PID: 4608 (tuned)
    Tasks: 5
   CGroup: /system.slice/tuned.service
           └─4608 /usr/bin/python2 -Es /usr/sbin/tuned -l -P

May 07 10:11:41 intel-skylake-loaner.virt.lab.eng.bos.redhat.com systemd[1]: Starting Dynamic System Tuning Daemon...
May 07 10:11:43 intel-skylake-loaner.virt.lab.eng.bos.redhat.com systemd[1]: Started Dynamic System Tuning Daemon.
[root@intel-skylake-loaner ~]# tuned-adm active
Current active profile: balanced

[root@intel-skylake-loaner ~]# cat /sys/devices/system/cpu/online
0-35
[root@intel-skylake-loaner ~]# cat /etc/tuned/realtime-virtual-host-variables.conf
# Examples:
# isolated_cores=2,4-7
# isolated_cores=2-23
isolated_cores=19-35
[root@intel-skylake-loaner ~]# tuned-adm profile realtime-virtual-host
Cannot load profile(s) 'realtime-virtual-host': Assertion 'isolated_cores contains online CPU(s)' failed.
[root@intel-skylake-loaner ~]#

Comment 3 Luiz Capitulino 2019-05-07 16:04:40 UTC
Jaroslav,

As mentioned by Nitesh in the previous comment, this issue affects 7.6.z. Since
CPU isolation is necessary for KVM-RT to work properly and since we have
customers running 7.6.z on the field on Skylake, I think it's just a matter of
time they hit this issue which is a blocker for us.

Comment 4 Ondřej Lysoněk 2019-05-07 16:17:22 UTC
Hi, I'm working on this one, give me a second. Btw, I don't think it's a regression...

Comment 5 Jaroslav Škarvada 2019-05-07 16:19:42 UTC
No problem with the z-stream.

Comment 6 Ondřej Lysoněk 2019-05-07 16:26:25 UTC
Created attachment 1565282 [details]
patch

Can you check if it works with the attached patch applied to /usr/lib/python2.7/site-packages/tuned/profiles/functions/function_cpulist_online.py ?

Comment 7 Nitesh Narayan Lal 2019-05-07 16:41:39 UTC
Yes, it worked.
Thanks.

Comment 11 Jaroslav Škarvada 2019-05-09 08:32:04 UTC
Meged upstream PR:
https://github.com/redhat-performance/tuned/pull/183

Comment 12 Ondřej Lysoněk 2019-05-14 07:25:44 UTC
*** Bug 1643944 has been marked as a duplicate of this bug. ***

Comment 16 errata-xmlrpc 2019-08-06 13:04:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2201