RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1706171 - tuned fails to apply profile with the provided core list
Summary: tuned fails to apply profile with the provided core list
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: tuned
Version: 7.7
Hardware: x86_64
OS: Unspecified
unspecified
urgent
Target Milestone: rc
: ---
Assignee: Jaroslav Škarvada
QA Contact: Robin Hack
URL:
Whiteboard:
: 1643944 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-05-03 17:33 UTC by Nitesh Narayan Lal
Modified: 2019-08-06 13:05 UTC (History)
9 users (show)

Fixed In Version: tuned-2.11.0-3.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-08-06 13:04:46 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
patch (617 bytes, patch)
2019-05-07 16:26 UTC, Ondřej Lysoněk 		no flags Details | Diff
View All


Links
System ID Private Priority Status Summary Last Updated
Github redhat-performance tuned pull 183 0 None closed functions: Return an ordered cpu list in cpulist_online 2020-12-17 01:42:24 UTC
Red Hat Product Errata RHBA-2019:2201 0 None None None 2019-08-06 13:05:18 UTC

Description Nitesh Narayan Lal 2019-05-03 17:33:16 UTC 
Description of problem:
System with 2 NUMA Nodes and 36 cores.
lscpu | grep ^NUMA
NUMA node(s):          2
NUMA node0 CPU(s):     0-17
NUMA node1 CPU(s):     18-35

On isolating cores 19-35 tuned fails to activate the realtime-virtual-host profile. It results in the following error:
Cannot load profile(s) 'realtime-virtual-host': Assertion 'isolated_cores contains online CPU(s)' failed.


Version-Release number of selected component:
Kernel: 3.10.0-1040.rt56.999.el7.x86_64


How reproducible:
Every time for cores 19-35.

Steps to Reproduce:
[root@intel-skylake-loaner ~]# tuned-adm active
No current active profile.

[root@intel-skylake-loaner ~]# cat /etc/tuned/realtime-virtual-host-variables.conf
# Examples:
# isolated_cores=2,4-7
# isolated_cores=2-23
#
isolated_cores=32,33,34,35,19,20,21,22,23,24,25,26,27,28,29,30,31

[root@intel-skylake-loaner ~]# tuned-adm profile realtime-virtual-host
Cannot load profile(s) 'realtime-virtual-host': Assertion 'isolated_cores contains online CPU(s)' failed.

[root@intel-skylake-loaner ~]# cat /sys/devices/system/cpu/online
0-35


tuned.log:
2019-05-03 13:27:58,261 INFO     tuned.profiles.loader: loading profile: realtime-virtual-host
2019-05-03 13:27:58,266 ERROR    tuned.profiles.functions.function_assertion: assertion 'isolated_cores contains online CPU(s)' failed: '19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35' != '32,33,34,35,19,20,21,22,23,24,25,26,27,28,29,30,31'
2019-05-03 13:27:58,266 ERROR    tuned.daemon.controller: Failed to apply profile 'realtime-virtual-host'
Comment 2 Nitesh Narayan Lal 2019-05-07 14:23:37 UTC 
The issue is reproducible on RT kernel(3.10.0-957.19.1.rt56.931.el7) and on non-RT kernel(3.10.0-1040.el7.x86_64) as well.
Tuned version: 2.10.0

Following are some of the logs showing the error with both above-stated kernels.

RT Kernel:
[root@intel-skylake-loaner ~]# uname -a
Linux intel-skylake-loaner.virt.lab.eng.bos.redhat.com 3.10.0-957.19.1.rt56.931.el7.x86_64 #1 SMP PREEMPT RT Mon Apr 29 10:09:25 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@intel-skylake-loaner ~]# uptime 
 10:17:55 up 1 min,  1 user,  load average: 0.35, 0.22, 0.08
[root@intel-skylake-loaner ~]# systemctl status tuned
● tuned.service - Dynamic System Tuning Daemon
   Loaded: loaded (/usr/lib/systemd/system/tuned.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-05-07 10:16:44 EDT; 2min 11s ago
     Docs: man:tuned(8)
           man:tuned.conf(5)
           man:tuned-adm(8)
 Main PID: 8061 (tuned)
    Tasks: 5
   CGroup: /system.slice/tuned.service
           └─8061 /usr/bin/python2 -Es /usr/sbin/tuned -l -P

May 07 10:16:43 intel-skylake-loaner.virt.lab.eng.bos.redhat.com systemd[1]: Starting Dynamic System Tuning Daemon...
May 07 10:16:44 intel-skylake-loaner.virt.lab.eng.bos.redhat.com systemd[1]: Started Dynamic System Tuning Daemon.
[root@intel-skylake-loaner ~]# tuned-adm active
Current active profile: balanced
[root@intel-skylake-loaner ~]# cat /etc/tuned/realtime-virtual-host-variables.conf
# Examples:
# isolated_cores=2,4-7
# isolated_cores=2-23
isolated_cores=19-35
[root@intel-skylake-loaner ~]# cat /sys/devices/system/cpu/online
0-35
[root@intel-skylake-loaner ~]# tuned-adm profile realtime-virtual-host
Cannot load profile(s) 'realtime-virtual-host': Assertion 'isolated_cores contains online CPU(s)' failed.
[root@intel-skylake-loaner ~]# 


Non-RT Kernel:

[root@intel-skylake-loaner ~]# uname -a
Linux intel-skylake-loaner.virt.lab.eng.bos.redhat.com 3.10.0-1040.el7.x86_64 #1 SMP Thu Apr 18 15:22:56 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@intel-skylake-loaner ~]# uptime 
 10:11:50 up 0 min,  1 user,  load average: 0.70, 0.21, 0.07
[root@intel-skylake-loaner ~]# tuned -v
tuned 2.10.0
[root@intel-skylake-loaner ~]# systemctl status tuned
● tuned.service - Dynamic System Tuning Daemon
   Loaded: loaded (/usr/lib/systemd/system/tuned.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-05-07 10:11:43 EDT; 17s ago
     Docs: man:tuned(8)
           man:tuned.conf(5)
           man:tuned-adm(8)
 Main PID: 4608 (tuned)
    Tasks: 5
   CGroup: /system.slice/tuned.service
           └─4608 /usr/bin/python2 -Es /usr/sbin/tuned -l -P

May 07 10:11:41 intel-skylake-loaner.virt.lab.eng.bos.redhat.com systemd[1]: Starting Dynamic System Tuning Daemon...
May 07 10:11:43 intel-skylake-loaner.virt.lab.eng.bos.redhat.com systemd[1]: Started Dynamic System Tuning Daemon.
[root@intel-skylake-loaner ~]# tuned-adm active
Current active profile: balanced

[root@intel-skylake-loaner ~]# cat /sys/devices/system/cpu/online
0-35
[root@intel-skylake-loaner ~]# cat /etc/tuned/realtime-virtual-host-variables.conf
# Examples:
# isolated_cores=2,4-7
# isolated_cores=2-23
isolated_cores=19-35
[root@intel-skylake-loaner ~]# tuned-adm profile realtime-virtual-host
Cannot load profile(s) 'realtime-virtual-host': Assertion 'isolated_cores contains online CPU(s)' failed.
[root@intel-skylake-loaner ~]#
Comment 3 Luiz Capitulino 2019-05-07 16:04:40 UTC 
Jaroslav,

As mentioned by Nitesh in the previous comment, this issue affects 7.6.z. Since
CPU isolation is necessary for KVM-RT to work properly and since we have
customers running 7.6.z on the field on Skylake, I think it's just a matter of
time they hit this issue which is a blocker for us.
Comment 4 Ondřej Lysoněk 2019-05-07 16:17:22 UTC 
Hi, I'm working on this one, give me a second. Btw, I don't think it's a regression...
Comment 5 Jaroslav Škarvada 2019-05-07 16:19:42 UTC 
No problem with the z-stream.
Comment 6 Ondřej Lysoněk 2019-05-07 16:26:25 UTC 
Created attachment 1565282 [details]
patch

Can you check if it works with the attached patch applied to /usr/lib/python2.7/site-packages/tuned/profiles/functions/function_cpulist_online.py ?
Comment 7 Nitesh Narayan Lal 2019-05-07 16:41:39 UTC 
Yes, it worked.
Thanks.
Comment 11 Jaroslav Škarvada 2019-05-09 08:32:04 UTC 
Meged upstream PR:
https://github.com/redhat-performance/tuned/pull/183
Comment 12 Ondřej Lysoněk 2019-05-14 07:25:44 UTC 
*** Bug 1643944 has been marked as a duplicate of this bug. ***
Comment 16 errata-xmlrpc 2019-08-06 13:04:46 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2201
Note You need to log in before you can comment on or make changes to this bug.