Bug 1432240

Summary: tuned failed to activate profile with certain core list
Product: Red Hat Enterprise Linux 7 Reporter: jianzzha
Component: tunedAssignee: Jaroslav Škarvada <jskarvad>
Status: CLOSED ERRATA QA Contact: Tereza Cerna <tcerna>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 7.4CC: bhu, cchen, jeder, jskarvad, tcerna
Target Milestone: rcKeywords: Patch, Upstream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: tuned-2.8.0-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1434360 (view as bug list) Environment:
Last Closed: 2017-08-01 12:35:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description jianzzha 2017-03-14 21:06:08 UTC
Description of problem:
on a system with 2 processors (24 cores), tuned failed to activate  cpu-partitioning profile if the configured core-list are even cores (2,4,6,8). On the same system, if the configured core-list are odd cores (1,3,5,7), tuned was able to activate the cpu-partitioning profile.

Version-Release number of selected component 
kernel version: 3.10.0-514.2.2

How reproducible:
with odd-core list:
[root@controller-0 tuned]# cat cpu-partitioning-variables.conf
# Examples:
# isolated_cores=1,3,5,7
# isolated_cores=1,3,5,7
#
isolated_cores=1,3,5,7

[root@controller-0 tuned]# tuned-adm  profile cpu-partitioning
[root@controller-0 tuned]#

with even-core list:
[root@controller-0 tuned]# cat cpu-partitioning-variables.conf
# Examples:
# isolated_cores=2,4,6,8
# isolated_cores=2,4,6,8
#
isolated_cores=2,4,6,8

[root@controller-0 tuned]# tuned-adm  profile cpu-partitioning
Cannot load profile 'cpu-partitioning': Assertion 'isolated_cores contains present CPU(s)' failed.
[root@controller-0 tuned]#


Actual results:


Expected results:


Additional info:
[root@controller-0 tuned]# cat /proc/cpuinfo | grep processor
processor	: 0
processor	: 1
processor	: 2
processor	: 3
processor	: 4
processor	: 5
processor	: 6
processor	: 7
processor	: 8
processor	: 9
processor	: 10
processor	: 11
processor	: 12
processor	: 13
processor	: 14
processor	: 15
processor	: 16
processor	: 17
processor	: 18
processor	: 19
processor	: 20
processor	: 21
processor	: 22
processor	: 23

Comment 2 jianzzha 2017-03-14 21:41:45 UTC
further tests indicates this seems to happen when the core list across 8, 16.

for example "7,8" will have problem, while "8,9" is fine

Comment 3 Chen 2017-03-20 14:20:26 UTC
Hi team,

Sorry for raising the severity. This bug is preventing serveral huawei NFV PoC projects.

# cat /etc/tuned/cpu-partitioning-variables.conf 
# Examples:
# isolated_cores=2,4-7
# isolated_cores=2-23
#
isolated_cores=2,3,11

# tuned-adm profile cpu-partitioning
Cannot load profile 'cpu-partitioning': Assertion 'isolated_cores contains present CPU(s)' failed.

Snip of /var/log/tuned/tuned.log:

2017-03-20 10:15:18,321 INFO     tuned.profiles.loader: loading profile: cpu-partitioning
2017-03-20 10:15:18,323 ERROR    tuned.profiles.functions.function_assertion: assertion 'isolated_cores contains present CPU(s)' failed: '2,3,11' != '3,2,11'

<Snip>

2017-03-20 10:15:18,897 INFO     tuned.daemon.daemon: static tuning from profile 'cpu-partitioning' applied

By looking at the logs seems the profile has been applied. So is this a harmless warning or a real problem ?

Best Regards,
Chen

Comment 4 Jaroslav Škarvada 2017-03-21 08:43:40 UTC
I think the following upstream commit should resolve the problem:
https://github.com/redhat-performance/tuned/commit/e052fd14a17e7872d0880720b15db2097b58f5a4

Comment 5 Chen 2017-03-21 08:48:05 UTC
Hi Jaroslav,

Thank you for your reply.

I've confirmed that the upstream patch works.

Do you have any idea of the following question ?

>By looking at the logs seems the profile has been applied. So is this a harmless warning or a real problem ?

Best Regards,
Chen

Comment 6 Jaroslav Škarvada 2017-03-21 09:30:29 UTC
(In reply to Chen from comment #5)
> Hi Jaroslav,
> 
> Thank you for your reply.
> 
> I've confirmed that the upstream patch works.
> 
> Do you have any idea of the following question ?
> 
> >By looking at the logs seems the profile has been applied. So is this a harmless warning or a real problem ?

After the assertion kicks in it should stop loading the profile, but from your log it doesn't seem so. I was unable to reproduce the problem, but it could be different (probably minor) bug or maybe it's already fixed upstream.

What's the Tuned version used?
tuned-2.7.1-5.el7 from FDP?

In such case there is high probability that the problem is already fixed.

The latest scratch-build (unsupported) version for testing is:
tuned*-2.7.1-1.20170321gita50d5e02.el7

available from:
https://jskarvad.fedorapeople.org/tuned/devel/repo/

Or could you lend me the machine for closer observation what's going there?

Comment 9 Jaroslav Škarvada 2017-03-21 11:04:00 UTC
Clonned as bug 1434360.

Comment 12 Tereza Cerna 2017-05-04 14:03:04 UTC
Works as expected. Loading of profile doesn't stop with assertion.
 -> VERIFIED

======================================================
Verified in:
    tuned-2.8.0-2.el7.noarch
    tuned-profiles-cpu-partitioning-2.8.0-2.el7.noarch
PASS
======================================================

# tuned-adm profile balanced
# cat /etc/tuned/cpu-partitioning-variables.conf
isolated_cores=2,3,11
# tuned-adm profile cpu-partitioning
# tuned-adm active
Current active profile: cpu-partitioning
# reboot

# tuned-adm verify
Verfication succeeded, current system settings match the preset profile.
See tuned log file ('/var/log/tuned/tuned.log') for details.

======================================================
Reproduced in:
    tuned-2.7.1-5.20170314git92d558b8.el7.noarch
    tuned-profiles-cpu-partitioning-2.7.1-5.20170314git92d558b8.el7.noarch
FAIL
======================================================

# tuned-adm profile balanced
# cat /etc/tuned/cpu-partitioning-variables.conf 
isolated_cores=2,3,11
# tuned-adm profile cpu-partitioning
Cannot load profile 'cpu-partitioning': Assertion 'isolated_cores contains present CPU(s)' failed.
# tuned-adm active
Current active profile: balanced
# cat /var/log/tuned/tuned.log | grep ERROR
2017-05-04 09:56:12,644 ERROR    tuned.profiles.functions.function_assertion: assertion 'isolated_cores contains present CPU(s)' failed: '2,3,11' != '3,2,11'

Comment 13 errata-xmlrpc 2017-08-01 12:35:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2102

Comment 14 jianzzha 2018-10-25 17:23:58 UTC
In stead of having an assertion error, 
https://github.com/redhat-performance/tuned/blob/master/tuned/profiles/functions/function_assertion.py
line 22,

if we change the if statement on to:
  if ''.join(sorted(args[1].split(','))) != ''.join(sorted(args[2].split(','))):

that shall take care of the list ordering issue