Bug 1896348

Summary:	[RFE] Allow setting isolated_cores without touching the default irq affinity
Product:	Red Hat Enterprise Linux 8	Reporter:	Marcel Apfelbaum <mapfelba>
Component:	tuned	Assignee:	Jaroslav Škarvada <jskarvad>
Status:	CLOSED ERRATA	QA Contact:	Robin Hack <rhack>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	8.3	CC:	fromani, jeder, jmencak, jskarvad, rhack, yjoseph
Target Milestone:	rc	Keywords:	FutureFeature, Patch, TestCaseProvided, Triaged, Upstream
Target Release:	8.0	Flags:	pm-rhel: mirror+
Hardware:	Unspecified
OS:	Unspecified
URL:	https://github.com/redhat-performance/tuned/pull/306
Whiteboard:
Fixed In Version:	tuned-2.15.0-0.1.rc1.el8	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-05-18 15:05:25 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Marcel Apfelbaum 2020-11-10 10:37:58 UTC

Description of problem:
-----------------------
Setting the "isolated_cores" option will automatically mask the isolated cores  out from the default smp irq affinity (/proc/irq/default_smp_affinity)

However, sometimes there is a need to leave irqs enabled on isolated cores

Expected behavior:
------------------
Differentiate between "isolated_cores" set and the "default_smp_irq_affinity".

It can be done by adding  a new  option "default_irq_affinity" that will translate into setting the /proc/irq/default_smp_affinity.

The relationship between "isolate_cores" and "default_smp_irq_affinity" could be:

"isolate_cores" - will work as today if default_smp_irq_affinity is not set, otherwise will not "touch" the default smp irq affinity.

"default_smp_irq_affinity" - will work independently from other options and will simply set /proc/irq/default_smp_affinity 


Another interesting option is to use "banned_cpus" to achieve the same.
If an user sets "banned_cpus" in their profile, as an empty list or an actual set of CPUs, tuned may infer the default_smp_irq_affinity as the inverted mask instead of setting it to the isolate_cores list.


Additional info:
----------------
The requirement comes from OCP moving towards dynamic "Real Time" settings.
Sometimes the reserved CPUs are not enough to handle networking IRQs, in this case some of the isolated CPUs are needed to join the effort.

The way it is done is by annotating the pods that have stricter RT requirements and use irqbalance tool to remove the pod's CPUs from the list of the CPUs are allowed to handle IRQs, while the other pods will allow the IRQs to be handled by their CPUs. (all the mentioned pods run using the isolated cpus set)

However having the default_smp_affinity set to the isolated_cpus messes up the plan, since in this case banning the CPUs will not make sense.

Comment 1 Marcel Apfelbaum 2020-11-10 10:50:14 UTC

In "Additional info" I meant having default_smp_affinity set to the inverted mask of isolated_cpus messes up the plan.

Comment 2 Jaroslav Škarvada 2020-11-10 15:10:49 UTC

This is change of behavior. We need to keep backward compatibility.

We could:
a) add 'default_smp_irq_affinity' option and set it by default to the special value like e.g. 'calc_from_isolated_cores'. In such case it will work the same way. Other value will be taken as the cpuset specification, or

b) add 'set_default_smp_irq_affinity' option and set it by default to '1', if set to '0' TuneD will not touch the default smp irq affinity.

Comment 3 Jaroslav Škarvada 2020-11-10 15:14:31 UTC

(In reply to Jaroslav Škarvada from comment #2)
> This is change of behavior. We need to keep backward compatibility.
> 
> We could:
> a) add 'default_smp_irq_affinity' option and set it by default to the
> special value like e.g. 'calc_from_isolated_cores'. In such case it will
> work the same way. Other value will be taken as the cpuset specification, or
> 
Or use special values like 'calc' to calcalate and 'ignore' to ignore.

Comment 4 Marcel Apfelbaum 2020-11-10 15:41:26 UTC

(In reply to Jaroslav Škarvada from comment #2)
> This is change of behavior. We need to keep backward compatibility.
> 
> We could:
> a) add 'default_smp_irq_affinity' option and set it by default to the
> special value like e.g. 'calc_from_isolated_cores'. In such case it will
> work the same way. Other value will be taken as the cpuset specification,

I like this better since it will allow us set a value for default_smp_irq_affinity
from the profile (in the future)

> or
> b) add 'set_default_smp_irq_affinity' option and set it by default to '1',
> if set to '0' TuneD will not touch the default smp irq affinity.

This also would be enough, however the above adds more functionality.

Thank you,
Marcel

Comment 5 Jaroslav Škarvada 2020-11-10 18:18:44 UTC

Upstream PR:
https://github.com/redhat-performance/tuned/pull/306

Comment 16 errata-xmlrpc 2021-05-18 15:05:25 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (tuned bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1671