Bug 1403309

Is it also request for adding "*pmd*,*PMD*,^DPDK" into cpu-partitioning Tuned profile whitelist? Or to different profile? Or no default whitelist, just request for the whitelisting feature?

Currently Tuned use Tuna for the cores isolation and unfortunately Tuna doesn't have whitelisting functionality. For the 7.4 I am trying to workaround it by multiple Tuna calls and some scripting-fu. For the 7.5 we should probably introduce Tuned plugin for cores isolation.

Is the following workaround OK?:

- snapshot CPU affinity of processes
- isolate cores by:
# tuna -C isolated_cores -i  # now all processes moved outside of isolated cores
- revert the affinity for whitelisted processes

Is it OK? Or is it unacceptable to touch the whitelisted processes (e.g. due to performance related reasons)?

If the presented workaround is OK, I will be able to script it with Tuna. If not, I will probably have to write the Tuned plugin.

In case you go for the regex approach, please also add the qemu-kvm process since it will impact customer workloads.

(In reply to fiezzi from comment #9)
> In case you go for the regex approach, please also add the qemu-kvm process
> since it will impact customer workloads.

What's the right regex for it? '.*qemu-kvm.*'? just guessing :)

(In reply to Jaroslav Škarvada from comment #8)
> Is the following workaround OK?:
> 
> - snapshot CPU affinity of processes
> - isolate cores by:
> # tuna -C isolated_cores -i  # now all processes moved outside of isolated
> cores
> - revert the affinity for whitelisted processes
> 
> Is it OK? Or is it unacceptable to touch the whitelisted processes (e.g. due
> to performance related reasons)?
> 
> If the presented workaround is OK, I will be able to script it with Tuna. If
> not, I will probably have to write the Tuned plugin.

TLDR the workaround consists of moving all processes out of the isolated cores and then returning the whitelisted processes back (i.e. setting back their previous affinity).

(In reply to Jaroslav Škarvada from comment #10)
> (In reply to fiezzi from comment #9)
> > In case you go for the regex approach, please also add the qemu-kvm process
> > since it will impact customer workloads.
> 
> What's the right regex for it? '.*qemu-kvm.*'? just guessing :)

Correct.

About the workaround, someone should check what would happen to move the PMD threads on a different NUMA node. E.g. the NUMA1 has the NIC and the PMD threads as well. Due to tuned the PMDs got relocated on NUMA0. I believe OVS-DPDK, at least, will complain in the logs.
Someone from NFV team should check this behavior.

Instead of hacking I implemented the feature into scheduler plugin:

Plugin scheduler now can do cores isolation on its own, Tuna is not needed for it, e.g.:

[scheduler]
isolated_cores=2-4
ps_blacklist=.*pmd.*;.*PMD.*;^DPDK;.*qemu-kvm.*

It will isolate cores 2-4, it will ignore processes which matched ps_blacklist regexes. Multiple regexes can be separated by ';'. Quoted semicolon, i.e. '\;', is taken literally.

It also supports 'ps_whitelist' which is by default set to '.*'.

It takes all processes which matches ps_whitelist than it removes those which matches ps_blacklist and move them out of the isolated_cores. When the profile is unloaded it allows all matching processes to run on all cores.

It changes processes affinities, threads affinities, IRQs affinities
and it sets default_smp_affinity for IRQs.

Also cpu-partitioning profile has been switched from Tuna to this mechanism.

Upstream commit adding this feature:
https://github.com/redhat-performance/tuned/commit/ac78f90c773cc97573844f521c2f67291f15d354

Available for testing as tuned-2.7.1-1.20170407gitac78f90c.el7 from:
https://jskarvad.fedorapeople.org/tuned/devel/repo/

I received bug report which may or may not be related to this new feature (I wasn't able to reproduce the problem myself). The truth is that this feature is very new and it would be good to give it some time to stabilize.

So the question is, could we do the following for 7.4?

- keep the code / support in RHEL-7.4 Tuned, so everybody who is interested could test it
- for safety do not use it in cpu-partitioning profile by default yet, switch back to Tuna for core isolation
- file new bug for RHEL-7.5 requesting switch from Tuna to this new code in cpu-partitioning profile

It turned out to be python-linux-procfs, workaround for it is already in Tuned upstream git.

That plan looks great to me.

Upstream commit making 'tuna' the default for 7.4 in cpu-partitioning, instructions how to enable Tuned for cores isolation with process blacklisting support (it was previously wrongly called whitelisting) are provided in the tuned.conf:
https://github.com/redhat-performance/tuned/commit/8cde3e1b3c9103f0b3e46175aff671f85cd8dcc1

Cloning the BZ for 7.5 to make Tuned the default for cores isolation in cpu-partitioning profile.

(In reply to Jaroslav Škarvada from comment #19)
> Upstream commit making 'tuna' the default for 7.4 in cpu-partitioning,
> instructions how to enable Tuned for cores isolation with process
> blacklisting support (it was previously wrongly called whitelisting) are
> provided in the tuned.conf:
> https://github.com/redhat-performance/tuned/commit/
> 8cde3e1b3c9103f0b3e46175aff671f85cd8dcc1
> 
> Cloning the BZ for 7.5 to make Tuned the default for cores isolation in
> cpu-partitioning profile.

Bug 1442230.

(In reply to Jaroslav Škarvada from comment #6)
> Is it also request for adding "*pmd*,*PMD*,^DPDK" into cpu-partitioning
> Tuned profile whitelist?
Yes

> Or to different profile? Or no default whitelist,
> just request for the whitelisting feature?
Ideally this feature could be used in any profile using CPU pinning, as this feature permit not to re-pin the processes matching the regex.

(In reply to Franck Baudin from comment #21)
> (In reply to Jaroslav Škarvada from comment #6)
> > Is it also request for adding "*pmd*,*PMD*,^DPDK" into cpu-partitioning
> > Tuned profile whitelist?
> Yes
>
Tracking in bug 1442230.

> > Or to different profile? Or no default whitelist,
> > just request for the whitelisting feature?
> Ideally this feature could be used in any profile using CPU pinning, as this
> feature permit not to re-pin the processes matching the regex.

Understood. The core functionality is already in tuned-2.8.0, but it's disabled by default in Tuned profiles. We are going to enable it by default in RHEL-7.5.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2102