Bug 1676962

Just noticed that same problem applies to ksoftirqd(s) (and I guess
to all changes related to priority tuning).

After reboot ksoftirqs(s) are back to SCHED_NORMAL, instead of
SCHED_FIFO with priority 2.

Hmm, I can't reproduce it as follows in a VM with a fresh base rhel-8 system. Perhaps some race condition is involved. I'll take a look again next week.
1. yum install @nfv
2. configure /etc/tuned/realtime-virtual-guest-variables.conf
3. tuned-adm profile realtime-virtual-guest
4. reboot
5. check priority of ktimersoftd/ksoftirqd
6. repeat 4. and 5. multiple times

This is happening on the host for me, so it's
realtime-virtual-host profile.

Not sure it would make any difference though.

Anyway, I'll try again on a fresh system next week as well.

(In reply to Juri Lelli from comment #3)
> This is happening on the host for me, so it's
> realtime-virtual-host profile.
> 
> Not sure it would make any difference though.

The profiles are virtually identical, so it shouldn't. Nevertheless I tried it with realtime-virtual-host on bare metal and got the same results.

To rule out the obvious possibility, are you sure that tuned is running and the correct profile is applied after reboot? (tuned-adm active). Perhaps tuned crashed or something.

(In reply to Ondřej Lysoněk from comment #4)
> (In reply to Juri Lelli from comment #3)
> > This is happening on the host for me, so it's
> > realtime-virtual-host profile.
> > 
> > Not sure it would make any difference though.
> 
> The profiles are virtually identical, so it shouldn't. Nevertheless I tried
> it with realtime-virtual-host on bare metal and got the same results.
> 
> To rule out the obvious possibility, are you sure that tuned is running and
> the correct profile is applied after reboot? (tuned-adm active). Perhaps
> tuned crashed or something.

On the system where I noticed the problem I checked again right after reboot
and can confirm realtime-virtual-host is active, but k{timersoft,softirq}d
settings are not correct.

This is what I could get from /var/log/tuned.log (notice units.manager at the
end of the log output).

--->8---
2019-02-19 08:22:10,685 INFO     tuned.daemon.application: dynamic tuning is globally disabled
2019-02-19 08:22:10,720 INFO     tuned.daemon.daemon: using sleep interval of 1 second(s)
2019-02-19 08:22:10,724 INFO     tuned.profiles.loader: loading profile: realtime-virtual-host
2019-02-19 08:22:10,745 INFO     tuned.daemon.controller: starting controller
2019-02-19 08:22:10,745 INFO     tuned.daemon.daemon: starting tuning
2019-02-19 08:22:10,771 INFO     tuned.plugins.base: instance cpu: assigning devices cpu11, cpu4, cpu0, cpu7, cpu1, cpu10, cpu2, cpu6, cpu9, cpu8, cpu5, cpu3
2019-02-19 08:22:10,848 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu11'
2019-02-19 08:22:10,896 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu4'
2019-02-19 08:22:10,934 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu0'
2019-02-19 08:22:10,974 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu7'
2019-02-19 08:22:10,998 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu1'
2019-02-19 08:22:11,015 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu10'
2019-02-19 08:22:11,031 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu2'
2019-02-19 08:22:11,050 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu6'
2019-02-19 08:22:11,080 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu9'
2019-02-19 08:22:11,128 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu8'
2019-02-19 08:22:11,168 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu5'
2019-02-19 08:22:11,242 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu3'
2019-02-19 08:22:11,242 INFO     tuned.plugins.plugin_cpu: setting new cpu latency 1
2019-02-19 08:22:11,475 ERROR    tuned.utils.commands: Executing sysctl error: sysctl: cannot stat /proc/sys/net/core/busy_read: No such file or directory
2019-02-19 08:22:11,494 ERROR    tuned.utils.commands: Executing sysctl error: sysctl: cannot stat /proc/sys/net/core/busy_poll: No such file or directory
2019-02-19 08:22:11,531 ERROR    tuned.utils.commands: Executing sysctl error: sysctl: cannot stat /proc/sys/kernel/numa_balancing: No such file or directory
2019-02-19 08:22:11,630 INFO     tuned.plugins.plugin_sysctl: reapplying system sysctl
2019-02-19 08:22:11,641 WARNING  tuned.plugins.plugin_vm: Option 'transparent_hugepages' is not supported on current hardware.
2019-02-19 08:22:11,643 INFO     tuned.plugins.plugin_bootloader: installing additional boot command line parameters to grub2
2019-02-19 08:22:11,784 INFO     tuned.plugins.plugin_script: calling script '/usr/lib/tuned/realtime/script.sh' with arguments '['start']'
2019-02-19 08:22:11,828 INFO     tuned.plugins.plugin_script: calling script '/usr/lib/tuned/realtime-virtual-host/script.sh' with arguments '['start']'
2019-02-19 08:22:12,021 ERROR    tuned.units.manager: BUG: Unhandled exception in start_tuning: dictionary changed size during iteration
2019-02-19 08:22:12,021 INFO     tuned.daemon.daemon: static tuning from profile 'realtime-virtual-host' applied
--->8---

Having said this, I'm also not able to reproduce the same on another fresh box from Beaker.
Might be related to specific system config or series of actions (the one that shows the
problem I used for some time to work on a BZ, so several kernels have been installed/removed
and quite some tuning happened as well..)?

Please let me know if I could provide more debug info from the box I'm using.

Thanks for the log, that helped. I think you've hit bug#1669294. Can you please upgrade python3-linux-procfs on the machine and test it again?

(In reply to Ondřej Lysoněk from comment #6)
> Thanks for the log, that helped. I think you've hit bug#1669294. Can you
> please upgrade python3-linux-procfs on the machine and test it again?

Ah, indeed! Upgrading python3-linux-procfs fixed the problem.

Thanks a lot.

Can we then close this one as dupe of bug 1669294?

*** This bug has been marked as a duplicate of bug 1669294 ***