Bug 1394965

After my own mini-research and discussion with systemd guys it seems the only way how to resolve this is to update the initrd.

Jaroslav,

Updating the initrd image works, but it's not doable in practice because we have to update the initrd image for all installed kernels and that takes too long (and actually breaks tuned).

A much better solution is to the the following:

1. Implement initrd image layering in tuned as described in bug 1414098
2. Use image layering to pull in only the /etc/systemd/system.conf file in the initrd image

(In reply to Luiz Capitulino from comment #2)
> Jaroslav,
> 
> Updating the initrd image works, but it's not doable in practice because we
> have to update the initrd image for all installed kernels and that takes too
> long (and actually breaks tuned).
> 
> A much better solution is to the the following:
> 
> 1. Implement initrd image layering in tuned as described in bug 1414098
> 2. Use image layering to pull in only the /etc/systemd/system.conf file in
> the initrd image

OK working on it, I tried to point out that it cannot be fixed in systemd itself or addressed as systemd RFE. Systemd is trying to update the affinity for everything (except specially prefixed processes), but for timers, it technically cannot do it.

Oh, OK. That's correct. CPUAffinity will only set the affinity of systemd itself (and by extension, all processes it starts).

(In reply to Jaroslav Škarvada from comment #3)
> (In reply to Luiz Capitulino from comment #2)
> > Jaroslav,
> > 
> > Updating the initrd image works, but it's not doable in practice because we
> > have to update the initrd image for all installed kernels and that takes too
> > long (and actually breaks tuned).
> > 
> > A much better solution is to the the following:
> > 
> > 1. Implement initrd image layering in tuned as described in bug 1414098
> > 2. Use image layering to pull in only the /etc/systemd/system.conf file in
> > the initrd image
> 
> OK working on it, I tried to point out that it cannot be fixed in systemd
> itself or addressed as systemd RFE. Systemd is trying to update the affinity
> for everything (except specially prefixed processes), but for timers, it
> technically cannot do it.

I.e. the current systemd behaviour is to re-set the CPU affinity from filesystem system.conf to takeover initrd system.conf, but for timers it cannot do it.

After another discussion with systemd guys I think we came to the best solution, to propose kernel boot command line parameter (e.g. CPUAffinity) which would be understood by systemd and overrides /etc/system/system.conf settings. Upstream RFE:
https://github.com/systemd/systemd/issues/5368

I think this would resolve this problem very clean way.

I understand you need the fix quickly and the systemd fix could take some time, so I am also going to introduce the initrd workaround in Tuned.

Yes, the kernel command-line parameter solves this BZ. But note that we also have bug 1395899, which requires initrd support. So, we end having to implement bug 1414098 anyways.

It's preferred to solve this by the systemd RFE (comment 6) which has been accepted upstream. Until it's finished we can use the following workaround (relying on the functionality introduced by RFE bug 1414098 and RFE bug 1246172):

Profile:
[systemd]
cpu_affinity=YOUR_AFFINITY

[script]
priority=5
script=script.sh

[bootloader]
priority=10
initrd_add_dir=tuned-overlay.img

# mkdir -p tuned-overlay.img/etc/systemd
# cat script.sh
#!/bin/bash

. /usr/lib/tuned/functions

start() {
    cp /etc/systemd/system.conf tuned-overlay.img/etc/systemd/
    return 0
}

stop() {
    return 0
}

process $@

----
I.e. it introduces priority=5 to the script which guarantees that it will run after the [systemd] plugin sets the affinity in /etc/systemd/system.conf, than the [script] copy the systemd settings to the overlay and finally the [bootloader] creates the initrd and installs it as a overlay.

Regarding problem related to the workqueue-cpumask I will comment to the bug 1395899.

OK, I tried the patch at the bottom but tuned doesn't create the overlay image. It does add the following to grub:

set tuned_initrd=""

But as we can see, it's empty.

Also, in /var/log/tuned/tuned.log there is:

"""
2017-03-01 13:32:22,591 INFO     tuned.plugins.plugin_script: calling script '/usr/lib/tuned/cpu-partitioning/script.sh' with arguments '['start']'
2017-03-01 13:32:22,686 INFO     tuned.plugins.plugin_bootloader: installing additional boot command line parameters to grub2
2017-03-01 13:32:22,686 ERROR    tuned.plugins.plugin_bootloader: error: cannot create initrd image, source directory '/usr/lib/tuned/latency-performance/tuned-overlay.img' doesn't exist
2017-03-01 13:32:22,690 INFO     tuned.daemon.daemon: static tuning from profile 'cpu-partitioning' applied
"""

So, it looks like tuned is confusing cpu-partitioning with latency-performance?

diff -Nparu cpu-partitioning.orig/script.sh cpu-partitioning/script.sh
--- cpu-partitioning.orig/script.sh     2017-01-10 07:56:20.000000000 -0500
+++ cpu-partitioning/script.sh  2017-03-01 13:29:10.826909265 -0500
@@ -8,6 +8,7 @@ start() {
     sed -i '/^IRQBALANCE_BANNED_CPUS=/d' /etc/sysconfig/irqbalance
     echo "IRQBALANCE_BANNED_CPUS=$TUNED_isolated_cpumask" >>/etc/sysconfig/irqbalance
     setup_kvm_mod_low_latency
+    cp /etc/systemd/system.conf tuned-overlay.img/etc/systemd/
     return "$?"
 }

diff -Nparu cpu-partitioning.orig/tuned.conf cpu-partitioning/tuned.conf
--- cpu-partitioning.orig/tuned.conf    2017-01-10 07:44:28.000000000 -0500
+++ cpu-partitioning/tuned.conf 2017-03-01 13:28:04.201401503 -0500
@@ -31,10 +31,13 @@ kernel.timer_migration = 1
 /sys/devices/system/machinecheck/machinecheck*/ignore_ce = 1

 [script]
+priority=5
 script=script.sh

 [systemd]
 cpu_affinity=${not_isolated_cores_expanded}

 [bootloader]
+priority=10
+initrd_add_dir=tuned-overlay.img
 cmdline= nohz=on nohz_full=${isolated_cores} rcu_nocbs=${isolated_cores} intel_pstate=disable nosoftlockup

(In reply to Luiz Capitulino from comment #9)

Thanks for info and let me check it. Our tests for this feature ran only from /etc, but IMHO it shouldn't be hard to fix this problem.

Upstream related fix:
https://github.com/redhat-performance/tuned/commit/8cef7cdb21264c5030c5f2cf5609935819a28de5

Let's try:
tuned-2.7.1-1.20170302git00746dc6.el7

This almost worked. The only problem now is that the /etc/systemd/system.conf was copied into the initrd image before it was modified to contain the CPUAffinity setting. This gives the impression that the "priority" lines are not working as expected (that is, "[bootloader]" ran first than "[systemd]").

PS: I wasn't able to find tuned-2.7.1-1.20170302git00746dc6.el7 in your repo, so I just built tuned from the git repo.

A small problem I just found out: when I switch to another profile, grub.cfg is updated correctly (that is, tuned_initrd is set to empty). However, the /boot/tuned-overlay.img file is still there. I think it should be removed.

(In reply to Luiz Capitulino from comment #13)
> A small problem I just found out: when I switch to another profile, grub.cfg
> is updated correctly (that is, tuned_initrd is set to empty). However, the
> /boot/tuned-overlay.img file is still there. I think it should be removed.

It could remove it, but I left it there, because it's harmless and not to overwrite something which may be needed later. It will be overwritten again, when the profile is activated.

IMO, this is the profile leaking data. If different profiles use different initrd images, then we'll have several stale files in /boot. And /boot is not /tmp :) I think tuned profiles should always undo what they have done when they are turned off or when there's a profile switch. This includes cleaning up stale files (and I'll also update my patch to delete tuned-overlay.img/etc/systemd/system.conf).

(In reply to Luiz Capitulino from comment #15)
> IMO, this is the profile leaking data. If different profiles use different
> initrd images, then we'll have several stale files in /boot. And /boot is
> not /tmp :) I think tuned profiles should always undo what they have done
> when they are turned off or when there's a profile switch. This includes
> cleaning up stale files (and I'll also update my patch to delete
> tuned-overlay.img/etc/systemd/system.conf).

Valid point, I will remove them. I wanted to track and remove them in rpms.

(In reply to Jaroslav Škarvada from comment #16)
> (In reply to Luiz Capitulino from comment #15)
> > IMO, this is the profile leaking data. If different profiles use different
> > initrd images, then we'll have several stale files in /boot. And /boot is
> > not /tmp :) I think tuned profiles should always undo what they have done
> > when they are turned off or when there's a profile switch. This includes
> > cleaning up stale files (and I'll also update my patch to delete
> > tuned-overlay.img/etc/systemd/system.conf).
> 
> Valid point, I will remove them. I wanted to track and remove them in rpms.

Upstream commit:
https://github.com/redhat-performance/tuned/commit/d89ab6731b41c9beac2f11e9588916f2d0397e93

Cool! So, I take it you're still looking at the priority problem from comment 12?

(In reply to Luiz Capitulino from comment #18)
> Cool! So, I take it you're still looking at the priority problem from
> comment 12?

Trying to reproduce, sorry for delay.

No problem! Just wanted to know we were on the same page.

Hmm, I am still unable to reproduce:

script.sh:
#!/bin/sh

. /usr/lib/tuned/functions

start() {
    cp /etc/systemd/system.conf ./tuned-initrd.img/etc/systemd/
    return 0
}

stop() {
    return 0
}

process $@

---

tuned.conf:
[systemd]
cpu_affinity = 2

[script]
priority = 5
script = script.sh

[bootloader]
priority = 10
initrd_add_dir = tuned-initrd.img

---

Tuned run...

# lsinitrd -f /etc/systemd/system.conf /boot/tuned-initrd.img | grep CPUAffinity
#CPUAffinity=1 2
CPUAffinity=2

I.e. it works as expected, which is confirmed by Tuned log:
2017-03-03 16:27:56,200 INFO     tuned.plugins.plugin_systemd: setting 'CPUAffinity' to '2' in the '/etc/systemd/system.conf'
2017-03-03 16:27:56,201 INFO     tuned.plugins.plugin_script: calling script '/etc/tuned/test2/script.sh' with arguments '['start']'
2017-03-03 16:27:56,215 INFO     tuned.plugins.plugin_bootloader: generating initrd image from directory '/etc/tuned/test2/tuned-initrd.img'
2017-03-03 16:27:56,229 INFO     tuned.plugins.plugin_bootloader: installing initrd image as '/boot/tuned-initrd.img'

I.e. it patches system.conf first, then it runs the helper script and finally it generates and install initrd.

Could you provide snip from your tuned.log?

I have rebooted into a different profile a few times to do other work on that machine, so I'll have to reproduce again.

However, I did not set the priority for the "[script]" section so maybe it's running first? I'll fix that. But I'm taking the day off today, so this will have to wait until next week.

Thanks for the feedback.

(In reply to Luiz Capitulino from comment #22)
> However, I did not set the priority for the "[script]" section so maybe it's
> running first? I'll fix that.

Yes, it's important. If no 'priority' is specified the plugins are executed in order which depends on python implementation and is unrelated to their position in the tuned.conf. On my Fedora it executes in correct order even without the priority, but it's pure coincidence.

> But I'm taking the day off today, so this will
> have to wait until next week.
> 
OK, NP, let me know next week.

Improved by following upstream commit:
https://github.com/redhat-performance/tuned/commit/4eadc691fc3ae6f300087ccef893e5f5031253a2

This adds 'initrd_remove_dir' option which if set to 'True' (is false by default) removes the source directory from which the initrd image is build, e.g.:

[bootloader]
initrd_remove_dir = True
initrd_add_dir = /tmp/tuned-initrd.img

This will create initrd image from the /tmp/tuned-initrd.img directory and
then it removes the tuned-initrd.img directory from the /tmp.

This is an commit putting it all together in cpu-partitioning profile:
https://github.com/redhat-performance/tuned/commit/843dc8cf5f3a9ea176e13f6d4086e909c5fbde0e

Available in:
tuned-2.7.1-1.20170303git843dc8cf.el7

This is working as expected. I can see that the initrd contains the correct system.conf file. Also, initial zero-loss test-case passed. I'm now running a long duration test.

My tests passed, very well done!

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2102