Bug 1259039

OK, I know what's happening. Here's a simple reproducer:

1. If your active profile is realtime-virtual-host, change to another profile

  # tuned-adm active
  Current active profile: realtime-virtual-host
  # tuned-adm profile virtual-host

2. Check tuned_params in grub.cfg

  # grep tuned_params= /boot/grub2/grub.cfg
  set tuned_params=""

3. Activate the realtime-virtual-host profile and confirm it succeeded

 # tuned-adm profile realtime-virtual-host
 # echo $?
 0

4. Check tuned_params in grub.cfg

  # grep tuned_params= /boot/grub2/grub.cfg
  set tuned_params=""

If you reboot the machine at this point, you end up with no kernel command-line options. This happened to me twice already. If you wait some minutes (2 minutes on the machine I'm testing), grub.cfg will eventually be updated. Also, after you reboot grub.cfg will eventually be updated too.

I believe this is so because of run-tscdeadline-latency.sh, which takes a lot to run.

Can we lower the number of iterations the script do or speed it up by different way?

(In reply to Jaroslav Škarvada from comment #3)
> Can we lower the number of iterations the script do or speed it up by
> different way?

Its similar to a calibration process, so lets assume we could reduce it from 2 minutes to 30 seconds. 
The problem would still exist, even if the time was reduced, right? 

So don't see why reducing the runtime of run-tscdeadline-latency.sh gets rid of the problem.

Jarda,

Although this issue only happens with tuned-profiles-nfv right now, I'm under the impression that this is a more general design issue with tuned.

Is tuned designed for profiles which may block or take a long time to finish?

Marcelo can answer if run-tscdeadline-latency,sh can finish sooner or whether we could try to come up with some trick, but the issue at hand is that we have a profile which may need minutes to finish. Is this allowed at all in tuned? If not, why not?

(In reply to Luiz Capitulino from comment #6)
> Jarda,
> 
> Although this issue only happens with tuned-profiles-nfv right now, I'm
> under the impression that this is a more general design issue with tuned.
> 
> Is tuned designed for profiles which may block or take a long time to finish?
> 
IMHO it shouldn't be problem. The only limit which could affect it is IIRC the 3 minutes (or so) systemd service timeout, but IMHO it shouldn't be in effect for profile loading (I will check it).

Back to comment 1, there is written that the problem occurre if the machine is rebooted during the calibration. I am afraid we cannot do much about it - attempts to block or postpone reboots/shut downs are bad and usually ineffective (there is always power switch somewhere which can be cut). Is there another reproducer?

(In reply to Jaroslav Škarvada from comment #7)
> (In reply to Luiz Capitulino from comment #6)
> > Jarda,
> > 
> > Although this issue only happens with tuned-profiles-nfv right now, I'm
> > under the impression that this is a more general design issue with tuned.
> > 
> > Is tuned designed for profiles which may block or take a long time to finish?
> > 
> IMHO it shouldn't be problem. The only limit which could affect it is IIRC
> the 3 minutes (or so) systemd service timeout, but IMHO it shouldn't be in
> effect for profile loading (I will check it).
> 
> Back to comment 1, there is written that the problem occurre if the machine
> is rebooted during the calibration. I am afraid we cannot do much about it -
> attempts to block or postpone reboots/shut downs are bad and usually
> ineffective (there is always power switch somewhere which can be cut). Is
> there another reproducer?

Maybe we could re-run the calibration on the next tuned start if the first calibration failed or was interrupted. I will check.

That could be helpful. But IMHO the problem is that tuned-adm profile returns success while the profile configuration is still taking place. If instead of returning success it printed a warning to the user saying that this profile takes longer than usual to finish and then waited for it to finish this would solve the problem.

In Comment 9, Luiz writes:
>IMHO the problem is that tuned-adm profile returns success
>while the profile configuration is still taking place. 

Indeed, this is the fundamental flaw. Why? Because the profile configuration is going to alter some configuration files. This must be allowed to complete or the setup is not finished, but it appears to the user that it was successful.

Consider that the user may also want to make changes to the same files that tuned will modify. There is a race-condition here and a possible loss of data if one overwrites the settings made by the other.

The user really needs to know when it's safe to proceed.

In Comment 8, Jaroslav writes:
>Maybe we could re-run the calibration on the next tuned start
>if the first calibration failed or was interrupted.

This would help. I encourage you to inform the user that this happened.

I think this is resolved by sync mode, which is by default on, i.e. 'tuned-adm profile' doesn't return until the profile is fully applied.

There may be a problem, when the profile is autodetected and enabled upon tuned installation and then the machine is quickly rebooted. IMHO Tuned now waits till the configuration is finished and then shutdowns. Well, it may be still hard killed by the systemd after 90 seconds timeout, but it's better than nothing.

Even in such case no much harm should be done - the profile is simply re-applied and the script re-run after the reboot.

For profiles with long running initialization we should fix their script code to re-run the initilization if the previous initialization wasn't completed. I think it's the best what we can do.

(In reply to Jaroslav Škarvada from comment #11)
> I think this is resolved by sync mode, which is by default on, i.e.
> 'tuned-adm profile' doesn't return until the profile is fully applied.
> 
> There may be a problem, when the profile is autodetected and enabled upon
> tuned installation and then the machine is quickly rebooted. IMHO Tuned now
> waits till the configuration is finished and then shutdowns. Well, it may be
> still hard killed by the systemd after 90 seconds timeout, but it's better
> than nothing.
> 
> Even in such case no much harm should be done - the profile is simply
> re-applied and the script re-run after the reboot.
> 
> For profiles with long running initialization we should fix their script
> code to re-run the initilization if the previous initialization wasn't
> completed. I think it's the best what we can do.

Regarding the NFV profiles scripts it seems all the relevant "helper" files are recreated if they don't exist. But I did only quick code check.

Jaroslav,

Just to make sure I understood correctly, you backported sync mode to 7.3 (or maybe did a full rebase upstream) and it fixes this issue?

(In reply to Luiz Capitulino from comment #15)
> Jaroslav,
> 
> Just to make sure I understood correctly, you backported sync mode to 7.3
> (or maybe did a full rebase upstream) and it fixes this issue?

The 7.3 is receiving the latest tuned-2.7.0 with the sync mode enabled by default. I wasn't able to reproduce the problem, but there still may be some different problem. Feel free to test and let me know.

Close as SanityOnly bug.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2479.html