(In reply to Jaroslav Škarvada from comment #0)
> Steps to Reproduce:
> 1. from systemd service which is shutting down, execute 'systemctl
> is-system-running'

Do you mean "service which is being stopped"? Or does it only happen if the service is being stopped _at shutdown_?

> Actual results:
> Exit code 15

That's weird. If I understand the code correctly `systemctl is-system-running` cannot return anything except 0 and 1...

(In reply to David Tardon from comment #1)
> > Actual results:
> > Exit code 15
> 
> That's weird. If I understand the code correctly `systemctl
> is-system-running` cannot return anything except 0 and 1...

Maybe this is not an exit code but a signal number, i.e., the systemctl process has been terminated by SIGTERM (by systemd, as it cleans up tuned.service's cgroup). That would make sense and it would also explain why it only happens sometimes.

(In reply to David Tardon from comment #1)
> (In reply to Jaroslav Škarvada from comment #0)
> > Steps to Reproduce:
> > 1. from systemd service which is shutting down, execute 'systemctl
> > is-system-running'
> 
> Do you mean "service which is being stopped"? Or does it only happen if the
> service is being stopped _at shutdown_?
> 
Yes, the service which is being stopped, I haven't tried it on machine shutdown.

(In reply to David Tardon from comment #2)
> (In reply to David Tardon from comment #1)
> > > Actual results:
> > > Exit code 15
> > 
> > That's weird. If I understand the code correctly `systemctl
> > is-system-running` cannot return anything except 0 and 1...
> 
> Maybe this is not an exit code but a signal number, i.e., the systemctl
> process has been terminated by SIGTERM (by systemd, as it cleans up
> tuned.service's cgroup). That would make sense and it would also explain why
> it only happens sometimes.

Yes, you are right. Sorry for the noise. From the python doc for the popen returncode [1]:
> A negative value -N indicates that the child was terminated by signal N (POSIX only).

It receives -15 literally, which means SIGTERM. I will update the TuneD code to cope with it.

The remaining question is why is the TuneD service receiving the second SIGTERM? I.e. the first SIGTERM triggers the shutdown code on the line 181 of daemon.py. It's quite quick, no more than one or two seconds. But why it receives another SIGTERM while in the 'popen/exec' call trying to run 'systemctl is-system-running'? I wasn't able to reproduce this on RHEL-8.7, but on RHEL-9.1 it happens quite often.

[1] https://docs.python.org/3/library/subprocess.html#subprocess.Popen.returncode

(In reply to Jaroslav Škarvada from comment #4)
> The remaining question is why is the TuneD service receiving the second
> SIGTERM?

That SIGTERM is received not by the TuneD service, but by the forked systemctl. systemd sends the signal to _all_ processes in the cgroup, even to newly forked ones.

> I wasn't able to reproduce this on RHEL-8.7,
> but on RHEL-9.1 it happens quite often.

I guess it's a difference between cgroups v1 and v2.

(In reply to David Tardon from comment #5)
> (In reply to Jaroslav Škarvada from comment #4)
> > The remaining question is why is the TuneD service receiving the second
> > SIGTERM?
> 
> That SIGTERM is received not by the TuneD service, but by the forked
> systemctl. systemd sends the signal to _all_ processes in the cgroup, even
> to newly forked ones.
> 
> > I wasn't able to reproduce this on RHEL-8.7,
> > but on RHEL-9.1 it happens quite often.
> 
> I guess it's a difference between cgroups v1 and v2.

Thanks for info, it now makes sense.

Maybe one more question, how to reliably detect whether the TuneD service is shut down by systemctl or by system shutdown/reboot? In the latter case we need to do full rollback in TuneD. I guess that in this case just calling the 'systemctl' command again from the TuneD wouldn't help.

I also don't understand why it's reproducible sometimes.

We need to handle it somehow, thus reopening and moving to TuneD.

(In reply to Jaroslav Škarvada from comment #7)
> Maybe one more question, how to reliably detect whether the TuneD service is
> shut down by systemctl or by system shutdown/reboot? 

The way you already do it. If the system is shutting down, `systemctl is-system-running` returns "stopping" (this is also available via D-Bus as org.freedesktop.systemd1.Manager#SystemState property on /org/freedesktop/systemd1).

> I also don't understand why it's reproducible sometimes.

Well, there's a race there between systemd reacting to a new process in the cgroup and the process finishing.

I can think of several ways to avoid the issue:
1. Use D-Bus API instead of calling systemctl.
2. Implement some stop signalling mechanism and use it in ExecStop=. Note that this must be synchronous, i.e., the ExecStop= command must wait till the daemon finishes stopping, as the killing of the cgroup processes will start just after the ExecStop= command finishes. (This method is discouraged in general, but it might be a suitable choice here.)
3. If the rollback can be done independently of the tuned daemon (i.e., the states/changes are somehow serialized on disk), do it from a separate script in ExecStopPost=. (This does have the additional advantage that the rollback will be done even if tuned crashes or is killed.)
4. Use KillMode=process. (This is not recommended, but possible, if you're absolutely sure that tuned doesn't leave any stray processes around. In any case, it can be used as a temp. workaround.)