Description of problem: The final bug from discussions at LinuxCon Prague with Lennart and Kay. It is common for high availability clusters to mange system services (like SYSV/upstart/systemd). In such scenarios, it would be problematic if systemd adhered to the respawn/recovery policy in the unit file. Doing so would confuse the cluster and, depending on the timing, mask problems which would inhibit the recovery preferences configured by the admin in the cluster. The use of override files was discussed and ultimately deemed unsuitable as the behaviour should depend on who starts the service. Eg. Taking the service away from the cluster and starting it manually should use the respawn policy from the unit file (and vice-versa). So the request is for there to be a provision that we can disable any configured respawn policy when we send a message via dbus to start a service.
Changing the org.freedesktop.systemd1.Unit interface to allow <property name="OnFailure" type="as" access="read"/> to be read/write would probably solve it. Would that be possible?
Hmm, so we now have a nice way to turn off restart of a service persistently (via a drop-in file in /etc), and from inside the service one-time (via RestartPreventExitStatus=). But you are looking for a way how you can turn this off, from the outside, but still only for a single run?
Essentially, yes. Is comment #1 feasible?
(In reply to comment #3) > Essentially, yes. > Is comment #1 feasible? Well, it's not that easy. We drop all configuration when "systemctl daemon-reload" is invoked, and then reread all settings fresh. That would mean if we'd make the prop writable it would be lost on the next reload, which is probably not what you want. Here's an idea. On git versions of systemd (i.e. F19 material, but also RHEL7) you can do the following: # mkdir -p /run/systemd/system/mydaemon.service.d/ # cat > /run/systemd/system/mydaemon.service.d/50-turn-off-automatic-restart.conf << EOF [Service] Restart=no EOF i.e. by dropping in a simple config snippet you can alter mydaemon.service as required. By doing this in /run, rather than in /etc this change will be temporary only, i.e. lost at next reboot, but will stay around on "systemctl daemon-reload". That way the cluster software could create a simple file to torn off auto-restart, and then reenable it by deleting that file again. Does that make sense? Would that work for you?
Sorry for the delay, some 16-node clusters were busy kicking my ass :) The /run option is less worse than /etc, but still not as preferable as an API call. Cluster software generally tries not to write to disk if it can be avoided. We can probably make it work if we have to though.
Hmm is this not being approached wrong as I see it the units are behaving correctly and the only bad restart option is "always" What's missing, is for systemd to be able to signal the HA/Cluster applications that it has declare the service ( even whole targets and containers ) dead and it the cluster should take action, with a knob in systemd.conf which would alter distribution's default systemd profile to HA/Cluster one which for example set the correct restart behaviour global for units etc ( not be changing this in units/targets ) anyway this is probably something that needs/should be discussed at systemd/FAD BRNO.
(In reply to comment #6) > Hmm is this not being approached wrong as I see it the units are behaving > correctly and the only bad restart option is "always" > > What's missing, is for systemd to be able to signal the HA/Cluster > applications that it has declare the service ( even whole targets and > containers ) dead and it the cluster should take action, with a knob in > systemd.conf which would alter distribution's default systemd profile to > HA/Cluster one which for example set the correct restart behaviour global > for units etc ( not be changing this in units/targets ) anyway this is > probably something that needs/should be discussed at systemd/FAD BRNO. OnFailure= should probably be extended to signal the ha/cluster software and or another systemd instance running on another host, to take action for active/standby failover setups.
Currently we do polling to check the status of systemd based services. So we do notice when services die, but finding out asynchronously would be even better. However even if there is support for OnFailure=tell-the-cluster, ideally we'd still want a way to allow the cluster to set that programatically.
(In reply to comment #5) > Sorry for the delay, some 16-node clusters were busy kicking my ass :) > > The /run option is less worse than /etc, but still not as preferable as an > API call. Cluster software generally tries not to write to disk if it can > be avoided. Well, /run is not "disk", it's a tmpfs So you are not actually writing to disk there... It's basically a way to communicate a runtime setting, not more...
Is there a #define or pkg-config variable that holds the /run/systemd/system/ path?
Also, is the equivalent of "systemctl daemon-reload" required after creating/deleting one of these files?
(In reply to Andrew Beekhof from comment #10) > Is there a #define or pkg-config variable that holds the > /run/systemd/system/ path? Hmm, nothing except: /usr/share/pkgconfig/systemd.pc: systemdsystemunitpath=${systemdsystemconfdir}:/etc/systemd/system:/run/systemd/system:/usr/local/lib/systemd/system:${systemdsystemunitdir}:/usr/lib/systemd/system:/lib/systemd/system
(In reply to Harald Hoyer from comment #12) > (In reply to Andrew Beekhof from comment #10) > > Is there a #define or pkg-config variable that holds the > > /run/systemd/system/ path? > > Hmm, nothing except: > > /usr/share/pkgconfig/systemd.pc: > > systemdsystemunitpath=${systemdsystemconfdir}:/etc/systemd/system:/run/ > systemd/system:/usr/local/lib/systemd/system:${systemdsystemunitdir}:/usr/ > lib/systemd/system:/lib/systemd/system /run/systemd is pretty much hardcoded in systemd, so I think it will never change.
(In reply to Andrew Beekhof from comment #11) > Also, is the equivalent of "systemctl daemon-reload" required after > creating/deleting one of these files? yes
(In reply to Harald Hoyer from comment #14) > (In reply to Andrew Beekhof from comment #11) > > Also, is the equivalent of "systemctl daemon-reload" required after > > creating/deleting one of these files? > > yes That kinda sucks. Isn't there any way to limit the scope to that one resource? Otherwise thats a rather heavy stick to be throwing around.