When calling "systemctl stop" (or a result of automatic stop based on some dependencies defined within units) for a socket unit, systemd should actually remove the socket/fifo, not leaving it in the system. This comes handy if trying to quickly detect whether the daemon is running or not by directly doing a simple "stat" in the filesystem for the socket/fifo (resp. the daemon is prepared to be run through the systemd socket activation mechanism). Otherwise, when the socket/fifo stays in the system even when the socket unit is stopped, we still need to do extra work to actually see if the socket is usable or not by means of trying to connect to it which ends up with unnecessary error state in case it's not usable... Using "pid" files (which are deprecated anyway) is not usable as well since the pid file does not exist yet if systemd socket activation is used.
Not removing the socket/FIFO file from the filesystem is currently intentional. This comment is in socket_close_fds(): /* One little note: we should never delete any sockets * in the file system here! After all some other * process we spawned might still have a reference of * this fd and wants to continue to use it. Therefore * we delete sockets in the file system before we * create a new one, not after we stopped using * one! */ But I can see how leaving a FIFO file around may cause problems for unsuspecting clients. I do not quite understand the reasoning given in the comment. If I do "systemctl stop foo.socket", I should not be surprised that no more clients can connect to the service. I do not see what the problem would be if we deleted the socket/FIFO files on socket unit stop. Lennart, am I missing something?
This is especially a problem with FIFOs, because opening an unused FIFO will simply block indefinitely.
Consider this: a service A.service uses a socket A.socket. Now both are started, and then you stop A.socket but A.service continues to run. I think in that case the service should have every right to continue use it, and we shouldn't break its current usage. In fact I think I originally deleted those sockets, but ran into a problem with the deletion behaviour which is why added this comment. Don't remember the problem in question however. I think it is quite crucial here to understand that "a socket unit is running" means "systemd will do actviation for it". It does not mean "this socket exists in the file system". Now, what I might be open to here, is to add an option "DeleteSockets=yes" or so (which defaults to no), to optionally enable the behaviour Peter asks for.
(In reply to comment #3) > Consider this: a service A.service uses a socket A.socket. Now both are > started, and then you stop A.socket but A.service continues to run. I think in > that case the service should have every right to continue use it, and we > shouldn't break its current usage. In fact I think I originally deleted those > sockets, but ran into a problem with the deletion behaviour which is why added > this comment. Don't remember the problem in question however. to extend on this: we have a couple of services that you can run with and without socket activation. If we care about supporting those we probably should stay away as much as possible from deleting sockets unless we really have to.
(In reply to comment #2) > This is especially a problem with FIFOs, because opening an unused FIFO will > simply block indefinitely. If you add O_NDELAY to your open flags if you open the fifo then they won't block?
(In reply to comment #3) > Consider this: a service A.service uses a socket A.socket. Now both are > started, and then you stop A.socket but A.service continues to run. I think in > that case the service should have every right to continue use it, and we > shouldn't break its current usage. In fact I think I originally deleted those > sockets, but ran into a problem with the deletion behaviour which is why added > this comment. Don't remember the problem in question however. An example: udev has told systemd to listen on an AF_UNIX socket on its behalf. If now udev's socket unit is stopped it's not systemd's call to break udev, it must be udev's control when that happens. I.e. systemd is only working here on behalf of somebody else and hence should muck with the stuff as little as possible. But yeah, I think an option for this, DeleteSocketsOnStop=yes would make sense, so that we can cover Peter's usecase. (But then again, I think it would make a lot more sense if clients simply make use of sockets instead of checking whether they exitst before, because that is necessarily racy. And using FIFOs for any serious communication is very questionnable too, since FIFOs are vulnerable to multiple clients writing at the same time and getting their stuff interleaved. What are the DM tools doing there to avoid this problem, btw? It really sounds as if you should just use proper sockets instead of FIFOs and that your client should just connect to them, instead of having "first-stat()-then-connect()" logic)
(In reply to comment #6) > But yeah, I think an option for this, DeleteSocketsOnStop=yes would make sense, Yes, that would be fine. > so that we can cover Peter's usecase. (But then again, I think it would make a > lot more sense if clients simply make use of sockets instead of checking > whether they exitst before, because that is necessarily racy. And using FIFOs > for any serious communication is very questionnable too, since FIFOs are > vulnerable to multiple clients writing at the same time and getting their stuff > interleaved. What are the DM tools doing there to avoid this problem, btw? (as discussed on #systemd)
I think that DeleteSocketsOnStop=yes/no wouldn't really solve the proposed use case, as one would have to first check if DeleteSocketsOnStop is set for a specific .socket, and then do the "quick stat check". If anything, it seems to me that systemd could be smart enough to know if anything is serving on a socket. I.e. if systemd has activated a service that has Accept=no, and the service is still active, then don't delete the socket. Otherwise delete it when the socket is stopped.
(In reply to comment #8) > I think that DeleteSocketsOnStop=yes/no wouldn't really solve the proposed > use case, as one would have to first check if DeleteSocketsOnStop is set for > a specific .socket, and then do the "quick stat check". Well, but since the .socket unit would is usually written by the same folks who write the client library for this, they can rely on it being set. Of course, doing this is racy, since there's a time window between when the process died and when systemd gets the SIGCHLD for it and can remove the socket. Which makes me a bit cool for adding this feature. We should do better than adding new functionality that is inherently racy from day 1. > If anything, it seems to me that systemd could be smart enough to know if > anything is serving on a socket. I.e. if systemd has activated a service > that has Accept=no, and the service is still active, then don't delete the > socket. Otherwise delete it when the socket is stopped. Well, but that would open an additional race... No sure how much I like this. Maybe we should find a different way to handle this usecase without mucking with the socket node...
(In reply to comment #9) > > If anything, it seems to me that systemd could be smart enough to know if > > anything is serving on a socket. I.e. if systemd has activated a service > > that has Accept=no, and the service is still active, then don't delete the > > socket. Otherwise delete it when the socket is stopped. > > Well, but that would open an additional race... I don't see the race. systemd is running synchronously and in the .socket stop job it can check if the activated unit is active. I see a different race (maybe that's what you're talking about): if the .service dies during .socket shutdown, we might *not* remove the FIFO or unix socket. But's that's current behaviour, and it's not really a race, since the file we'll also be left behind if the .service dies later. > No sure how much I like this. Maybe we should find a different way to handle > this usecase without mucking with the socket node...
(In reply to comment #10) > (In reply to comment #9) > > > If anything, it seems to me that systemd could be smart enough to know if > > > anything is serving on a socket. I.e. if systemd has activated a service > > > that has Accept=no, and the service is still active, then don't delete the > > > socket. Otherwise delete it when the socket is stopped. > > > > Well, but that would open an additional race... > I don't see the race. systemd is running synchronously and in the .socket > stop job it can check if the activated unit is active. Well, it would be a race where clients would get different errors on shutdown... i.e. if they are quick enough they hang for a while and then, wehn systemd closes the socket they will get ECONNRESET or so. Or, if they are slower, then they will get ECONNREFUSED or so... Now, which error they get is probably not too important, but it does suck a bit if a client's connect() succeeded and then fails, rather than connect() failing right-away if you see what i mean...
> Now, which error they get is probably not too important, but it does suck a > bit if a client's connect() succeeded and then fails, rather than connect() > failing right-away if you see what i mean... Yeah, I see the trouble. And if the socket was deleted before the service was stopped? Are existing connections allowed to finish when the socket file is deleted?
Seems this is now in latest systemd version 214 - the "RemoveOnStop" option. Thanks!