Bug 615527 - RFE: systemd should reload changed unit files automatically
Summary: RFE: systemd should reload changed unit files automatically
Keywords:
Status: CLOSED CANTFIX
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: rawhide
Hardware: All
OS: Linux
low
medium
Target Milestone: ---
Assignee: Lennart Poettering
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-07-16 22:24 UTC by Orion Poplawski
Modified: 2013-09-15 13:45 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-08-31 00:41:15 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)

Description Orion Poplawski 2010-07-16 22:24:35 UTC
Description of problem:

It appears that you need to run "systemctl daemon-reload" to reload unit files.  I was writing a unit file and was trying to figure out why my changes weren't doing anything until systemctl show indicated that it wasn't loading the changes.
Why not use inotify to watch for changes?

Version-Release number of selected component (if applicable):
systemd-3-3.fc14.x86_64

Comment 1 Lennart Poettering 2010-07-17 13:42:17 UTC
Using inotify like this is a racy operation, because it doesn't give us an atomic view on the file system. i.e. let's say you rename "foo.service" to "bar.service" in the fs, and systemd is currently reading the unit directories and reads the units bit by bit, then it might happen that at the time where it should read bar.service it is not yet there, and then a bit later when it should read foo.service it is not there anymore. Or vice versa: it first reads foo.service and succeeds and then a bit later it reads bar.service and reads it too because at the same time the rename was executed, although actually only one of the two files existed at a time. Both cases might result in really unwanted scenarios (i.e. services vanishing from the view entirely, or you might end up suddenly having two conflicting services) and so we decided not to do this logic.

This could only be fixed if the fs would support transactions in some way, so that systemd could get an atomic view on the file system. But that is not in sight and the question if that even is desirable.

However, I recently changed systemd git to do the next best thing: whenever you execute an operation on a service we check the modification time of the backing unit file and compare it with the data from when we read the file originally. You'll then see a nice warning which tells you to reload the daemon configuration, something like "Unit file foo changed, it is recommended to run systemctl daemon-reload". 

Yes, it would be great if we could do something like inotify here and we have really tried to come up with nice ways to make this workable despite the races pointed out above. However, we couldn't come up with anything and because systemd is not a peripheral daemon of the system, but at its very core we chose to better be safe than sorry. And then we did the next best thing we could do.

You could even see that as feature, btw. Think of git's staging area. Everybody loves that. And this is very similar: you make your unit file changes at various places and then, when you have finished them all you activate them in one step.

Sorry if this reply is disappointing...

Comment 2 Eric Paris 2010-08-26 01:41:23 UTC
I've decided to reopen this bug, as if you didn't have enough work to do.  You can still close it base as just a 'wont fix' but I think we can overcome your problems with a new (complex and convoluted) approach.

register an inotify watch on every directory you care about.  You are going to use that watch just to find that files were created/renamed/unlinked.  On every file register an fanotify watch (fanotify being a new feature in 2.6.36-rc1 so not really broadly available yet) just watching for closed_for_writing.

Since fanotify delivers and open fd to the object that was closed for writing you don't have the races you described above.  It doesn't matter if the file was renamed or even unlinked as you'll have access to it.

I hope to expand fanotify such that you can use it instead of inotify to get the link/rename/unlink type operations, but that information isn't available where it is needed in the kernel yet.....

Comment 3 Denys Vlasenko 2011-05-14 00:11:23 UTC
(In reply to comment #2)
> I've decided to reopen this bug, as if you didn't have enough work to do.  You
> can still close it base as just a 'wont fix' but I think we can overcome your
> problems with a new (complex and convoluted) approach.
> 
> register an inotify watch on every directory you care about.  You are going to
> use that watch just to find that files were created/renamed/unlinked.  On every
> file register an fanotify watch (fanotify being a new feature in 2.6.36-rc1 so
> not really broadly available yet) just watching for closed_for_writing.

That would be a TON of watches. (On my machine, systemd already uses 12 megabytes of heap (!), which is a lot for PID 1. What happened to efficiency?...)

I don't see reload races as untractable. First, after receiving inotify event, don't rescan at once, but wait a bit, let user finish messing with config. (If more notify events arrive, they should restart wait counter). One second wait should be enough, as it guarantees that current second changes (see below why).

Then remember current time and do reload. Then, recheck all mtimes on all config files, and check for more inotify events buffered while you were reloading. If you see updated mtime, or have an inotify event, repeat the wait and reload. (In fact, checking mtimes should be superfluous, but a bit of paranoid defensive programming wouldn't hurt here). 

I use a similar technique for async network config and it works great.

Comment 4 Lennart Poettering 2011-08-31 00:41:15 UTC
(In reply to comment #2)
> I've decided to reopen this bug, as if you didn't have enough work to do.  You
> can still close it base as just a 'wont fix' but I think we can overcome your
> problems with a new (complex and convoluted) approach.
> 
> register an inotify watch on every directory you care about.  You are going to
> use that watch just to find that files were created/renamed/unlinked.  On every
> file register an fanotify watch (fanotify being a new feature in 2.6.36-rc1 so
> not really broadly available yet) just watching for closed_for_writing.
> 
> Since fanotify delivers and open fd to the object that was closed for writing
> you don't have the races you described above.  It doesn't matter if the file
> was renamed or even unlinked as you'll have access to it.
> 
> I hope to expand fanotify such that you can use it instead of inotify to get
> the link/rename/unlink type operations, but that information isn't available
> where it is needed in the kernel yet.....

The big issue here is that when you install packets you usually install not only one unit but a couple (for example, CUPS installs cups.path, cups.socket and cups.service). You really want to make sure that you do not end up reloading the state after you installed two of these unit files, but not yet the third one. Because if you did systemd would see a half-upgraded unit set which we really want to avoid.

The problem is that we want atomic snapshots of the whole unit tree: either all of the upgraded/installed units are visible to systemd, or none. But POSIX file systems don't give us semantics like that. (maybe btrfs will one day...).

Yes, your fanotify approach could deal with renames, but partial upgrades might still make it through.

Comment 5 dries 2013-09-15 13:45:48 UTC
(In reply to Eric Paris from comment #2)
> I've decided to reopen this bug, as if you didn't have enough work to do. 
> You can still close it base as just a 'wont fix' but I think we can overcome
> your problems with a new (complex and convoluted) approach.
> 
> register an inotify watch on every directory you care about.  You are going
> to use that watch just to find that files were created/renamed/unlinked.  On
> every file register an fanotify watch (fanotify being a new feature in
> 2.6.36-rc1 so not really broadly available yet) just watching for
> closed_for_writing.
> 
> Since fanotify delivers and open fd to the object that was closed for
> writing you don't have the races you described above.  It doesn't matter if
> the file was renamed or even unlinked as you'll have access to it.
> 
> I hope to expand fanotify such that you can use it instead of inotify to get
> the link/rename/unlink type operations, but that information isn't available
> where it is needed in the kernel yet.....

Note that you could do all this and more outside of systemd. Simply make a tool/script that does what you describe above and have it call systemctl daemon-reload...

In fact, you could reuse said tool for other daemons as well.


Note You need to log in before you can comment on or make changes to this bug.