Bug 1116999

Basically I think systemd installing a symlink for systemd-resolved isn't really moving anything forward.

Yeah, I agree for the "generated by NM" case it would make sense in /run.  And NetworkManager *does* handle it being a symbolic link.

But what NM does is only part of the issue; at the moment Anaconda still supports (to a degree) installs without NetworkManager. For legacy scripts, /etc/resolv.conf is the canonical data store.

Even if we declare that path dead, what happens with resolv.conf starts to get into e.g.:

http://fedoraproject.org/wiki/Features/DNSSEC

(Not enabled by default, but...)

If your suggested vision here is the various components just learn to unlink(/etc/resolv.conf) before creating it, how does that really help anything?

It seems like something like NetworkManager writing data to systemd-resolved (and enabling it by default) would be a more useful path.  I'm curious what the NM people think about that.

(In reply to Colin Walters from comment #1)
> It seems like something like NetworkManager writing data to systemd-resolved
> (and enabling it by default) would be a more useful path.  I'm curious what
> the NM people think about that.

What is the benefit of having yet another daemon running on the system to take care solely of the /etc/resolv.conf, which is currently done just fine by NM?

Note that there are currently also other components on the system that are managing the /etc/resolv.conf if running, like dnssec-trigger. While with the dnssec-trigger we are trying to get rid of the daemon and let NM handle resolv.conf, there may be also other components doing that (none that I'm aware of).

Due to this fact, using introducing another daemon to handle /etc/resolv.conf
seems like a step back...

To be honest, I always disliked /etc/resolv.conf because it is runtime configuration stored in a physical filesystem. Moving it to /run would make sense even in the dnssec-trigger context. You would simply check the target of /etc/resolv.conf to se who's in charge. NetworkManager might also attempt to backup original /etc/resolv.conf when started and attempt to restore it when stopped.

I just attempted to use a /etc/resolv.conf symlink pointing to /var/run/dnssec-trigger/resolv.conf. For dnssec-trigger this is an easy change (it already supports an alternative resolv.conf location. But I'm curious whether changing /etc/resolv.conf to a symlink can cause problems to applications e.g. when using inotify to monitor it.

(In reply to Colin Walters from comment #1)
> Basically I think systemd installing a symlink for systemd-resolved isn't
> really moving anything forward.

Hmm? Well, it's an upstream default, since for man systems that are just barebones, that's what you want. If you install something else, then that's supposed to replace the symlink, and it won't be overwritten. Creating the symlink is simply done when there is no file existing yet.

> Yeah, I agree for the "generated by NM" case it would make sense in /run. 
> And NetworkManager *does* handle it being a symbolic link.
> 
> But what NM does is only part of the issue; at the moment Anaconda still
> supports (to a degree) installs without NetworkManager. For legacy scripts,
> /etc/resolv.conf is the canonical data store.
> 
> Even if we declare that path dead, what happens with resolv.conf starts to
> get into e.g.:
> 
> http://fedoraproject.org/wiki/Features/DNSSEC
> 
> (Not enabled by default, but...)
> 
> If your suggested vision here is the various components just learn to
> unlink(/etc/resolv.conf) before creating it, how does that really help
> anything?

Yeah, if packages take possession of /etc/resolv.conf they should do so by making it a link to some place they manage in /run or so. Then it is clear who owns it.

Most importantly though if it is a symlink, then /etc can be read-only. Which it should be for most cases.

> It seems like something like NetworkManager writing data to systemd-resolved
> (and enabling it by default) would be a more useful path.  I'm curious what
> the NM people think about that.

No. I am not interested in a private channel between nm and resolved. 

Either resolved or NM should be in possession of /etc/resolv.conf. It should declare that by making it a symlink to its own file in /run at installation time. And from that point on /etc should be treated read-only, and the file in /run is the one that should change.

resolved is now smart enough to detect whether /etc/resolv.conf is symlinked to somebody else's (such as NM's) resolv.conf file, in which case it reads its data and makes use of the configured DNS servers. However, if it is symlinked to resolved's own file, then it will ignore it, because it shouldn'r read back its own data...

That way, people can use NM in conjunction with resolved. Or only NM. Or only resolved. The right thing will always happen. If you boot up without /etc/resolv.conf however tmpfiles will add the symlink to resolved's resolv.conf file, since that is better than nothing at all.

(In reply to Tomas Hozza from comment #2)
> (In reply to Colin Walters from comment #1)
> > It seems like something like NetworkManager writing data to systemd-resolved
> > (and enabling it by default) would be a more useful path.  I'm curious what
> > the NM people think about that.
> 
> What is the benefit of having yet another daemon running on the system to
> take care solely of the /etc/resolv.conf, which is currently done just fine
> by NM?

Hmm? Nothing. systemd-resolved is nowadays a caching DNS and LLMNR stub resolver, and we are working on adding mDNS and DNSSEC support.

> Note that there are currently also other components on the system that are
> managing the /etc/resolv.conf if running, like dnssec-trigger. While with
> the dnssec-trigger we are trying to get rid of the daemon and let NM handle
> resolv.conf, there may be also other components doing that (none that I'm
> aware of).
> 
> Due to this fact, using introducing another daemon to handle /etc/resolv.conf
> seems like a step back...

There's no daemon handling that file. The admin/installer/package scripts should just pick one component that owns the file, and create a symlink to its version in /run, and then /etc is read-only, and it is clear who owns that file.

(In reply to Pavel Šimerda (pavlix) from comment #4)
> I just attempted to use a /etc/resolv.conf symlink pointing to
> /var/run/dnssec-trigger/resolv.conf. For dnssec-trigger this is an easy
> change (it already supports an alternative resolv.conf location. But I'm
> curious whether changing /etc/resolv.conf to a symlink can cause problems to
> applications e.g. when using inotify to monitor it.

Using inotify on /etc/resolv.conf is not really doable anway. I mean, if you want to be able to cover the case where /etc/resolv.conf is missing and is created you have to add /etc itself to your watchlist, which is really not a good idea, for performance reasons.

In general there's really no point in using inotify to watch /etc/resolv.conf. Clients which need the file should simply stat() the file on each request (possibly ratelimited) and reread it if it changed.

Finally, and most importantly: since a long time Debian has been managing /etc/resolv.conf with a a "resolvconf" package that generates it dynamically from various configuration sources, and actually turns /etc/resolv.conf into a symlink. Given that Debian has been doing this for a long time it should be enough for us, too.

I don't think there will be a problem with inotify for this.

(In reply to Lennart Poettering from comment #7)
> (In reply to Pavel Šimerda (pavlix) from comment #4)
> > I just attempted to use a /etc/resolv.conf symlink pointing to
> > /var/run/dnssec-trigger/resolv.conf. For dnssec-trigger this is an easy
> > change (it already supports an alternative resolv.conf location. But I'm
> > curious whether changing /etc/resolv.conf to a symlink can cause problems to
> > applications e.g. when using inotify to monitor it.
> 
> Using inotify on /etc/resolv.conf is not really doable anway. I mean, if you
> want to be able to cover the case where /etc/resolv.conf is missing and is
> created you have to add /etc itself to your watchlist, which is really not a
> good idea, for performance reasons.

OK.

> In general there's really no point in using inotify to watch
> /etc/resolv.conf. Clients which need the file should simply stat() the file
> on each request (possibly ratelimited) and reread it if it changed.
> 
> Finally, and most importantly: since a long time Debian has been managing
> /etc/resolv.conf with a a "resolvconf" package that generates it dynamically
> from various configuration sources, and actually turns /etc/resolv.conf into
> a symlink.

Good then. I will proceed with the dnssec-trigger (bug #1125267) then. Note that dnssec-trigger is taking over /etc/resolv.conf actively, so users of dnssec-trigger should expect the symlink to change when starting and stopping that service. Not sure whether it will change in the future, as there's no point in running dnssec-trigger and not touching resolv.conf at the same time.

By the way, from the perspective of dnssec-trigger, especially when stopping it to give full control back to NetworkManager (in the specific case), it would be great of NetworkManager always created its runtime version of /etc/resolv.conf and the dns=... directive would only say whether the symlink in /etc/resolv.conf is created or not.

(In reply to Pavel Šimerda (pavlix) from comment #9)
> By the way, from the perspective of dnssec-trigger, especially when stopping
> it to give full control back to NetworkManager (in the specific case), it
> would be great of NetworkManager always created its runtime version of
> /etc/resolv.conf and the dns=... directive would only say whether the
> symlink in /etc/resolv.conf is created or not.

This sounds like a great idea. We then won't have to restart NM any more. Just change the symlink to point to the NM up-to-date resolv.conf.

I submitted a patch upstream[1] that turns /etc/resolv.conf into a symlink while keeping most of the behavior same as before, i.e. NetworkManager still attempts to rewrite the symlink and still only writes the private resolv.conf when also going to write the symlink. Please regard it as just the first step.

[1] https://bugzilla.gnome.org/show_bug.cgi?id=732941#c1

Upstream now has two patches, let's see whether they accept one or both of them for NetworkManager 1.0.

If both are accepted:

 * NetworkManager will never open /etc/resolv.conf for writing and will instead replace it when applicable. Therefore it will no longer rewrite the private resolv.conf files belonging to other services.

 * NetworkManager will check /etc/resolv.conf before replacing it and will avoid doing so when /etc/resolv.conf is a symlink pointing to anything else than NetworkManager's private data. Therefore it will not destroy the symlink when pointing to the private resolv.conf of another service (even if the target of the symlink is currently missing).

 * NetworkManager itself will write out a private resolv.conf in /run/NetworkManager and install the /etc/resolv.conf symlink when applicable, so that other tools see that it's in charge of the contents.

This helps dnssec-trigger, I hope it will help systemd as well.

Pavel, that sounds like a good design.  I don't think it gets us out of the central question of "who owns the DNS configuration this release", but it does look like it'll make it *much* easier for a third party tool to read the data NM wants and integrate it, and vice versa.

(In reply to Colin Walters from comment #13)
> Pavel, that sounds like a good design.

Thanks, it's basically an implementation of what Lennart suggested plus some minor tweaks.

> I don't think it gets us out of the
> central question of "who owns the DNS configuration this release",

If I understand Lennart's request correctly, he just wanted NetworkManager to respect a decision to point /etc/resolv.conf symlink to another tool. I guess it's the Fedora products (or how it's called in Fedora.next) are there to decide the defaults for their use cases. 

> but it
> does look like it'll make it *much* easier for a third party tool to read
> the data NM wants and integrate it, and vice versa.

Definitely. Patches are submitted and reviewed. Now upstream has to decide whether the change is suitable for the 1.0 release and maintainers have to decide whether and when to include it in Fedora.

Would be a great improvement of the DNSSEC feature in that dnssec-triggerd stop/restart would not trigger NetworkManager restart.

I backported upstream commits 
  4805be2ed27b71a6099477d86dbc109adb41b819
  583568e12f9e580cd2903811637c9f9b7a2f1088
  4c691cf69ed33bcbaa0b4802e419b98ed687630b
to NetworkManager-1.0.0-3.fc22 (rawhide).

These commits are not yet in nm-1-0 stable branch, and hence not yet in Fedora 21 either.

The backport was committed to rawhide before the F22 branch, so it also made it into F22 when the branch happened.  Unfortunately F22 is not supposed to make this change (only F23+ should have the symlink behavior) so I had to revert it for F22.  rawhide/F23+ will continue to have the behavior though.