Bug 1884352

Hey Mattias,

Thanks for the report. I think we know what we need to do get this sorted out for Fedora 33 and I hope to have a something for you to test out tomorrow.

Can you give this a try Mattias?

https://download.copr.fedorainfracloud.org/results/jdoss/wireguard-tools/fedora-33-x86_64/01693084-wireguard-tools/

and let me know if that fixes the issue? I haven't upgraded to Fedora 33 yet.

Yes, this solves the issue for me. Looking at the fix, I'm a little worried that it seems to break if the user choose to go back to using NetworkManager handling resolv.conf. 
So possibly check if /etc/resolv.conf is a link to ../run/systemd/resolve/stub-resolv.conf and only use resolvconf in that case?

Is that something users are likely do actually want to do for good reasons?

I don't know systwmd-resolvd enough, but I followed some other blocker bug for the f33 beta, where they made sure to allow users to switch back to NetworkManager.

I just realized that even though the current fix makes wireguard work, it still seems like DNS requests are sent to both the wireguard interface DNS and the regular network interface DNS, hence leaking DNS requests. 
I assume the regular DNS should be unset (and restored on down), at least in the case where all traffic is sent through the wireguard interface.

I'm seeing the same issue in Fedora 33 with bout wg-quick and nmcli. DNS requests are sent on the wireguard interface and my other active interface. This is a pretty significant bug.

Just to specify: I'm seeing DNS requests sent to both my Wireguard specified DNS servers and my default DNS servers. All routed through my Wireguard interface. Since my firewall on the Wireguard side blocks requests to any other DNS servers apart from the specified ones, I'm getting all sorts of troubles since Fedora 33 tries to resolve all DNS requests using the default nameservers first, before falling back to the Wireguard specified ones.

(In reply to Jason A. Donenfeld from comment #4)
> Is that something users are likely do actually want to do for good reasons?

Yes. The change in Fedora's configuration is intended to support cases where not *all* traffic is supposed to go through the VPN (in this case, WireGuard), and the rest goes through the normal network.

(In reply to Edvard from comment #8)
> Just to specify: I'm seeing DNS requests sent to both my Wireguard specified
> DNS servers and my default DNS servers. All routed through my Wireguard
> interface. Since my firewall on the Wireguard side blocks requests to any
> other DNS servers apart from the specified ones, I'm getting all sorts of
> troubles since Fedora 33 tries to resolve all DNS requests using the default
> nameservers first, before falling back to the Wireguard specified ones.

That's worrying behavior. I have it on my list to look into.

Systemd's `resolvconf -x` alias is supposed to make things exclusive.

I filed a bug with the systemd project about this: https://github.com/systemd/systemd/issues/17529

According to the systemd bug, this actually looks like a networkmanager issue, where it's telling systemd-resolved to treat your non-VPN interface with the same type of DNS priority as a VPN interface would.

It would be a NetworkManager bug if you configured your Wireguard VPN using NetworkManager, of course. But you are using wg-quick instead of NetworkManager. I don't know anything about wg-quick, but it looks like that is almost certainly lower-level than NetworkManager, and therefore responsible for configuring systemd-resolved itself. Unless wg-quick is creating and/or configuring NetworkManager profiles, I suspect NetworkManager is unrelated to your problem.

Filing a systemd bug was likely not appropriate since the chances of systemd-resolved doing anything wrong are slim to none. It's going to send DNS exactly where you tell it to. NetworkManager should tell it to do the right thing. If you use VPN tools that bypass NetworkManager, those tools are responsible for telling systemd-resolved what to do using its D-Bus API.

Ah, looking closer at this, I see the problem. NetworkManager doesn't know about your Wireguard VPN connection, so has -- correctly -- configured a ~. DNS domain on enp0s2. wg-quick has configured the same on wg0 by passing -x (exclusive) to resolvconf. But -x is really only exclusive if only one network interface has the ~. DNS domain.

IMO NetworkManager is doing the right thing. It's not currently responsible for VPNs that are not configured by NetworkManager itself, and I guess it's not really reasonable to expect it to be? You should either (a) teach wg-quick to control NetworkManager, or (b) stop using either wg-quick or NetworkManager. Trying to set up VPNs that NetworkManager doesn't know about is just not going to work as expected.

Alternative: you could also try to have wg-quick unset the DNS domains of the other network interfaces, but that would be pretty unkind to other VPNs, and also unkind if the wireguard interface is not a full tunnel (e.g. that would be undesired for corporate VPNs), so I don't recommend that.

No, setting ~. on enp0s2 is not correct. It should only set ~. for VPN connections, not for ordinary ethernet cards. Otherwise external programs pushing DNS to systemd-resolved are robbed of an essential feature (`-x`).

(In reply to Jason A. Donenfeld from comment #15)
> No, setting ~. on enp0s2 is not correct. It should only set ~. for VPN
> connections, not for ordinary ethernet cards. Otherwise external programs
> pushing DNS to systemd-resolved are robbed of an essential feature (`-x`).

I've left a comment in the systemd bug to explain why that is necessary (it's required to avoid unexpected DNS leaks).

I think you're probably expecting too much of 'resolvconf -x'. That's a compatibility script. I'm not sure if it's really intended to be used by packaged software like wg-quick. The manpage documents its limitations: it doesn't work if a search domain for ~. exists. And with NetworkManager, that will always exist. So systemd's 'resolvconf -x' is just not comparable to Debian's resolvconf -x: it's not what you hope for it to be.

I would like to point out that this issue occurs even when you import the WG connection profile into Network Manager. It's not limited to wg-quick.

(In reply to Edvard from comment #18)
> I would like to point out that this issue occurs even when you import the WG
> connection profile into Network Manager. It's not limited to wg-quick.

I strongly suspect that's a totally separate issue, then. And that *would* be a NetworkManager bug (I think), worthy of a separate bug report against NetworkManager. (I understand Wireguard interfaces are treated differently than other VPNs by NetworkManager, so perhaps it doesn't consider the Wireguard interface to be a VPN when assigning the DNS domains. I don't know why Wireguard is treated differently.) Anyway, please report a new bug against NetworkManager for this, and we'll see where that goes.

Issue seems very similar, but I've created a separate issue for NetworkManager now.

I found you created bug #1895518. Here's why it's not as similar as it seems: in the wg-quick scenario, wg-quick is responsible for configuring systemd-resolved (and perhaps also NetworkManager) to its satisfaction. But in the NetworkManager scenario, NetworkManager is responsible for doing this.

An external tool needs to be able to tell systemd-resolved "make this the exclusive DNS interface." Otherwise, lots and lots of VPN software cannot work without horrible hacks. The standard interface for doing this is `resolvconf -x`, which is cross platform and added specifically for this purpose. The current bug in NetworkManager prevents this from working. If NetworkManager cannot be made to work with systemd-resolved properly, then either NetworkManager or systemd-resolved should be removed from Fedora, as preventing DNS leaks is an essential security concern of many VPNs. This is a real problem, and not something to be swatted away as somehow "intentional" just because it was part of the rhetoric you used for getting systemd-resolved into Fedora.

I really hope the issue is taken seriously and that we may see a swift solution. I only discovered the issue because my local firewall logs showed blocked DNS requests. Otherwise, I would have assumed everything was working as before. It's a serious security issue and a effectively a regression bug for Fedora users.

(In reply to Edvard from comment #23)
> I really hope the issue is taken seriously and that we may see a swift
> solution. I only discovered the issue because my local firewall logs showed
> blocked DNS requests. Otherwise, I would have assumed everything was working
> as before. It's a serious security issue and a effectively a regression bug
> for Fedora users.

I couldn't agree with you more. Leaking people's DNS traffic in the clear onto the Internet seems really quite bad.

(In reply to Jason A. Donenfeld from comment #24)
> I couldn't agree with you more. Leaking people's DNS traffic in the clear
> onto the Internet seems really quite bad.

So some history: in Fedora 32, in the time before systemd-resolved, there was really not much you could do to prevent DNS leaks. DNS would *usually* work as expected, because NetworkManager will list your VPN connections first in /etc/resolv.conf, but if your VPN's DNS server is slow or not responding, then your request is going to get leaked to the next server, which can be a disaster. But if you test for DNS leaks, it will look as if it's working fine. And it will usually work fine. But it's not actually safe, and your requests could start leaking at any time.

The goal of systemd-resolved is to prevent DNS leaks like those from ever happening again. And so far, this seems to be working well if you configure your VPNs in NetworkManager (except for the problem in bug #1895518). Since Fedora uses NetworkManager by default, I figured this is fine: anybody who goes to the effort to configure something custom outside NetworkManager is surely responsible for ensuring the custom configuration works properly.

Problem is, yes, this broke wg-quick, and I'm not entirely sure how to fix it -- we can continue to discuss in the upstream bug -- but please understand, the whole point of this change was to prevent DNS leaks for desktop users, where VPN configuration is expected to go through NetworkManager. It's very unlikely we would decide to bring back DNS leaks for graphical desktop users in order to fix a command-line tool that's not exposed to desktop users, especially since command-line users are expected to have greater technical knowledge. So unless we find some grand solution, that's the trade-off we're looking at here. See https://fedoraproject.org/wiki/Changes/systemd-resolved#Split_DNS for more DNS leak failure cases before we had systemd-resolved.

FEDORA-2020-7851982ff6 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-7851982ff6

FEDORA-2020-7851982ff6 has been pushed to the Fedora 33 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-7851982ff6`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-7851982ff6

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Has there been any progress on this issue?

This was fixed in NetworkManager 1.26.6: https://bodhi.fedoraproject.org/updates/FEDORA-2020-7851982ff6

The problem is, that FEDORA-2020-7851982ff6 breaks some CI tests related to SSSD (discussed on bodhi), and I didn't yet find the time to understand why. Hence, the update did not reach Fedora 33 stable yet.

FEDORA-2020-7851982ff6 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-7851982ff6

FEDORA-2020-7851982ff6 has been pushed to the Fedora 33 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-7851982ff6`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-7851982ff6

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2020-7851982ff6 has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.