Bug 1895518 - NetworkManager does not properly support systemd-resolved for wireguard connections, resulting in DNS leaks [NEEDINFO]
Summary: NetworkManager does not properly support systemd-resolved for wireguard conne...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Fedora
Classification: Fedora
Component: NetworkManager
Version: 33
Hardware: All
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: Lubomir Rintel
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-11-06 21:20 UTC by Edvard
Modified: 2020-11-18 15:03 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-11-18 14:05:54 UTC
Type: Bug
Jason: needinfo? (thaller)


Attachments (Terms of Use)

Description Edvard 2020-11-06 21:20:24 UTC
Description of problem:

When connecting to a imported Wireguard profile with NetworkManager, I can see DNS requests going to both name servers specified in the wireguard interface and the default network interface, resulting in DNS leaks. All DNS requests are tunneled through the Wireguard interface, but it will attempt to use the standard interface nameservers first, before falling back to the ones specified by the Wireguard interface.

Version-Release number of selected component (if applicable): 1.26.4-1.fc33


How reproducible: 100%


Steps to Reproduce:
1. Import wireguard profile: nmcli connection import type wireguard file /etc/wireguard/wg0.conf with DNS address being specified in the config.
2. Bring up the wireguard connection with nmcli.

Actual results:
resolvectl will show correct DNS servers for the different connections, but you'll see how any DNS lookups will be done using the primary interface DNS addresses and NOT the ones specified by the wireguard interface.

Expected results:
All DNS requests should use the DNS servers specified by the wireguard interface.

Comment 1 Michael Catanzaro 2020-11-06 21:30:17 UTC
(This was split from bug #1884352 at my request.)

Anyway, please post the output of 'resolvectl domain'. What I expect to find is that NetworkManager has created a ~. DNS domain for both your standard ethernet/wifi interface and also for the wireguard interface. But it should only be on the wireguard interface (because you are trying to do a full-tunnel VPN).

Comment 2 Jason A. Donenfeld 2020-11-06 21:34:39 UTC
I can confirm this issue in Fedora 33. The issue is that NetworkManager tells systemd-resolved to mark all interfaces, not just VPN interfaces, as 'exclusive'. This is a bug. The solution is for NetworkManager to mark normal interfaces as normal, and administrators who then want certain VPNs (for, e.g., having a leak-proof DNS) can mark those interfaces as exclusive. This is the way this stuff was designed to work, and what's happening here is really just a misunderstanding from NM on what systemd-resolved expected. The bug appears easily remediable.

Gif: https://user-images.githubusercontent.com/10643/98296467-401a3b00-1fb3-11eb-9b3a-6938ca1f55f2.gif

Comment 3 Edvard 2020-11-06 21:40:50 UTC
Output from resolvectl domain as requested.

resolvectl domain
Global:
Link 2 (wwp0s20f0u6i12):
Link 3 (enp0s31f6):
Link 4 (wlp4s0): ~. home
Link 5 (virbr0):
Link 6 (virbr0-nic):
Link 7 (wg0): saturn.local

Comment 4 Jason A. Donenfeld 2020-11-06 21:42:03 UTC
Oh. Saturn.local... that seems different.

Is your "AllowedIPs" 0.0.0.0/0?

Comment 5 Edvard 2020-11-06 21:44:49 UTC
Saturn.local is my local search domain. I have my own DNS server locally that I use with Wireguard peers.

Comment 6 Edvard 2020-11-06 21:45:40 UTC
And yes, "AllowedIPs" 0.0.0.0/0. 

Just to make it clear. This all worked fine before upgrading to Fedora 33.

Comment 7 Michael Catanzaro 2020-11-06 21:48:57 UTC
(In reply to Jason A. Donenfeld from comment #2)
> I can confirm this issue in Fedora 33. The issue is that NetworkManager
> tells systemd-resolved to mark all interfaces, not just VPN interfaces, as
> 'exclusive'. This is a bug. The solution is for NetworkManager to mark
> normal interfaces as normal,

We are discussing this in https://github.com/systemd/systemd/issues/17529. Suffice to say that Zbigniew and I strongly disagree. That should only happen full-tunnel VPN is in use; otherwise, DNS leaks will occur, which can be disastrous.

(In reply to Edvard from comment #3)
> Output from resolvectl domain as requested.
> 
> resolvectl domain
> Global:
> Link 2 (wwp0s20f0u6i12):
> Link 3 (enp0s31f6):
> Link 4 (wlp4s0): ~. home
> Link 5 (virbr0):
> Link 6 (virbr0-nic):
> Link 7 (wg0): saturn.local

OK, that's not quite what I expected, but still *probably* a NetworkManager bug. It should look like this:

resolvectl domain
> Global:
> Link 2 (wwp0s20f0u6i12):
> Link 3 (enp0s31f6):
> Link 4 (wlp4s0): home
> Link 5 (virbr0):
> Link 6 (virbr0-nic):
> Link 7 (wg0): saturn.local ~.

The ~. DNS domain is in the wrong place. I guess this might be a Wireguard-specific issue, because NetworkManager does the right thing in general for other VPN connections. (In contrast, in bug #1884352 you have two ~. DNS domains on both interfaces, whereas here you have only one, further illustrating that these are really two separate bugs, even if the symptoms appear similar.)

You might also want to post your /etc/wireguard/wg0.conf (feel free to censor any IP addresses or domain names that you don't want us to see) in case the NetworkManager developers need to see it (possible if they are unable to reproduce).

Comment 8 Edvard 2020-11-06 22:04:06 UTC
Wireguard config.

[Interface]
Address = 10.10.0.4
PrivateKey = <redacted>
ListenPort = <redacted>
DNS = 10.10.8.5

[Peer]
PublicKey = <redacted>
Endpoint = <redacted>
AllowedIPs = 0.0.0.0/0

Again, this all worked fine on all my systems before upgrade to Fedora 33. Now, all my systems exhibit the described behavior.

Comment 9 Michael Catanzaro 2020-11-06 22:31:04 UTC
(In reply to Edvard from comment #8)
> Again, this all worked fine on all my systems before upgrade to Fedora 33.
> Now, all my systems exhibit the described behavior.

(That's because Fedora 32 did not use systemd-resolved.)

Comment 10 Tore Anderson 2020-11-16 17:25:47 UTC
Possibly related:

https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/405
https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/7b630d8dba5d1363be87dba499a5852aec67da9a

Edvard, what is your ipv4.dns-priority setting on wlp4s0 and wg0?

(Do "nmcli con show --active" to map those device names to NetworkManager connection profiles, then do "nmcli con show <conname> | grep dns-prio" for each of those.)

Comment 11 Tore Anderson 2020-11-16 20:14:48 UTC
Actually, on second look, regarding this in comment #3:

> Link 7 (wg0): saturn.local

Where does this domain «saturn.local» come from? It is not present in the WireGuard config file posted in comment #8.

Was this added manually to the NetworkManager connection profile for wg0 (after importing it with 'nmcli connection import')?

Please post output of "nmcli con show <wg0-conname> | grep dns".

The default search domain for WireGuard connections is in recent NM versions set to «~» (cf. https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/88756703cf070e37fcfc5fc61a5c09b75de1de2c).

However, if you have overwritten the «~» default  with «saturn.local», it is the expected behaviour that the WireGuard DNS server 10.10.8.5 is only used for lookups of *.saturn.local (split horizon DNS).

Comment 12 Edvard 2020-11-17 13:57:48 UTC
"saturn.local" is indeed a local domain I manually added to the NetworkManager connection profile after importing it. 

Output as requested.

connection.mdns:                        -1 (default)
ipv4.dns:                               10.10.18.7
ipv4.dns-search:                        saturn.local
ipv4.dns-options:                       --
ipv4.dns-priority:                      0
ipv4.ignore-auto-dns:                   no
ipv6.dns:                               --
ipv6.dns-search:                        --
ipv6.dns-options:                       --
ipv6.dns-priority:                      0
ipv6.ignore-auto-dns:                   no


Also, all other network connections have priority 0 as well.

Comment 13 Tore Anderson 2020-11-17 15:05:26 UTC
In that case, there is no bug, as far as I can tell. By changing the default "dns-search", you have told NetworkManager that you want the 10.10.18.7 DNS servers to *only* be used for lookups of *.saturn.local. Therefore, all lookups (except *.saturn.local ones) go to a DNS server belonging to another interface.

Regarding "dns-priority" being 0: Have you also changed this setting from its default value (-10), or was this connection profile created with a version of NetworkManager older than v1.24 (which is the first version when the default "dns-priority" for WireGuard profiles changed to -10)?

Could you delete your current WireGuard connection profile and recreate it using your own «Steps to Reproduce» from comment #0 (without including any other "secret" steps, such as changing "dns-search")? I suspect that this is all it takes for the system to behave as you want.

(Afterwards, please post output of "nmcli con show <wg0-conname> | grep dns" and "resolvectl domain".)

Comment 14 Edvard 2020-11-18 11:20:38 UTC
I must admit that I'm not sure what NetworkManager version was used to originally import the profile. I have not manually manipulated the default value for the dns-priority.

At any rate, I removed the connection and started from scratch, and it seems indeed like importing the connection with NetworkManager 1.26.4 sets the priority to -10 when importing the Wireguard connection. However, I ran into other issues, specifically in regards to search domain.

Given the following wireguard profile, importing it with NetworkManager gave the following error: Error: failed to import '/etc/wireguard/wg-client0.conf': unrecognized line at /etc/wireguard/wg-client0.conf:6.

[Interface]
Address = <redacted>
PrivateKey = <redacted>
ListenPort = 51820
DNS = 10.10.18.5
DNSSearch = saturn.local

[Peer]
PublicKey = <redacted>
Endpoint = <redacted>
AllowedIPs = 0.0.0.0/0


DNSSearch was added in Wireguard 0.1.1.

Alternatively, using the old way of specifiying search domains by adding it ot the DNS parameter, resulted in nmcli returning: Error: failed to import '/etc/wireguard/wg-client0.conf': Failed to create WireGuard connection: ipv6.dns-search: this property is not allowed for 'method=disabled'.


So therefore, I remove the search domain from the wireguard config entirely, and import it.

The connection profile then seems to work just fine, except my DNS search domain not being added.

Manually adding it by calling: nmcli connection modify wg-client0 ipv4.dns-search "saturn.local" results in the specified DNS server only being used for that search domain and the non-wireguard specified DNS server being used for everything else. If this is the intended behaviour, I hope someone can explain to me how I am supposed to set a DNS search domain for my wireguard connection when using it with NetworkManager.

All that said, it seems to original issue I had was related to older versions of network manager setting the incorrect dns-priority which is fixed in newer versions. However, my debugging confused me further by the fact that specifying a search domain for the nm connection did not work as I expected it to.

Hope that made sense.

Comment 15 Tore Anderson 2020-11-18 13:46:20 UTC
(In reply to Edvard from comment #14)
> I must admit that I'm not sure what NetworkManager version was used to
> originally import the profile. I have not manually manipulated the default
> value for the dns-priority.

Probably pre-1.24.

> Given the following wireguard profile, importing it with NetworkManager gave
> the following error: Error: failed to import
> '/etc/wireguard/wg-client0.conf': unrecognized line at
> /etc/wireguard/wg-client0.conf:6.

I suppose NM 1.26.4 simply does not understand the new wg-quick DNSSearch option. This should probably be submitted as a separate issue at the upstream tracker at https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues (assuming that it hasn't already been fixed in Git master or that someone hasn't already submitted such an issue).

> Manually adding it by calling: nmcli connection modify wg-client0
> ipv4.dns-search "saturn.local" results in the specified DNS server only
> being used for that search domain and the non-wireguard specified DNS server
> being used for everything else. If this is the intended behaviour, I hope
> someone can explain to me how I am supposed to set a DNS search domain for
> my wireguard connection when using it with NetworkManager.

I believe that the problem is that you are *replacing* the default dns-search setting of «~» with «saturn.local». In doing so, you are telling NM that you do *not* want this connections used for the "everything else" queries (as those were represented by the default «~» entry that you just deleted).

I think that you instead need to *add* «saturn.local» to get the behaviour you want.

That is: nmcli connection modify wg-client0 ipv4.dns-search "saturn.local,~"

Does that work better?

Comment 16 Edvard 2020-11-18 14:05:54 UTC
That did indeed work better. 

Thank you. The real gotcha was probably the pre-1.24 thing. I think we can close this issue then. Thank you for taking the time to troubleshoot.

Comment 17 Michael Catanzaro 2020-11-18 15:03:04 UTC
(In reply to Tore Anderson from comment #15)
> I believe that the problem is that you are *replacing* the default
> dns-search setting of «~» with «saturn.local». In doing so, you are telling
> NM that you do *not* want this connections used for the "everything else"
> queries (as those were represented by the default «~» entry that you just
> deleted).
> 
> I think that you instead need to *add* «saturn.local» to get the behaviour
> you want.
> 
> That is: nmcli connection modify wg-client0 ipv4.dns-search "saturn.local,~"

Nice, thanks for solving this mystery!


Note You need to log in before you can comment on or make changes to this bug.