Bug 1895518

(This was split from bug #1884352 at my request.)

Anyway, please post the output of 'resolvectl domain'. What I expect to find is that NetworkManager has created a ~. DNS domain for both your standard ethernet/wifi interface and also for the wireguard interface. But it should only be on the wireguard interface (because you are trying to do a full-tunnel VPN).

I can confirm this issue in Fedora 33. The issue is that NetworkManager tells systemd-resolved to mark all interfaces, not just VPN interfaces, as 'exclusive'. This is a bug. The solution is for NetworkManager to mark normal interfaces as normal, and administrators who then want certain VPNs (for, e.g., having a leak-proof DNS) can mark those interfaces as exclusive. This is the way this stuff was designed to work, and what's happening here is really just a misunderstanding from NM on what systemd-resolved expected. The bug appears easily remediable.

Gif: https://user-images.githubusercontent.com/10643/98296467-401a3b00-1fb3-11eb-9b3a-6938ca1f55f2.gif

Output from resolvectl domain as requested.

resolvectl domain
Global:
Link 2 (wwp0s20f0u6i12):
Link 3 (enp0s31f6):
Link 4 (wlp4s0): ~. home
Link 5 (virbr0):
Link 6 (virbr0-nic):
Link 7 (wg0): saturn.local

Oh. Saturn.local... that seems different.

Is your "AllowedIPs" 0.0.0.0/0?

Saturn.local is my local search domain. I have my own DNS server locally that I use with Wireguard peers.

And yes, "AllowedIPs" 0.0.0.0/0. 

Just to make it clear. This all worked fine before upgrading to Fedora 33.

(In reply to Jason A. Donenfeld from comment #2)
> I can confirm this issue in Fedora 33. The issue is that NetworkManager
> tells systemd-resolved to mark all interfaces, not just VPN interfaces, as
> 'exclusive'. This is a bug. The solution is for NetworkManager to mark
> normal interfaces as normal,

We are discussing this in https://github.com/systemd/systemd/issues/17529. Suffice to say that Zbigniew and I strongly disagree. That should only happen full-tunnel VPN is in use; otherwise, DNS leaks will occur, which can be disastrous.

(In reply to Edvard from comment #3)
> Output from resolvectl domain as requested.
> 
> resolvectl domain
> Global:
> Link 2 (wwp0s20f0u6i12):
> Link 3 (enp0s31f6):
> Link 4 (wlp4s0): ~. home
> Link 5 (virbr0):
> Link 6 (virbr0-nic):
> Link 7 (wg0): saturn.local

OK, that's not quite what I expected, but still *probably* a NetworkManager bug. It should look like this:

resolvectl domain
> Global:
> Link 2 (wwp0s20f0u6i12):
> Link 3 (enp0s31f6):
> Link 4 (wlp4s0): home
> Link 5 (virbr0):
> Link 6 (virbr0-nic):
> Link 7 (wg0): saturn.local ~.

The ~. DNS domain is in the wrong place. I guess this might be a Wireguard-specific issue, because NetworkManager does the right thing in general for other VPN connections. (In contrast, in bug #1884352 you have two ~. DNS domains on both interfaces, whereas here you have only one, further illustrating that these are really two separate bugs, even if the symptoms appear similar.)

You might also want to post your /etc/wireguard/wg0.conf (feel free to censor any IP addresses or domain names that you don't want us to see) in case the NetworkManager developers need to see it (possible if they are unable to reproduce).

Wireguard config.

[Interface]
Address = 10.10.0.4
PrivateKey = <redacted>
ListenPort = <redacted>
DNS = 10.10.8.5

[Peer]
PublicKey = <redacted>
Endpoint = <redacted>
AllowedIPs = 0.0.0.0/0

Again, this all worked fine on all my systems before upgrade to Fedora 33. Now, all my systems exhibit the described behavior.

(In reply to Edvard from comment #8)
> Again, this all worked fine on all my systems before upgrade to Fedora 33.
> Now, all my systems exhibit the described behavior.

(That's because Fedora 32 did not use systemd-resolved.)

Possibly related:

https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/405
https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/7b630d8dba5d1363be87dba499a5852aec67da9a

Edvard, what is your ipv4.dns-priority setting on wlp4s0 and wg0?

(Do "nmcli con show --active" to map those device names to NetworkManager connection profiles, then do "nmcli con show <conname> | grep dns-prio" for each of those.)

Actually, on second look, regarding this in comment #3:

> Link 7 (wg0): saturn.local

Where does this domain «saturn.local» come from? It is not present in the WireGuard config file posted in comment #8.

Was this added manually to the NetworkManager connection profile for wg0 (after importing it with 'nmcli connection import')?

Please post output of "nmcli con show <wg0-conname> | grep dns".

The default search domain for WireGuard connections is in recent NM versions set to «~» (cf. https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/88756703cf070e37fcfc5fc61a5c09b75de1de2c).

However, if you have overwritten the «~» default  with «saturn.local», it is the expected behaviour that the WireGuard DNS server 10.10.8.5 is only used for lookups of *.saturn.local (split horizon DNS).

"saturn.local" is indeed a local domain I manually added to the NetworkManager connection profile after importing it. 

Output as requested.

connection.mdns:                        -1 (default)
ipv4.dns:                               10.10.18.7
ipv4.dns-search:                        saturn.local
ipv4.dns-options:                       --
ipv4.dns-priority:                      0
ipv4.ignore-auto-dns:                   no
ipv6.dns:                               --
ipv6.dns-search:                        --
ipv6.dns-options:                       --
ipv6.dns-priority:                      0
ipv6.ignore-auto-dns:                   no


Also, all other network connections have priority 0 as well.

In that case, there is no bug, as far as I can tell. By changing the default "dns-search", you have told NetworkManager that you want the 10.10.18.7 DNS servers to *only* be used for lookups of *.saturn.local. Therefore, all lookups (except *.saturn.local ones) go to a DNS server belonging to another interface.

Regarding "dns-priority" being 0: Have you also changed this setting from its default value (-10), or was this connection profile created with a version of NetworkManager older than v1.24 (which is the first version when the default "dns-priority" for WireGuard profiles changed to -10)?

Could you delete your current WireGuard connection profile and recreate it using your own «Steps to Reproduce» from comment #0 (without including any other "secret" steps, such as changing "dns-search")? I suspect that this is all it takes for the system to behave as you want.

(Afterwards, please post output of "nmcli con show <wg0-conname> | grep dns" and "resolvectl domain".)

I must admit that I'm not sure what NetworkManager version was used to originally import the profile. I have not manually manipulated the default value for the dns-priority.

At any rate, I removed the connection and started from scratch, and it seems indeed like importing the connection with NetworkManager 1.26.4 sets the priority to -10 when importing the Wireguard connection. However, I ran into other issues, specifically in regards to search domain.

Given the following wireguard profile, importing it with NetworkManager gave the following error: Error: failed to import '/etc/wireguard/wg-client0.conf': unrecognized line at /etc/wireguard/wg-client0.conf:6.

[Interface]
Address = <redacted>
PrivateKey = <redacted>
ListenPort = 51820
DNS = 10.10.18.5
DNSSearch = saturn.local

[Peer]
PublicKey = <redacted>
Endpoint = <redacted>
AllowedIPs = 0.0.0.0/0


DNSSearch was added in Wireguard 0.1.1.

Alternatively, using the old way of specifiying search domains by adding it ot the DNS parameter, resulted in nmcli returning: Error: failed to import '/etc/wireguard/wg-client0.conf': Failed to create WireGuard connection: ipv6.dns-search: this property is not allowed for 'method=disabled'.


So therefore, I remove the search domain from the wireguard config entirely, and import it.

The connection profile then seems to work just fine, except my DNS search domain not being added.

Manually adding it by calling: nmcli connection modify wg-client0 ipv4.dns-search "saturn.local" results in the specified DNS server only being used for that search domain and the non-wireguard specified DNS server being used for everything else. If this is the intended behaviour, I hope someone can explain to me how I am supposed to set a DNS search domain for my wireguard connection when using it with NetworkManager.

All that said, it seems to original issue I had was related to older versions of network manager setting the incorrect dns-priority which is fixed in newer versions. However, my debugging confused me further by the fact that specifying a search domain for the nm connection did not work as I expected it to.

Hope that made sense.

(In reply to Edvard from comment #14)
> I must admit that I'm not sure what NetworkManager version was used to
> originally import the profile. I have not manually manipulated the default
> value for the dns-priority.

Probably pre-1.24.

> Given the following wireguard profile, importing it with NetworkManager gave
> the following error: Error: failed to import
> '/etc/wireguard/wg-client0.conf': unrecognized line at
> /etc/wireguard/wg-client0.conf:6.

I suppose NM 1.26.4 simply does not understand the new wg-quick DNSSearch option. This should probably be submitted as a separate issue at the upstream tracker at https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues (assuming that it hasn't already been fixed in Git master or that someone hasn't already submitted such an issue).

> Manually adding it by calling: nmcli connection modify wg-client0
> ipv4.dns-search "saturn.local" results in the specified DNS server only
> being used for that search domain and the non-wireguard specified DNS server
> being used for everything else. If this is the intended behaviour, I hope
> someone can explain to me how I am supposed to set a DNS search domain for
> my wireguard connection when using it with NetworkManager.

I believe that the problem is that you are *replacing* the default dns-search setting of «~» with «saturn.local». In doing so, you are telling NM that you do *not* want this connections used for the "everything else" queries (as those were represented by the default «~» entry that you just deleted).

I think that you instead need to *add* «saturn.local» to get the behaviour you want.

That is: nmcli connection modify wg-client0 ipv4.dns-search "saturn.local,~"

Does that work better?

That did indeed work better. 

Thank you. The real gotcha was probably the pre-1.24 thing. I think we can close this issue then. Thank you for taking the time to troubleshoot.

(In reply to Tore Anderson from comment #15)
> I believe that the problem is that you are *replacing* the default
> dns-search setting of «~» with «saturn.local». In doing so, you are telling
> NM that you do *not* want this connections used for the "everything else"
> queries (as those were represented by the default «~» entry that you just
> deleted).
> 
> I think that you instead need to *add* «saturn.local» to get the behaviour
> you want.
> 
> That is: nmcli connection modify wg-client0 ipv4.dns-search "saturn.local,~"

Nice, thanks for solving this mystery!

I had quite a challenge to define my conf as I wanted to tunnel all traffic including DNS as I have DOT configured on the Wireguard Server.
I hope this post helps somebody else.

```
# cat /etc/wireguard/wg0.conf 
[Interface]
Address    = <CLIENT IP ADDRESS REGISTERED IN SERVER>/32
PrivateKey = <CLIENT PRIVATE KEY>
DNS = 10.100.0.1, ~.,localdomain

[Peer]
PublicKey    = <SERVER PUB KEY>
Endpoint     = <SERVER FQDN OR IP>:<SERVER PORT>
AllowedIPs   = 0.0.0.0/0
PersistentKeepalive = 20
```

Note: The config configures the DNS Server to always be 10.100.0.1 (accessible over the tunnel) and the server will be used for all queries (~.) and will append "localdomain" to all hostname searches (nslookup test == nslookup test.localdomain).
If you miss the "~." , for anything outside of "localdomain" systemd-resolved will be using main interface DNS (or any Fallback) which for DOH/DOT will not work.

Import the file:
```
# nmcli con import type wireguard file /etc/wireguard/wg0.conf
```

Validate if you are using the correct DNS (test with your DOH/DOT server - in my case it's CloudFlare DOT):
```
firefox  https://1.0.0.1/help
```