Bug 1883089 - systemd-resolved mangled /etc/resolv.conf to 127.0.0.53 after VPN down
Summary: systemd-resolved mangled /etc/resolv.conf to 127.0.0.53 after VPN down
Keywords:
Status: NEW
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: 33
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: systemd-maint
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-09-28 04:23 UTC by Paul Wouters
Modified: 2021-03-23 19:39 UTC (History)
16 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-08 12:37:48 UTC
Type: Bug


Attachments (Terms of Use)

Description Paul Wouters 2020-09-28 04:23:11 UTC
All my DNS stopped working after I closed a NetworkManager based openvpn connection.

Looking in /etc/resolv.conf I found an entry to 127.0.0.53

This

I will file a separate bug fedora 33 mistakenly installing systemd-resolved and me not being able to completely unstall systemd-resolved.

Comment 1 Zbigniew Jędrzejewski-Szmek 2020-09-28 05:40:52 UTC
Please provide the content of the file. "nameserver 127.0.0.53" is expected.

Comment 2 Fabrice 2020-10-27 09:35:27 UTC
I experience the same issue, after upgrading f32 -> f33, dns resolution for hosts on the vpn fails

$ cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0 trust-ad
search xxxxxxx.com

Comment 3 Zbigniew Jędrzejewski-Szmek 2020-10-27 12:19:51 UTC
> I experience the same issue, after upgrading f32 -> f33, dns resolution for hosts on the vpn fails

The original report was about not being able to resolve things after *closing* of a VPN connection.
If you cannot resolve hosts on the VPN connection, that it's rather unlikely it's the same issue.

Please, if you have a bug, then provide the necessary information. The bug template asks for this:
what software versions you were running, what is your setup, what actions where taken, what
the result was, and what you were expecting instead. Without that there just isn't much we can do.

Comment 4 Paul Wouters 2020-10-27 13:42:50 UTC
Let me give you additional information.


When I found:


# Generated by NetworkManager
search nohats.ca
nameserver 127.0.0.53

I changed it to:

# Leave me alone
search nohats.ca
nameserver 193.110.157.123

And stopped the systemd-resolved service.

When I bring up and then down a VPN connection using NM with openvpn (the redhat one), it changed to:

# Generated by NetworkManager
search nohats.ca
nameserver 127.0.0.53

Comment 5 Zbigniew Jędrzejewski-Szmek 2020-10-27 14:25:52 UTC
> # Generated by NetworkManager

It seems that NM is overwriting the file.
Does it help if you follow the instructions in https://askubuntu.com/a/623956/664164?

Comment 6 Fabrice 2020-10-27 14:53:22 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #5)
> > # Generated by NetworkManager
> 
> It seems that NM is overwriting the file.
> Does it help if you follow the instructions in
> https://askubuntu.com/a/623956/664164?

Yes, I had no time to fill the bug this morning so that's what I did (following https://askubuntu.com/questions/907246/how-to-disable-systemd-resolved-in-ubuntu ) and the problem is solved.
If you need I can go back to the original config after work and give all info you need

Comment 7 mhou 2020-12-01 10:14:29 UTC
Description of problem:
when enable systemd-resolved service and enable VPN, the tun interface will not use DNS server which VPN given.

Version-Release number of selected component (if applicable):
kernel version:
5.8.15-301.fc33.x86_64

systemd-resolved:
systemd-246.6-3.fc33.x86_64

How reproducible:
100%

Steps to Reproduce:
1.enable systemd-reslove service and check reslove.conf in etc.
systemctl status systemd-resolved 
● systemd-resolved.service - Network Name Resolution
     Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2020-12-01 17:26:55 CST; 553ms ago
       Docs: man:systemd-resolved.service(8)
             https://www.freedesktop.org/wiki/Software/systemd/resolved
             https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
             https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
   Main PID: 13987 (systemd-resolve)
     Status: "Processing requests..."
      Tasks: 1 (limit: 18884)
     Memory: 7.6M
        CPU: 85ms
     CGroup: /system.slice/systemd-resolved.service
             └─13987 /usr/lib/systemd/systemd-resolved

# cat /etc/reslove.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf

nameserver 127.0.0.53
options edns0 trust-ad
search redhat.com

2.access the VPN, check the vpn DNS

# resolvectl status
Global
       LLMNR setting: resolve             
MulticastDNS setting: no                  
  DNSOverTLS setting: no                  
      DNSSEC setting: no                  
    DNSSEC supported: no                  
Fallback DNS Servers: 1.1.1.1             
                      8.8.8.8             
                      1.0.0.1             
                      8.8.4.4             
                      2606:4700:4700::1111
                      2001:4860:4860::8888
                      2606:4700:4700::1001
                      2001:4860:4860::8844

Link 2 (enp0s31f6)
      Current Scopes: none
DefaultRoute setting: no  
       LLMNR setting: yes 
MulticastDNS setting: no  
  DNSOverTLS setting: no  
      DNSSEC setting: no  
    DNSSEC supported: no  

Link 3 (wlp0s20f3)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
DefaultRoute setting: yes                      
       LLMNR setting: yes                      
MulticastDNS setting: no                       
  DNSOverTLS setting: no                       
      DNSSEC setting: no                       
    DNSSEC supported: no                       
  Current DNS Server: 192.168.0.1              
         DNS Servers: 192.168.0.1              
          DNS Domain: ~.                       

Link 4 (virbr0)
      Current Scopes: none
DefaultRoute setting: no  
       LLMNR setting: yes 
MulticastDNS setting: no  
  DNSOverTLS setting: no  
      DNSSEC setting: no  
    DNSSEC supported: no  

Link 5 (virbr0-nic)
      Current Scopes: none
DefaultRoute setting: no  
       LLMNR setting: yes 
MulticastDNS setting: no  
  DNSOverTLS setting: no  
      DNSSEC setting: no  
    DNSSEC supported: no  

Link 7 (tun0)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
DefaultRoute setting: yes                      
       LLMNR setting: yes                      
MulticastDNS setting: no                       
  DNSOverTLS setting: no                       
      DNSSEC setting: no                       
    DNSSEC supported: no                       
  Current DNS Server: 10.72.17.5               
         DNS Servers: 10.72.17.5               
                      10.68.5.26               
          DNS Domain: redhat.com 

3. use traceroute to access google.com, and nslookup to check DNS server
# traceroute www.google.com
traceroute to www.google.com (74.86.228.110), 30 hops max, 60 byte packets
 1  192.168.0.1 (192.168.0.1)  0.952 ms  0.760 ms  0.704 ms
 2  192.168.1.1 (192.168.1.1)  2.268 ms  2.329 ms  2.326 ms
 3  100.92.0.1 (100.92.0.1)  25.733 ms  25.838 ms  25.808 ms
 4  36.112.252.185 (36.112.252.185)  4.802 ms  4.849 ms  5.334 ms
 5  bj141-135-174.bjtelecom.net (219.141.135.174)  8.749 ms bj141-135-162.bjtelecom.net (219.141.135.162)  7.147 ms bj141-135-174.bjtelecom.net (219.141.135.174)  8.692 ms
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

# nslookup www.google.com 
Server:		127.0.0.53
Address:	127.0.0.53#53

Non-authoritative answer:
Name:	www.google.com
Address: 31.13.68.1
Name:	www.google.com
Address: 2001::b33c:c004

# nslookup www.google.com 10.68.5.26
Server:		10.68.5.26
Address:	10.68.5.26#53

Non-authoritative answer:
Name:	www.google.com
Address: 74.125.130.106
Name:	www.google.com
Address: 74.125.130.147
Name:	www.google.com
Address: 74.125.130.104
Name:	www.google.com
Address: 74.125.130.105
Name:	www.google.com
Address: 74.125.130.99
Name:	www.google.com
Address: 74.125.130.103
Name:	www.google.com
Address: 2404:6800:4003:c01::93
Name:	www.google.com
Address: 2404:6800:4003:c01::68
Name:	www.google.com
Address: 2404:6800:4003:c01::63
Name:	www.google.com
Address: 2404:6800:4003:c01::6a

# nslookup www.google.com 10.72.17.5
Server:		10.72.17.5
Address:	10.72.17.5#53

Non-authoritative answer:
Name:	www.google.com
Address: 74.125.130.104
Name:	www.google.com
Address: 74.125.130.99
Name:	www.google.com
Address: 74.125.130.106
Name:	www.google.com
Address: 74.125.130.147
Name:	www.google.com
Address: 74.125.130.103
Name:	www.google.com
Address: 74.125.130.105
Name:	www.google.com
Address: 2404:6800:4003:c01::93
Name:	www.google.com
Address: 2404:6800:4003:c01::69
Name:	www.google.com
Address: 2404:6800:4003:c01::6a
Name:	www.google.com
Address: 2404:6800:4003:c01::68



4. check the route in system
# ip route
default via 192.168.0.1 dev wlp0s20f3 proto dhcp metric 600 
10.0.0.0/8 via 10.72.12.1 dev tun0 proto static metric 50 
10.72.12.0/22 dev tun0 proto kernel scope link src 10.72.13.141 metric 50 
64.18.0.0/20 via 10.72.12.1 dev tun0 proto static metric 50 
64.233.160.0/19 via 10.72.12.1 dev tun0 proto static metric 50 
66.102.0.0/20 via 10.72.12.1 dev tun0 proto static metric 50 
66.249.80.0/20 via 10.72.12.1 dev tun0 proto static metric 50 
72.14.192.0/18 via 10.72.12.1 dev tun0 proto static metric 50 
74.125.0.0/16 via 10.72.12.1 dev tun0 proto static metric 50 
77.95.64.0/23 via 10.72.12.1 dev tun0 proto static metric 50 
108.177.0.0/17 via 10.72.12.1 dev tun0 proto static metric 50 
119.254.120.114 via 192.168.0.1 dev wlp0s20f3 proto static metric 600 
142.250.0.0/15 via 10.72.12.1 dev tun0 proto static metric 50 
172.217.0.0/16 via 10.72.12.1 dev tun0 proto static metric 50 
172.253.0.0/16 via 10.72.12.1 dev tun0 proto static metric 50 
173.194.0.0/16 via 10.72.12.1 dev tun0 proto static metric 50 
192.168.0.0/24 dev wlp0s20f3 proto kernel scope link src 192.168.0.104 metric 600 
192.168.0.1 dev wlp0s20f3 proto static scope link metric 600 
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 metric 425 linkdown 
203.208.42.0/23 via 10.72.12.1 dev tun0 proto static metric 50 
207.126.144.0/20 via 10.72.12.1 dev tun0 proto static metric 50 
209.85.128.0/17 via 10.72.12.1 dev tun0 proto static metric 50 
216.58.192.0/19 via 10.72.12.1 dev tun0 proto static metric 50 
216.239.32.0/19 via 10.72.12.1 dev tun0 proto static metric 50 

Actual results:
1. After access the VPN, system can't access the google.com. System will use default DNS to reslove google.com.

Expected results:
1. After access the VPN, system can access the google.com. System will use DNS which VPN given to reslove google.com.

Additional info:
following https://askubuntu.com/questions/907246/how-to-disable-systemd-resolved-in-ubuntu, The problem can be avioded.

Comment 8 Petr Menšík 2021-02-20 14:04:29 UTC
I would only guess resolvconf binary is responsible. systemd-resolved is restarted every time resolvectl is executed. resolvconf is just link to resolvectl.

It might help instead just stopping to disable systemd-resolved: systemctl disable --now systemd-resolved

Comment 9 Villy Kruse 2021-02-20 17:05:08 UTC
(In reply to Paul Wouters from comment #0)
> All my DNS stopped working after I closed a NetworkManager based openvpn
> connection.
> 
> Looking in /etc/resolv.conf I found an entry to 127.0.0.53
> 
> This
> 
> I will file a separate bug fedora 33 mistakenly installing systemd-resolved
> and me not being able to completely unstall systemd-resolved.

If you add following to /etc/NetworkManager/NetworkManager.conf

[main]
dns=none
systemd-resolved=false

Better still, add these line to a new file in /etc/NetworkManager/conf.d/

Then make sure that /etc/resolv.conf is a regular file.  Then neither NetworkManager, nor systemd-resolved will mess with your resolv.conf file.

Also remove the word "resolve" from /etc/nsswitch.conf
man NetworkManager.conf for further detail.

Comment 10 Zbigniew Jędrzejewski-Szmek 2021-03-23 18:14:34 UTC
(In reply to mhou from comment #7)
> Description of problem:
> when enable systemd-resolved service and enable VPN, the tun interface will
> not use DNS server which VPN given.

Thank you for the detailed report. But I'm not seeing any actual error in the output you posted.
Please note that dns routing (i.e. to which interfaces systemd-resolved sends a query) is completely
unrelated to how the kernel routes IP packets. So commands like 'ip route' and 'traceroute' show
information which is not related to resolved operation, they describe something that happens at
a later step, once the name has already been resolved to an IP address.

> 2.access the VPN, check the vpn DNS
> 
> # resolvectl status
...
> Link 3 (wlp0s20f3)
...
>   Current DNS Server: 192.168.0.1              
>          DNS Servers: 192.168.0.1              
>           DNS Domain: ~.                       
...
> Link 7 (tun0)
>       Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
> DefaultRoute setting: yes                      
...
>   Current DNS Server: 10.72.17.5               
>          DNS Servers: 10.72.17.5               
>                       10.68.5.26               
>           DNS Domain: redhat.com 

This configuration means that tun0 should be used for any name ending in redhat.com,
and wlp0s20f3 should be used for all other names. I don't see anything that would contradict
this in the output you posted.

Comment 11 Zbigniew Jędrzejewski-Szmek 2021-03-23 18:22:40 UTC
> (In reply to Paul Wouters from comment #0)
> > All my DNS stopped working after I closed a NetworkManager based openvpn
> > connection.
> > 
> > Looking in /etc/resolv.conf I found an entry to 127.0.0.53
> > 
> > This
> > 
> > I will file a separate bug fedora 33 mistakenly installing systemd-resolved
> > and me not being able to completely unstall systemd-resolved.
> 
> If you add following to /etc/NetworkManager/NetworkManager.conf
> 
> [main]
> dns=none
> systemd-resolved=false
> 
> Better still, add these line to a new file in /etc/NetworkManager/conf.d/
> 
> Then make sure that /etc/resolv.conf is a regular file.  Then neither
> NetworkManager, nor systemd-resolved will mess with your resolv.conf file.

This isn't really related to systemd-resolved, but to how NM manages resolv.conf. And
there seems to be no bug.

Comment 12 Villy Kruse 2021-03-23 19:39:03 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #11)
> > (In reply to Paul Wouters from comment #0)
> > > All my DNS stopped working after I closed a NetworkManager based openvpn
> > > connection.
> > > 
> > > Looking in /etc/resolv.conf I found an entry to 127.0.0.53
> > > 
> > > This
> > > 
> > > I will file a separate bug fedora 33 mistakenly installing systemd-resolved
> > > and me not being able to completely unstall systemd-resolved.
> > 
> > If you add following to /etc/NetworkManager/NetworkManager.conf
> > 
> > [main]
> > dns=none
> > systemd-resolved=false
> > 
> > Better still, add these line to a new file in /etc/NetworkManager/conf.d/
> > 
> > Then make sure that /etc/resolv.conf is a regular file.  Then neither
> > NetworkManager, nor systemd-resolved will mess with your resolv.conf file.
> 
> This isn't really related to systemd-resolved, but to how NM manages
> resolv.conf. And
> there seems to be no bug.

That is correct; there is no bug with respect to resolv.conf.


Note You need to log in before you can comment on or make changes to this bug.