Bug 1933863 - Broken hostname setting via reverse lookup with NetworkManager + systemd-resolved
Summary: Broken hostname setting via reverse lookup with NetworkManager + systemd-reso...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: NetworkManager
Version: 34
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Lubomir Rintel
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-03-01 21:24 UTC by Jonathan Lebon
Modified: 2021-03-19 20:14 UTC (History)
11 users (show)

Fixed In Version: NetworkManager-1.30.2-1.fc34
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-03-19 20:14:03 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
freedesktop.org Gitlab NetworkManager NetworkManager merge_requests 770 0 None None None 2021-03-04 15:25:47 UTC

Description Jonathan Lebon 2021-03-01 21:24:06 UTC
Description of problem:

In Fedora CoreOS 34, we've noticed that hostname setting via reverse DNS doesn't work because of what seems to be a conceptual broken dependency between the lookup and updating DNS.

In my test, I'm using static IP to force NM not to get the hostname from DHCP and to fallback to reverse DNS (because AFAICT the NM in f34 doesn't yet have `hostname.from-dhcp` from https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/669). Here's the connection file I'm using in a libvirt VM:

[ipv4]
address1=192.168.122.171/24,192.168.122.1
dns=192.168.122.1;
dns-search=
method=manual

I have dnsmasq set up on my host to return `foobar` when querying the IP 192.168.122.171 (by adding the relevant line to `/etc/hosts`).

Here's the relevant logs from NetworkManager (with trace logging):

```
Mar 01 21:03:18 localhost NetworkManager[674]: <info>  [1614632598.9301] manager: NetworkManager state is now CONNECTED_SITE
Mar 01 21:03:18 localhost NetworkManager[674]: <info>  [1614632598.9308] policy: set 'Wired connection 1' (ens3) as default for IPv4 routing and DNS
Mar 01 21:03:18 localhost NetworkManager[674]: <debug> [1614632598.9310] manager: PrimaryConnection now Wired connection 1
Mar 01 21:03:18 localhost NetworkManager[674]: <trace> [1614632598.9320] policy: set-hostname: updating hostname (routing and dns)
Mar 01 21:03:18 localhost NetworkManager[674]: <trace> [1614632598.9321] policy: get-hostname: "localhost" (from dbus)
...
Mar 01 21:03:18 localhost NetworkManager[674]: <trace> [1614632598.9328] device[3c55bb4a7b7a290e] (ens3): hostname-from-dns: ipv4 resolver state wait-address, old address (null), new address 192.168.122.171
Mar 01 21:03:18 localhost NetworkManager[674]: <trace> [1614632598.9329] device[3c55bb4a7b7a290e] (ens3): hostname-from-dns: starting lookup for address 192.168.122.171
Mar 01 21:03:18 localhost NetworkManager[674]: <debug> [1614632598.9358] dns-mgr: (update_routing_and_dns): DNS configuration changed
Mar 01 21:03:18 localhost NetworkManager[674]: <debug> [1614632598.9361] dns-mgr: (update_routing_and_dns): no DNS changes to commit (1)
Mar 01 21:03:18 localhost NetworkManager[674]: <debug> [1614632598.9362] dns-mgr: (device_state_changed): DNS configuration changed
Mar 01 21:03:18 localhost NetworkManager[674]: <debug> [1614632598.9363] dns-mgr: (device_state_changed): committing DNS changes (0)
Mar 01 21:03:18 localhost NetworkManager[674]: <debug> [1614632598.9364] dns-mgr: update-dns: not updating resolv.conf
Mar 01 21:03:18 localhost NetworkManager[674]: <trace> [1614632598.9365] dns-mgr: config:      100 best    v4 2     : 192.168.122.1
Mar 01 21:03:18 localhost NetworkManager[674]: <trace> [1614632598.9367] dns-mgr: plugin: add domain <auto-default> (i=2, p=100)
Mar 01 21:03:18 localhost NetworkManager[674]: <trace> [1614632598.9368] dns-mgr: plugin: settings: ifindex=2, priority=100, default-route=1, search=, reverse=122.168.192.in-addr.arpa,171.122.168.192.in-addr.arpa
Mar 01 21:03:18 localhost NetworkManager[674]: <debug> [1614632598.9370] dns-mgr: update-dns: updating plugin systemd-resolved
Mar 01 21:03:18 localhost NetworkManager[674]: <trace> [1614632598.9371] dns-sd-resolved[d26e64a84b337b7a]: send-updates: start 5 requests
Mar 01 21:03:18 localhost systemd-resolved[638]: ens3: Bus client set default route setting: yes
Mar 01 21:03:18 localhost systemd-resolved[638]: ens3: Bus client set DNS server list to: 192.168.122.1
Mar 01 21:03:18 localhost NetworkManager[674]: <trace> [1614632598.9434] dns-mgr: update-resolv-no-stub: '/run/NetworkManager/no-stub-resolv.conf' successfully written
Mar 01 21:03:18 localhost NetworkManager[674]: <trace> [1614632598.9438] dns-mgr: update-resolv-conf: write internal file /run/NetworkManager/resolv.conf succeeded
Mar 01 21:03:18 localhost NetworkManager[674]: <trace> [1614632598.9439] dns-mgr: current configuration: [{'nameservers': <['192.168.122.1']>, 'interface': <'ens3'>, 'priority': <100>, 'vpn': <false>}]
...
Mar 01 21:03:18 localhost NetworkManager[674]: <debug> [1614632598.9867] device[3c55bb4a7b7a290e] (ens3): hostname-from-dns: lookup done for 192.168.122.171, result "localhost"
...
Mar 01 21:03:18 localhost NetworkManager[674]: <trace> [1614632598.9887] policy: set-hostname: hostname already set to 'localhost' (from address lookup)
```

So what I'm seeing here is that NM is starting the reverse DNS lookup *before* it actually tells systemd-resolved what the new DNS server is, hence causing address lookup to fail (and stay on localhost).

If I do `systemctl restart NetworkManager`, NM correctly finds the `foobar` hostname from reverse DNS because DNS has already been set up:

```
Mar 01 21:03:34 localhost NetworkManager[796]: <trace> [1614632614.7156] device[4fe1b4205a9fa185] (ens3): hostname-from-dns: ipv4 resolver state wait-address, old address (null), new address 192.168.122.171
Mar 01 21:03:34 localhost NetworkManager[796]: <trace> [1614632614.7157] device[4fe1b4205a9fa185] (ens3): hostname-from-dns: starting lookup for address 192.168.122.171
...
Mar 01 21:03:34 localhost NetworkManager[796]: <debug> [1614632614.7367] device[4fe1b4205a9fa185] (ens3): hostname-from-dns: lookup done for 192.168.122.171, result "foobar"
Mar 01 21:03:34 localhost NetworkManager[796]: <trace> [1614632614.7369] policy: set-hostname: updating hostname (lookup finished)
Mar 01 21:03:34 localhost NetworkManager[796]: <trace> [1614632614.7370] policy: get-hostname: "localhost" (from dbus)
Mar 01 21:03:34 localhost NetworkManager[796]: <trace> [1614632614.7372] policy: device hostname info:
Mar 01 21:03:34 localhost NetworkManager[796]: <trace> [1614632614.7376] policy:   - prio:  100 ipv4 (def) dhcp  dns  dev:ens3
Mar 01 21:03:34 localhost NetworkManager[796]: <trace> [1614632614.7377] policy:   - prio:  100 ipv6       dhcp  dns  dev:ens3
Mar 01 21:03:34 localhost NetworkManager[796]: <trace> [1614632614.7378] device[4fe1b4205a9fa185] (ens3): hostname-from-dns: ipv4 resolver state done, old address 192.168.122.171, new address 192.168.122.171
Mar 01 21:03:34 localhost NetworkManager[796]: <trace> [1614632614.7380] policy: get-hostname: "localhost" (from dbus)
Mar 01 21:03:34 localhost NetworkManager[796]: <info>  [1614632614.7381] policy: set-hostname: set hostname to 'foobar' (from address lookup)
...
Mar 01 21:03:34 foobar systemd-hostnamed[685]: Hostname set to <foobar> (transient)
Mar 01 21:03:34 foobar systemd-resolved[638]: System hostname changed to 'foobar'.
```

So it seems like NM needs to be tweaked to delay hostname-from-dns until after DNS has been updated?

Version-Release number of selected component (if applicable):

# rpm -q NetworkManager systemd
NetworkManager-1.30.0-1.fc34.x86_64
systemd-248~rc2-1.fc34.x86_64

How reproducible:

100%

Steps to Reproduce:
1. See above

Actual results:

Hostname setting via reverse DNS doesn't work until NetworkManager.service is restarted.

Expected results:

Hostname setting via reverse DNS works.

Additional info:

Fedora CoreOS currently neuters systemd-resolved so that it just directly points to the underlying DNS server. We also set the fallback hostname to `localhost` and the `nsswitch.conf` config to match f32 (where `dns` is ordered higher than `myhostname` and `resolve` isn't in the list)  See all that here (with links to background):

https://github.com/coreos/fedora-coreos-config/blob/72a2e7e5b89746b7ea320ee9da057bd8ccba52e0/manifests/fedora-coreos-base.yaml#L82-L110

Comment 1 Jonathan Lebon 2021-03-02 17:34:42 UTC
OK right, I think what's going on here is that in the systemd-resolved case, NM considers /etc/resolv.conf unmanaged. This means that when the dns-manager updates systemd-resolved with the new DNS config, it doesn't emit a CONFIG_CHANGED signal. And so the hostname lookup isn't restarted.

So... should that code be relaxed a bit so that CONFIG_CHANGED is emitted even if we're not managing /etc/resolv.conf but we updated a caching plugin? Something like:

diff --git a/src/core/dns/nm-dns-manager.c b/src/core/dns/nm-dns-manager.c
index d30c3fa549..5e03b102b7 100644
--- a/src/core/dns/nm-dns-manager.c
+++ b/src/core/dns/nm-dns-manager.c
@@ -1815,8 +1815,8 @@ plugin_skip:;
                            NM_DNS_MANAGER_RESOLV_CONF_MAN_UNMANAGED);
     }

-    /* signal that resolv.conf was changed */
-    if (do_update && result == SR_SUCCESS)
+    /* signal that DNS resolution configs were changed */
+    if ((do_update || caching) && result == SR_SUCCESS)
         g_signal_emit(self, signals[CONFIG_CHANGED], 0);

     nm_clear_pointer(&priv->config_variant, g_variant_unref);

Comment 2 Beniamino Galvani 2021-03-04 10:08:29 UTC
Hi Jonathan, thanks for looking into this issue.

The patch looks correct to me. Would you like to open a merge request at [1] for it or do you prefer if I do it?

[1] https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests

Comment 3 Jonathan Lebon 2021-03-04 15:25:48 UTC
Thanks for confirming Beniamino! I've tested the patch and confirmed it fixes the issue. Posted an MR.

Comment 4 Jonathan Lebon 2021-03-05 15:18:07 UTC
Thanks for merging!

Are you planning to rebase NM in f34 before GA? If not, would it be possible to backport this patch to f34? I can draft up the dist-git PR for that if you'd like.

Comment 5 Beniamino Galvani 2021-03-08 09:20:26 UTC
We will do a 1.30.2 minor release this week and we will rebase the F34 package to this new version (which includes your commit).

Comment 6 Fedora Update System 2021-03-12 17:07:01 UTC
FEDORA-2021-fb173bc82f has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-fb173bc82f

Comment 7 Fedora Update System 2021-03-12 18:57:22 UTC
FEDORA-2021-fb173bc82f has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-fb173bc82f`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-fb173bc82f

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 8 Fedora Update System 2021-03-19 20:14:03 UTC
FEDORA-2021-fb173bc82f has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.