Bug 1992515

what gives `resolvectl` with 1.26.0-X (RHEL8.3) vs 1.30.0-Y (RHEL8.4)?

Also useful would be: `ip addr`, `ip -4 route show table all`, `ip -6 route show table all`, `ip -4 rule`, `ip -6 rule`.

Created attachment 1814334 [details]
resolvectl while NetworkManager-1.26.0 is used

Created attachment 1814335 [details]
resolvectl while NetworkManager-1.30.0 is used

The diff is 1.26.0 has "~." in DNS domain, but not 1.30.0:

# diff resolvectl.NM-1.26.0 resolvectl.NM-1.30.0 
47,48c47
<           DNS Domain: ~.
<                       libvirt
---
>           DNS Domain: libvirt

To confirm, is the problem avoided if you explicitly add "~." as a search domain in NetworkManager?

That should also lead to the same resolvectl output.




While testing, please collect full level=TRACE logs. See https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/blob/main/contrib/fedora/rpm/NetworkManager.conf#L27 for hints about logging.

Since 1.30, NetworkManager no longer automatically adds ~. to the search domain. You could get that by setting ipv4.dns-priority to a negative value (but that should not be necessary in this case). Instead, it relies on the "DefaultRoute" setting. On RHEL-8, systemd-resolved does not support to have DefaultRoute configurable for NetworkManager (which we should see in the trace logfile). But it is supposed to still work, because systemd-resolved would default to have the behavior as if DefaultRoute were enabled (which is what we would want here).

As you noticed, the different between the two NM versions is in how the domains are configured in systemd-resolved.

Python's socket.getfqdn() computes the result with the following pseudo-code:

    name = socket.gethostname()         #  => foobar
    addr = socket.getaddrinfo(name)     #  => 172.25.1.99
    fqdn = socket.gethostbyaddr(addr)   #  => foobar.libvirt

Note that this is different from what "hostname -f" does, which is equivalent to:

    r = getaddrinfo(gethostname(), NULL, flags=AI_CANONNAME, ...)
    fqdn = r->ai_cannoname

"hostname -f" doesn't seem to have the issue.

The cause of the problem is explained by this commands:

    [root@foobar ~]# resolvectl status veth0
    Link 5 (veth0)
          Current Scopes: DNS LLMNR/IPv4
           LLMNR setting: yes
    MulticastDNS setting: no
      DNSOverTLS setting: no
          DNSSEC setting: no
        DNSSEC supported: no
      Current DNS Server: 172.25.1.1
             DNS Servers: 172.25.1.1
              DNS Domain: libvirt

    [root@foobar ~]# resolvectl query 172.25.1.99
    172.25.1.99%veth0: foobar

    -- Information acquired via protocol LLMNR/IPv4 in 2.4ms.
    -- Data is authenticated: yes

    [root@foobar ~]# resolvectl domain veth0 libvirt ~.
    [root@foobar ~]# resolvectl status veth0
    Link 5 (veth0)
          Current Scopes: DNS LLMNR/IPv4
           LLMNR setting: yes
    MulticastDNS setting: no
      DNSOverTLS setting: no
          DNSSEC setting: no
	DNSSEC supported: no
      Current DNS Server: 172.25.1.1
             DNS Servers: 172.25.1.1
              DNS Domain: libvirt
                          ~.

    [root@foobar ~]# resolvectl query 172.25.1.99
    172.25.1.99: foobar.libvirt

    -- Information acquired via protocol DNS in 4.9ms.
    -- Data is authenticated: no

Basically, where there is no "~." domain, systemd-resolved uses LLMNR to resolve the address. In that case, the following packets are seen on the network:

    16:13:03.472419 IP (tos 0x0, ttl 64, id 61182, offset 0, flags [DF], proto UDP (17), length 81)
        172.25.1.99.56216 > 172.25.1.1.53: 14265+ [1au] PTR? 99.1.25.172.in-addr.arpa. (53)

    16:13:03.472494 IP (tos 0x0, ttl 64, id 61183, offset 0, flags [DF], proto UDP (17), length 81)
        172.25.1.99.46958 > 172.25.1.1.53: 53506+ [1au] PTR? 99.1.25.172.in-addr.arpa. (53)

    16:13:03.472663 IP (tos 0x0, ttl 64, id 11871, offset 0, flags [DF], proto UDP (17), length 109)
        172.25.1.1.53 > 172.25.1.99.56216: 14265* 1/0/1 99.1.25.172.in-addr.arpa. PTR foobar.libvirt. (81)

    16:13:03.472686 IP (tos 0xc0, ttl 64, id 62381, offset 0, flags [none], proto ICMP (1), length 137)
        172.25.1.99 > 172.25.1.1: ICMP 172.25.1.99 udp port 56216 unreachable, length 117
        IP (tos 0x0, ttl 64, id 11871, offset 0, flags [DF], proto UDP (17), length 109)
        172.25.1.1.53 > 172.25.1.99.56216: 14265* 1/0/1 99.1.25.172.in-addr.arpa. PTR foobar.libvirt. (81)

    16:13:03.472702 IP (tos 0x0, ttl 64, id 11872, offset 0, flags [DF], proto UDP (17), length 109)
        172.25.1.1.53 > 172.25.1.99.46958: 53506* 1/0/1 99.1.25.172.in-addr.arpa. PTR foobar.libvirt. (81)

    16:13:03.472708 IP (tos 0xc0, ttl 64, id 62382, offset 0, flags [none], proto ICMP (1), length 137)
        172.25.1.99 > 172.25.1.1: ICMP 172.25.1.99 udp port 46958 unreachable, length 117
        IP (tos 0x0, ttl 64, id 11872, offset 0, flags [DF], proto UDP (17), length 109)
        172.25.1.1.53 > 172.25.1.99.46958: 53506* 1/0/1 99.1.25.172.in-addr.arpa. PTR foobar.libvirt. (81)

It looks like systemd-resolved sends two DNS queries and then closes the socket, possibly because at the same time it already gets a synthetic answer from LLMNR, and this causes the two ICMP port-unreachable packets.

Created attachment 1816077 [details]
Reproducer

@Renaud, a possible workaround is to disable LLMNR in systemd-resolved; either globally by setting LLMNR=no in /etc/systemd/resolved.conf, or per-interface by setting 'connection.llmnr=no' in the NM connection.

The RFC of LLMNR indicating the DNS server should return both first section of hostname and FQDN.

Beniamino is investigating this and working with systemd-resovled.

This seems a systemd-resolved issue. When resolving a local address to
name via systemd-resolved, a single-label name is returned via LLMNR:

  [root@localhost ~]# ip a show dev veth0
  5: veth0@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
      link/ether 00:11:22:33:44:55 brd ff:ff:ff:ff:ff:ff link-netns ns1
      inet 172.25.1.99/24 brd 172.25.1.255 scope global dynamic noprefixroute veth0
         valid_lft 65sec preferred_lft 65sec

  [root@localhost ~]# resolvectl status veth0
  Link 5 (veth0)
        Current Scopes: DNS LLMNR/IPv4
         LLMNR setting: yes
  MulticastDNS setting: no
    DNSOverTLS setting: no
	DNSSEC setting: no
      DNSSEC supported: no
    Current DNS Server: 172.25.1.1
           DNS Servers: 172.25.1.1
            DNS Domain: libvirt

  [root@localhost ~]# resolvectl query 172.25.1.99
  172.25.1.99%veth0: foobar <---------------------------------- single-label

  -- Information acquired via protocol LLMNR/IPv4 in 10.9ms.
  -- Data is authenticated: yes

  [root@localhost ~]# rpm -q systemd
  systemd-239-44.el8.x86_64


On more recent version of systemd from F34, the FQDN from DNS is
always returned:

  root@f34  ~ # rpm -q systemd
  systemd-248.7-1.fc34.x86_64

  root@f34  ~ # resolvectl status veth0
  Link 39 (veth0)
      Current Scopes: DNS LLMNR/IPv4
           Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
  Current DNS Server: 172.25.1.1
         DNS Servers: 172.25.1.1
          DNS Domain: libvirt

  root@f34  ~ # resolvectl query 172.25.1.99
  172.25.1.99: foobar.libvirt                    -- link: veth0

  -- Information acquired via protocol DNS in 4.3ms.
  -- Data is authenticated: no; Data was acquired via local or encrypted transport: no
  -- Data from: network

I also tried disabling DefaultRoute for the link, but the result is
still the same.

Also, when forcing the query via LLMNR both single-label and FQDN
results are returned on F34:

  root@f34  ~ # resolvectl -p llmnr query 172.25.1.99
  172.25.1.99: foobar                            -- link: veth0
               foobar.local                      -- link: veth0

  -- Information acquired via protocol LLMNR/IPv4 in 151.0ms.
  -- Data is authenticated: yes; Data was acquired via local or encrypted transport: yes
  -- Data from: synthetic

So it seems that the systemd-resolved version in RHEL 8.4 is returning the wrong result.

Reassigning to systemd for investigation.

So, if I understand this correctly, the problem is that LLMNR is used in preference of DNS for resolving local addresses.

*** This bug has been marked as a duplicate of bug 1739689 ***