Bug 2148301

Summary: Version dnsmasq-2.86-5 breaks properly processing /etc/NetworkManager/dnsmasq.d/
Product: [Fedora] Fedora Reporter: Petr Menšík <pemensik>
Component: dnsmasqAssignee: Petr Menšík <pemensik>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 37CC: aegorenkov.91, aegorenk, ahwatts, code, dns-sig, dougsland, extras-qa, jsiero, pemensik, rmetrich, rolf, veillard
Target Milestone: ---Keywords: Regression, Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
URL: https://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2022q2/016442.html
Whiteboard:
Fixed In Version: dnsmasq-2.87-3.fc37 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 2061944 Environment:
Last Closed: 2022-12-08 02:05:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2061944    
Bug Blocks:    

Description Petr Menšík 2022-11-24 23:05:41 UTC 
+++ This bug was initially created as a clone of Bug #2061944 +++

Description of problem:
Version dnsmasq-2.86-5 breaks properly processing /etc/NetworkManager/dnsmasq.d/

Version-Release number of selected component (if applicable):
dnsmasq-2.86-5

How reproducible:
100%

Steps to Reproduce:
1. dnf update
2. no static dns as before
3.

Actual results:
DNS resolving specific subdomains fails

Expected results:
DNS resolving specific subdomains succeeds

Additional info (obfuscated):
our static setup:
cat /etc/NetworkManager/dnsmasq.d/static.conf 
listen-address=127.0.0.35
server=/local.example.com/10.120.100.51
server=/local.example.com/10.120.100.52
server=/3.2.1.in-addr.arpa/192.168.0.1
server=/120.10.in-addr.arpa/10.120.100.51
server=/120.10.in-addr.arpa/10.120.100.52
addn-hosts=/usr/share/laptop/dnsmasq.hosts



journal logs:
Mar 08 16:30:11 localhost.localdomain dnsmasq[1629]: DBus support enabled: connected to system bus
Mar 08 16:30:11 localhost.localdomain dnsmasq[1629]: using nameserver 10.120.100.51#53 for domain local.example.com
Mar 08 16:30:11 localhost.localdomain dnsmasq[1629]: using nameserver 10.120.100.52#53 for domain local.example.com
Mar 08 16:30:11 localhost.localdomain dnsmasq[1629]: using nameserver 192.168.0.1#53 for domain 3.2.1.in-addr.arpa
Mar 08 16:30:11 localhost.localdomain dnsmasq[1629]: using nameserver 10.120.100.51#53 for domain 120.10.in-addr.arpa
Mar 08 16:30:11 localhost.localdomain dnsmasq[1629]: using nameserver 10.120.100.52#53 for domain 120.10.in-addr.arpa
Mar 08 16:30:11 localhost.localdomain dnsmasq[1629]: read /usr/share/laptop/dnsmasq.hosts - 2 addresses
Mar 08 16:30:11 localhost.localdomain dnsmasq[1629]: setting upstream servers from DBus
Mar 08 16:30:11 localhost.localdomain dnsmasq[1629]: using nameserver 10.120.100.51#53 for domain local.example.com
Mar 08 16:30:11 localhost.localdomain dnsmasq[1629]: using nameserver 10.120.100.52#53 for domain local.example.com
Mar 08 16:30:11 localhost.localdomain dnsmasq[1629]: using nameserver 192.168.0.1#53 for domain 3.2.1.in-addr.arpa
Mar 08 16:30:11 localhost.localdomain dnsmasq[1629]: using nameserver 10.120.100.51#53 for domain 120.10.in-addr.arpa
Mar 08 16:30:11 localhost.localdomain dnsmasq[1629]: using nameserver 10.120.100.52#53 for domain 120.10.in-addr.arpa
Mar 08 16:30:11 localhost.localdomain dnsmasq[1629]: using nameserver 192.168.2.1#53(via wlp2s0)
Mar 08 16:30:11 localhost.localdomain dnsmasq[1629]: using nameserver 192.168.2.1#53 for domain 2.168.192.in-addr.arpa
Mar 08 16:30:11 localhost.localdomain dnsmasq[1629]: using nameserver 192.168.2.1#53 for domain 123.2.168.192.in-addr.arpa
Mar 08 16:30:11 localhost.localdomain dnsmasq[1629]: read /usr/share/laptop/dnsmasq.hosts - 2 addresses
Mar 08 16:30:11 localhost.localdomain dnsmasq[1629]: setting upstream servers from DBus
Mar 08 16:30:11 localhost.localdomain dnsmasq[1629]: using nameserver 192.168.2.1#53(via wlp2s0)
Mar 08 16:30:11 localhost.localdomain dnsmasq[1629]: using nameserver 192.168.2.1#53 for domain 2.168.192.in-addr.arpa
Mar 08 16:30:11 localhost.localdomain dnsmasq[1629]: using nameserver 192.168.2.1#53 for domain 123.2.168.192.in-addr.arpa
Mar 08 16:30:11 localhost.localdomain dnsmasq[1629]: read /usr/share/tb-laptop/dnsmasq.hosts - 2 addresses

--- Additional comment from Rolf Fokkens on 2022-03-08 21:25:47 CET ---

We built 2.87test8 as an RPM (dnsmasq-2.87-0.test8) and tried this: same issue. So the issue is in 2.87test and has been backported to the 2.86 package.

--- Additional comment from Andrew Watts on 2022-05-27 21:43:22 CEST ---

This issue still appears to be present in Fedora 36, with dnsmasq-2.86-6.fc36.x86_64.

I use dnsmasq to refer the domains rn-default and orgs-default to refer to dnsmasq instances running inside Docker networks, so that I can access servers within the Docker networks by name.

My config in /etc/NetworkManager/dnsmasq.d/*.conf:

server=/rn-default/172.21.2.2
server=/orgs-default/172.22.2.2

As you can see from the logs below following a systemctl restart NetworkManager, the servers I've specified are present when the config is initially loaded, and even after the first batch of new servers come in from DBus, when my wired connection (enp0s20f0u4u3, through a USB-C docking station) gets activated. But the second time it gets an update from DBus, when my wireless connection (wlp4s0) is activated, the servers from the config file are no longer present. Then when IPv6 gets set up, the servers are still missing.

May 27 15:24:52 ironic-mullet.example.com dnsmasq[134667]: started, version 2.86 cachesize 400
May 27 15:24:52 ironic-mullet.example.com dnsmasq[134667]: compile time options: IPv6 GNU-getopt DBus no-UBus no-i18n IDN2 DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth cryptohash DNSSEC loop-detect inotify dumpfile
May 27 15:24:52 ironic-mullet.example.com dnsmasq[134667]: chown of PID file /run/NetworkManager/dnsmasq.pid failed: Operation not permitted
May 27 15:24:52 ironic-mullet.example.com dnsmasq[134667]: DBus support enabled: connected to system bus
May 27 15:24:52 ironic-mullet.example.com dnsmasq[134667]: using nameserver 172.21.2.2#53 for domain rn-default
May 27 15:24:52 ironic-mullet.example.com dnsmasq[134667]: using nameserver 172.22.2.2#53 for domain orgs-default
May 27 15:24:52 ironic-mullet.example.com dnsmasq[134667]: cleared cache
May 27 15:24:52 ironic-mullet.example.com dnsmasq[134667]: setting upstream servers from DBus
May 27 15:24:52 ironic-mullet.example.com dnsmasq[134667]: using nameserver 172.21.2.2#53 for domain rn-default
May 27 15:24:52 ironic-mullet.example.com dnsmasq[134667]: using nameserver 172.22.2.2#53 for domain orgs-default
May 27 15:24:52 ironic-mullet.example.com dnsmasq[134667]: using nameserver 192.168.1.1#53(via enp0s20f0u4u3)
May 27 15:24:52 ironic-mullet.example.com dnsmasq[134667]: using nameserver 192.168.1.1#53 for domain example.net
May 27 15:24:52 ironic-mullet.example.com dnsmasq[134667]: using nameserver 192.168.1.1#53 for domain 1.168.192.in-addr.arpa
May 27 15:24:52 ironic-mullet.example.com dnsmasq[134667]: cleared cache
May 27 15:24:55 ironic-mullet.example.com dnsmasq[134667]: setting upstream servers from DBus
May 27 15:24:55 ironic-mullet.example.com dnsmasq[134667]: using nameserver 192.168.1.1#53(via enp0s20f0u4u3)
May 27 15:24:55 ironic-mullet.example.com dnsmasq[134667]: using nameserver 192.168.1.1#53 for domain example.net
May 27 15:24:55 ironic-mullet.example.com dnsmasq[134667]: using nameserver 192.168.1.1#53 for domain 1.168.192.in-addr.arpa
May 27 15:24:55 ironic-mullet.example.com dnsmasq[134667]: using nameserver 192.168.1.1#53(via wlp4s0)
May 27 15:24:55 ironic-mullet.example.com dnsmasq[134667]: using nameserver 192.168.1.1#53 for domain example.net
May 27 15:24:55 ironic-mullet.example.com dnsmasq[134667]: using nameserver 192.168.1.1#53 for domain 1.168.192.in-addr.arpa
May 27 15:24:55 ironic-mullet.example.com dnsmasq[134667]: cleared cache
May 27 15:24:57 ironic-mullet.example.com dnsmasq[134667]: setting upstream servers from DBus
May 27 15:24:57 ironic-mullet.example.com dnsmasq[134667]: using nameserver 192.168.1.1#53(via enp0s20f0u4u3)
May 27 15:24:57 ironic-mullet.example.com dnsmasq[134667]: using nameserver 192.168.1.1#53 for domain example.net
May 27 15:24:57 ironic-mullet.example.com dnsmasq[134667]: using nameserver 192.168.1.1#53 for domain 1.168.192.in-addr.arpa
May 27 15:24:57 ironic-mullet.example.com dnsmasq[134667]: using nameserver fe80::hhhh:hhhh:hhhh:hhhh%wlp4s0#53
May 27 15:24:57 ironic-mullet.example.com dnsmasq[134667]: using nameserver fe80::hhhh:hhhh:hhhh:hhhh%wlp4s0#53 for domain 5.7.5.a.7.0.6.6.0.8.0.6.3.0.6.2.ip6.arpa
May 27 15:24:57 ironic-mullet.example.com dnsmasq[134667]: using nameserver fe80::hhhh:hhhh:hhhh:hhhh%wlp4s0#53 for domain 0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa
May 27 15:24:57 ironic-mullet.example.com dnsmasq[134667]: using nameserver 192.168.1.1#53(via wlp4s0)
May 27 15:24:57 ironic-mullet.example.com dnsmasq[134667]: using nameserver 192.168.1.1#53 for domain example.net
May 27 15:24:57 ironic-mullet.example.com dnsmasq[134667]: using nameserver 192.168.1.1#53 for domain 1.168.192.in-addr.arpa
May 27 15:24:57 ironic-mullet.example.com dnsmasq[134667]: cleared cache
May 27 15:24:57 ironic-mullet.example.com dnsmasq[134667]: setting upstream servers from DBus
May 27 15:24:57 ironic-mullet.example.com dnsmasq[134667]: using nameserver 192.168.1.1#53(via enp0s20f0u4u3)
May 27 15:24:57 ironic-mullet.example.com dnsmasq[134667]: using nameserver 192.168.1.1#53 for domain example.net
May 27 15:24:57 ironic-mullet.example.com dnsmasq[134667]: using nameserver 192.168.1.1#53 for domain 1.168.192.in-addr.arpa
May 27 15:24:57 ironic-mullet.example.com dnsmasq[134667]: using nameserver gggg:gggg:gggg:gggg::1#53(via wlp4s0)
May 27 15:24:57 ironic-mullet.example.com dnsmasq[134667]: using nameserver gggg:gggg:gggg:gggg::1#53 for domain f.d.a.a.b.0.4.2.0.0.0.0.0.0.0.0.5.7.5.a.7.0.6.6.0.8.0.6.3.0.6.2.ip6.arpa
May 27 15:24:57 ironic-mullet.example.com dnsmasq[134667]: using nameserver gggg:gggg:gggg:gggg::1#53 for domain 5.7.5.a.7.0.6.6.0.8.0.6.3.0.6.2.ip6.arpa
May 27 15:24:57 ironic-mullet.example.com dnsmasq[134667]: using nameserver gggg:gggg:gggg:gggg::1#53 for domain 0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa
May 27 15:24:57 ironic-mullet.example.com dnsmasq[134667]: using nameserver fe80::hhhh:hhhh:hhhh:hhhh%wlp4s0#53
May 27 15:24:57 ironic-mullet.example.com dnsmasq[134667]: using nameserver fe80::hhhh:hhhh:hhhh:hhhh%wlp4s0#53 for domain f.d.a.a.b.0.4.2.0.0.0.0.0.0.0.0.5.7.5.a.7.0.6.6.0.8.0.6.3.0.6.2.ip6.arpa
May 27 15:24:57 ironic-mullet.example.com dnsmasq[134667]: using nameserver fe80::hhhh:hhhh:hhhh:hhhh%wlp4s0#53 for domain 5.7.5.a.7.0.6.6.0.8.0.6.3.0.6.2.ip6.arpa
May 27 15:24:57 ironic-mullet.example.com dnsmasq[134667]: using nameserver fe80::hhhh:hhhh:hhhh:hhhh%wlp4s0#53 for domain 0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa
May 27 15:24:57 ironic-mullet.example.com dnsmasq[134667]: using nameserver 192.168.1.1#53(via wlp4s0)
May 27 15:24:57 ironic-mullet.example.com dnsmasq[134667]: using nameserver 192.168.1.1#53 for domain example.net
May 27 15:24:57 ironic-mullet.example.com dnsmasq[134667]: using nameserver 192.168.1.1#53 for domain 1.168.192.in-addr.arpa
May 27 15:24:57 ironic-mullet.example.com dnsmasq[134667]: cleared cache

(some of the domain names and IP addresses have been changed here to anonymize things)

--- Additional comment from Petr Menšík on 2022-06-04 22:44:50 CEST ---

So if I understand correctly, dnsmasq is integrated with Network Manager. Dbus configuration changes not only configuration previously set by DBus interface, but also static configuration set in file. Is that correct?

From comment #2 it seems static configuration survives first configuration via DBus, but disappears on the second. Or when two interfaces were used.

--- Additional comment from Jasper Siero on 2022-06-07 10:47:42 CEST ---

The only solution for us is to remove Patch30 and rebuild the package for ourself but this is ofcourse a temporarily solution. Every new release of the package breaks our system until we fix it again, recently with the upgrade to Fedora 36. But we are getting the hang of it.

--- Additional comment from Andrew Watts on 2022-06-08 21:36:28 CEST ---

@pem That appears to be correct. It's as if the static configuration is downgraded to a temporary configuration by the first DBus reconfiguration, and then the second DBus event replaces it. I haven't tried it with a single interface being reset multiple times, only with multiple interfaces / IPv6, but I would expect the same behavior in that case, too.

--- Additional comment from Petr Menšík on 2022-06-17 11:59:32 CEST ---

I think I have found a reason for this. It was well hidden. The error is inside add_update_server function. There is a loop searching marked server with matching domain. But it does not update up pointer correctly. That then replaces the record following the marked server to beginning, which creates a memleak and forgets static servers defined by configuration file (which has flags == 0).

Solution is to update up pointer at the end of loop. Even latest master branch is affected, I do not think this were reported to upstream yet.

--- Additional comment from Petr Menšík on 2022-06-17 12:03:20 CEST ---



--- Additional comment from Petr Menšík on 2022-06-17 12:07:47 CEST ---



--- Additional comment from Renaud Métrich on 2022-06-17 12:17:23 CEST ---

+1 because it's super annoying :)

--- Additional comment from Petr Menšík on 2022-06-17 12:43:55 CEST ---

My test build above test8 tag works. I will prepare rawhide update, but this works to me now:

https://copr.fedorainfracloud.org/coprs/pemensik/dnsmasq/package/dnsmasq/

--- Additional comment from Petr Menšík on 2022-06-17 12:45:07 CEST ---

Reported upstream with candidate patch:
https://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2022q2/016442.html

--- Additional comment from Petr Menšík on 2022-06-17 13:08:08 CEST ---

Pushed fix to rawhide branch, build is running. Because it has no good workaround, not waiting for upstream response. Will replace the patch eventually.

--- Additional comment from Fedora Update System on 2022-06-17 13:37:58 CEST ---

FEDORA-2022-12673da823 has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-12673da823

--- Additional comment from Fedora Update System on 2022-06-17 13:38:42 CEST ---

FEDORA-2022-73122722e6 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-73122722e6

--- Additional comment from Fedora Update System on 2022-06-18 03:24:55 CEST ---

FEDORA-2022-73122722e6 has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-73122722e6`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-73122722e6

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

--- Additional comment from Fedora Update System on 2022-06-18 03:40:33 CEST ---

FEDORA-2022-12673da823 has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-12673da823`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-12673da823

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

--- Additional comment from Andrew Watts on 2022-06-19 02:40:48 CEST ---

Installing dnsmasq-2.86-9.fc36.x86_64 package from the updates-testing repository fixed the issue for me.

--- Additional comment from Renaud Métrich on 2022-06-20 08:32:32 CEST ---

Seems to work for me as well.

--- Additional comment from Jasper Siero on 2022-06-21 10:05:52 CEST ---

The new package dnsmasq-2.86-9.fc36.x86_64 in testing works for me and fixes the problem

--- Additional comment from Fedora Update System on 2022-06-22 02:47:49 CEST ---

FEDORA-2022-12673da823 has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.

--- Additional comment from Fedora Update System on 2022-07-03 03:19:45 CEST ---

FEDORA-2022-73122722e6 has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.

--- Additional comment from Andrew Watts on 2022-11-22 04:18:10 CET ---

This problem appears to have returned with Fedora 37 and dnsmasq-2.87-1.fc37.x86_64. It looks like the upstream patch fixing this issue, which I believe is this one: 

https://thekelleys.org.uk/gitweb/?p=dnsmasq.git;a=commit;h=930428fb970f4991e5c2933fd5a5d2504c18a551

is not in 2.87 and won't land until 2.88.
Comment 1 Petr Menšík 2022-11-24 23:08:25 UTC 
Cloning this bug again, since it seems I have failed to notice this change were not properly integrated into upstream release.
Thank you for reporting it again. I guess a regression test for it would be useful.
Comment 2 Petr Menšík 2022-11-25 10:29:58 UTC 
Previously fixed by patch file dnsmasq-2.87-reuse-server.patch. Changed more that strictly needed. Using upstream patch this time.
Comment 3 Fedora Update System 2022-11-25 11:23:50 UTC 
FEDORA-2022-3e6837f10f has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2022-3e6837f10f
Comment 4 Fedora Update System 2022-11-26 01:29:54 UTC 
FEDORA-2022-3e6837f10f has been pushed to the Fedora 37 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2022-3e6837f10f`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-3e6837f10f

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Comment 5 Andrew Watts 2022-12-06 02:30:33 UTC 
dnsmasq-2.87-3.fc37.x86_64 from the testing repository appears to fix the issue for me.
Comment 6 Fedora Update System 2022-12-08 02:05:50 UTC 
FEDORA-2022-3e6837f10f has been pushed to the Fedora 37 stable repository.
If problem still persists, please make note of it in this bug report.
Comment 7 Renaud Métrich 2022-12-09 13:57:24 UTC 
I can confirm the bug is fixed.