Bug 1927973 - dnsmasq --bind-dynamic stops replying to upstream queries
Summary: dnsmasq --bind-dynamic stops replying to upstream queries
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: dnsmasq
Version: 8.3
Hardware: x86_64
OS: Linux
Target Milestone: rc
: 8.0
Assignee: Petr Menšík
QA Contact: Petr Sklenar
Depends On: 1887649
TreeView+ depends on / blocked
Reported: 2021-02-12 01:10 UTC by Petr Menšík
Modified: 2021-11-09 19:30 UTC (History)
8 users (show)

Fixed In Version: dnsmasq-2.79-17.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1887649
Last Closed: 2021-11-09 17:37:01 UTC
Type: Bug
Target Upstream Version:

Attachments (Terms of Use)
Main patch reducing netlink congestion (3.95 KB, patch)
2021-02-18 13:27 UTC, Petr Menšík
no flags Details | Diff
Remove whole process delay on ENOBUFS (1001 bytes, patch)
2021-02-18 13:27 UTC, Petr Menšík
no flags Details | Diff

System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2021:4153 0 None None None 2021-11-09 17:37:10 UTC

Comment 1 Petr Menšík 2021-02-16 18:55:37 UTC
I dont understand why the same fix, as on RHEL7 and Fedora, does not work on RHEL8. It seems its kernel ignores :

      setsockopt(daemon->netlinkfd, SOL_NETLINK, NETLINK_NO_ENOBUFS, &opt, sizeof(opt)) == -1)

Where it works as expected on RHEL7 and Fedora. I rebuilt latest dnsmasq-2.84-1.el8.x86_64 on RHEL8, it still fails the test. But the same fix just does not work on RHEL8, at least on 1minutetip machines.

Fedora passes the test just fine sice 2.81 inclusive. I narrowed by testing to upstream commit [1], which is the one fixing this issue.

1. http://thekelleys.org.uk/gitweb/?p=dnsmasq.git;a=commit;h=1627d577af03cdf747285e79fa747b6aaae8033f

Comment 2 Petr Menšík 2021-02-18 10:02:46 UTC
RHEL8 keeps spawning ENOBUFS error, even when it is requested to omit it. I think I found a reason for it, it would need change even in the latest upstream release.
Current netlink change observation algorithm ignores netlink broadcast information delivered. Instead, it just writes new address event into queue. I think when changes are fast enough, queue writing blocks netlink loop and is responsible for ENOBUFS error therefore.

Proposed fix is reducing queue writes. Because it does not use broadcasted information, but forces netlink re-reading repeated the same time as number of incoming events. Because it writes queue every time. Instead, it should just re-read once AFTER reading all asynchronous messages. Multiple events are not necessary and take not required CPU cycles. Instead of serving user requests it spends time reading netlink again and again.

Pushed multiple commits to github repo: https://github.com/InfrastructureServices/dnsmasq/tree/netlink-tests

Comment 3 Petr Menšík 2021-02-18 11:10:14 UTC
Pushed used test to public repository, to help discussion in upstream.

1. https://github.com/InfrastructureServices/dnsmasq-tests/tree/master/Regression/netlink-interface-fast-changes

Comment 4 Petr Menšík 2021-02-18 13:25:50 UTC
Posted patches to upstream mailing list: https://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2021q1/014709.html

Comment 5 Petr Menšík 2021-02-18 13:27:09 UTC
Created attachment 1757792 [details]
Main patch reducing netlink congestion

Comment 6 Petr Menšík 2021-02-18 13:27:59 UTC
Created attachment 1757794 [details]
Remove whole process delay on ENOBUFS

Comment 7 Petr Menšík 2021-02-19 23:43:13 UTC
After repeated testing of my changes, it seems it does not improve rhel8 results significantly.

Current state seems strange, like it stayed in enumerate_interfaces function, after netlink request was finished. According to strace, it continues reading after NLMSG_DONE is received.
Because iface_enumerate uses blocking reads, times start getting longer. I have no good theory how that happened and why.

My current attempts behave similar way on RHEL8 and worse way than vanilla upstream. I am missing some required fix still.

Comment 30 errata-xmlrpc 2021-11-09 17:37:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: dnsmasq security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.