Bug 1927973
| Summary: | dnsmasq --bind-dynamic stops replying to upstream queries | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Petr Menšík <pemensik> | ||||||
| Component: | dnsmasq | Assignee: | Petr Menšík <pemensik> | ||||||
| Status: | CLOSED ERRATA | QA Contact: | Petr Sklenar <psklenar> | ||||||
| Severity: | urgent | Docs Contact: | |||||||
| Priority: | urgent | ||||||||
| Version: | 8.3 | CC: | aegorenk, braander, bshirren, jmaxwell, psklenar, qe-baseos-daemons, rhowe, sjayapra | ||||||
| Target Milestone: | rc | Keywords: | AutoVerified, TestCaseProvided, Triaged | ||||||
| Target Release: | 8.0 | ||||||||
| Hardware: | x86_64 | ||||||||
| OS: | Linux | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | dnsmasq-2.79-17.el8 | Doc Type: | If docs needed, set a value | ||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | 1887649 | Environment: | |||||||
| Last Closed: | 2021-11-09 17:37:01 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Bug Depends On: | 1887649 | ||||||||
| Bug Blocks: | |||||||||
| Attachments: |
|
||||||||
RHEL8 keeps spawning ENOBUFS error, even when it is requested to omit it. I think I found a reason for it, it would need change even in the latest upstream release. Current netlink change observation algorithm ignores netlink broadcast information delivered. Instead, it just writes new address event into queue. I think when changes are fast enough, queue writing blocks netlink loop and is responsible for ENOBUFS error therefore. Proposed fix is reducing queue writes. Because it does not use broadcasted information, but forces netlink re-reading repeated the same time as number of incoming events. Because it writes queue every time. Instead, it should just re-read once AFTER reading all asynchronous messages. Multiple events are not necessary and take not required CPU cycles. Instead of serving user requests it spends time reading netlink again and again. Pushed multiple commits to github repo: https://github.com/InfrastructureServices/dnsmasq/tree/netlink-tests Pushed used test to public repository, to help discussion in upstream. 1. https://github.com/InfrastructureServices/dnsmasq-tests/tree/master/Regression/netlink-interface-fast-changes Posted patches to upstream mailing list: https://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2021q1/014709.html Created attachment 1757792 [details]
Main patch reducing netlink congestion
Created attachment 1757794 [details]
Remove whole process delay on ENOBUFS
After repeated testing of my changes, it seems it does not improve rhel8 results significantly. Current state seems strange, like it stayed in enumerate_interfaces function, after netlink request was finished. According to strace, it continues reading after NLMSG_DONE is received. Because iface_enumerate uses blocking reads, times start getting longer. I have no good theory how that happened and why. My current attempts behave similar way on RHEL8 and worse way than vanilla upstream. I am missing some required fix still. Patches merged upstream: http://thekelleys.org.uk/gitweb/?p=dnsmasq.git;a=commit;h=a8c14745625c0862fd77b6307ea00c2f67c9e5e3 http://thekelleys.org.uk/gitweb/?p=dnsmasq.git;a=commit;h=8b8a4148ec14363d75ad76c1fb9542c85b66b8c5 http://thekelleys.org.uk/gitweb/?p=dnsmasq.git;a=commit;h=4c0aecc68524c5fd74053244a605e905dc644228 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: dnsmasq security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:4153 |
I dont understand why the same fix, as on RHEL7 and Fedora, does not work on RHEL8. It seems its kernel ignores : setsockopt(daemon->netlinkfd, SOL_NETLINK, NETLINK_NO_ENOBUFS, &opt, sizeof(opt)) == -1) Where it works as expected on RHEL7 and Fedora. I rebuilt latest dnsmasq-2.84-1.el8.x86_64 on RHEL8, it still fails the test. But the same fix just does not work on RHEL8, at least on 1minutetip machines. Fedora passes the test just fine sice 2.81 inclusive. I narrowed by testing to upstream commit [1], which is the one fixing this issue. 1. http://thekelleys.org.uk/gitweb/?p=dnsmasq.git;a=commit;h=1627d577af03cdf747285e79fa747b6aaae8033f