Bug 2188443
| Summary: | dnsmasq: dying after some time, file descriptors and too many open files | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | Dave Cain <dcain> | ||||
| Component: | dnsmasq | Assignee: | Petr Menšík <pemensik> | ||||
| Status: | CLOSED MIGRATED | QA Contact: | rhel-cs-infra-services-qe <rhel-cs-infra-services-qe> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 9.1 | CC: | jnunez, pemensik, psklenar, rhel-cs-infra-services-qe | ||||
| Target Milestone: | rc | Keywords: | MigratedToJIRA, Triaged | ||||
| Target Release: | --- | Flags: | pm-rhel:
mirror+
|
||||
| Hardware: | Unspecified | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2023-09-21 19:06:13 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Hmm, interesting. Can you use lsof -np $(pidof dnsmasq) to list those descriptors? Are they related to DHCP or DNS? Is there anything notable about the configuration? Can you share all uncommented lines, possibly with replaced domains to example.{com,net,org}? I have not seen similar report, so there has to be something unusual for your configuration, which others do not need often.
I am afraid the only workaround until this is fixes would be periodic restarts, let's say each 3 hours.
Output as requested, this is after an hour of uptime on the service. Looks to be DNS query related looking at a few of the FDs: $ sudo lsof -np 353652 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME dnsmasq 353652 dnsmasq cwd DIR 253,0 235 128 / dnsmasq 353652 dnsmasq rtd DIR 253,0 235 128 / dnsmasq 353652 dnsmasq txt REG 253,0 434560 537834766 /usr/sbin/dnsmasq dnsmasq 353652 dnsmasq mem REG 253,0 53416 537568532 /usr/lib64/libnss_sss.so.2 dnsmasq 353652 dnsmasq mem REG 253,0 26988 537517833 /usr/lib64/gconv/gconv-modules.cache dnsmasq 353652 dnsmasq mem REG 253,0 153600 537517810 /usr/lib64/libgpg-error.so.0.32.0 dnsmasq 353652 dnsmasq mem REG 253,0 108144 540216284 /usr/lib64/libgcc_s-11-20220421.so.1 dnsmasq 353652 dnsmasq mem REG 253,0 1288512 537744235 /usr/lib64/libgcrypt.so.20.4.0 dnsmasq 353652 dnsmasq mem REG 253,0 36944 537517653 /usr/lib64/libcap.so.2.48 dnsmasq 353652 dnsmasq mem REG 253,0 144120 537517837 /usr/lib64/liblz4.so.1.9.3 dnsmasq 353652 dnsmasq mem REG 253,0 882376 537517664 /usr/lib64/libzstd.so.1.5.1 dnsmasq 353652 dnsmasq mem REG 253,0 178744 537517647 /usr/lib64/liblzma.so.5.2.5 dnsmasq 353652 dnsmasq mem REG 253,0 1591920 537517747 /usr/lib64/libunistring.so.2.1.0 dnsmasq 353652 dnsmasq mem REG 253,0 919336 537745891 /usr/lib64/libsystemd.so.0.33.0 dnsmasq 353652 dnsmasq mem REG 253,0 2386744 537517426 /usr/lib64/libc.so.6 dnsmasq 353652 dnsmasq mem REG 253,0 679080 537517743 /usr/lib64/libgmp.so.10.4.0 dnsmasq 353652 dnsmasq mem REG 253,0 627264 537517646 /usr/lib64/libhogweed.so.6.5 dnsmasq 353652 dnsmasq mem REG 253,0 341648 537517872 /usr/lib64/libnettle.so.8.5 dnsmasq 353652 dnsmasq mem REG 253,0 130952 537517841 /usr/lib64/libidn2.so.0.3.7 dnsmasq 353652 dnsmasq mem REG 253,0 343760 537745896 /usr/lib64/libdbus-1.so.3.19.13 dnsmasq 353652 dnsmasq mem REG 253,0 918104 537517416 /usr/lib64/ld-linux-x86-64.so.2 dnsmasq 353652 dnsmasq 0u CHR 1,3 0t0 4 /dev/null dnsmasq 353652 dnsmasq 1u CHR 1,3 0t0 4 /dev/null dnsmasq 353652 dnsmasq 2u CHR 1,3 0t0 4 /dev/null dnsmasq 353652 dnsmasq 3u REG 253,0 568 45139367 /var/lib/dnsmasq/dnsmasq.leases dnsmasq 353652 dnsmasq 4u IPv4 2168941 0t0 UDP *:bootps dnsmasq 353652 dnsmasq 5u netlink 0t0 2168942 ROUTE dnsmasq 353652 dnsmasq 6u IPv4 2168944 0t0 UDP *:domain dnsmasq 353652 dnsmasq 7u IPv4 2168945 0t0 TCP *:domain (LISTEN) dnsmasq 353652 dnsmasq 8u IPv4 2168946 0t0 UDP *:tftp dnsmasq 353652 dnsmasq 9u IPv6 2168947 0t0 UDP *:domain dnsmasq 353652 dnsmasq 10u IPv6 2168948 0t0 TCP *:domain (LISTEN) dnsmasq 353652 dnsmasq 11u IPv6 2168949 0t0 UDP *:tftp dnsmasq 353652 dnsmasq 12r FIFO 0,13 0t0 2168956 pipe dnsmasq 353652 dnsmasq 13w FIFO 0,13 0t0 2168956 pipe dnsmasq 353652 dnsmasq 14u unix 0xff2fdbcac7d6c400 0t0 2166214 type=DGRAM (CONNECTED) dnsmasq 353652 dnsmasq 16r FIFO 0,13 0t0 2195666 pipe dnsmasq 353652 dnsmasq 17r FIFO 0,13 0t0 2180555 pipe dnsmasq 353652 dnsmasq 18r FIFO 0,13 0t0 2195669 pipe dnsmasq 353652 dnsmasq 19r FIFO 0,13 0t0 2196716 pipe dnsmasq 353652 dnsmasq 20r FIFO 0,13 0t0 2183425 pipe dnsmasq 353652 dnsmasq 21r FIFO 0,13 0t0 2196719 pipe dnsmasq 353652 dnsmasq 22r FIFO 0,13 0t0 2190192 pipe dnsmasq 353652 dnsmasq 23r FIFO 0,13 0t0 2168432 pipe dnsmasq 353652 dnsmasq 24r FIFO 0,13 0t0 2182506 pipe dnsmasq 353652 dnsmasq 25r FIFO 0,13 0t0 2178054 pipe dnsmasq 353652 dnsmasq 26r FIFO 0,13 0t0 2184098 pipe dnsmasq 353652 dnsmasq 27r FIFO 0,13 0t0 2175961 pipe dnsmasq 353652 dnsmasq 28r FIFO 0,13 0t0 2195622 pipe dnsmasq 353652 dnsmasq 29r FIFO 0,13 0t0 2173844 pipe dnsmasq 353652 dnsmasq 30r FIFO 0,13 0t0 2195642 pipe dnsmasq 353652 dnsmasq 31r FIFO 0,13 0t0 2186968 pipe dnsmasq 353652 dnsmasq 32r FIFO 0,13 0t0 2195625 pipe dnsmasq 353652 dnsmasq 33r FIFO 0,13 0t0 2193009 pipe dnsmasq 353652 dnsmasq 34r FIFO 0,13 0t0 2175132 pipe dnsmasq 353652 dnsmasq 35r FIFO 0,13 0t0 2169954 pipe dnsmasq 353652 dnsmasq 36r FIFO 0,13 0t0 2169860 pipe dnsmasq 353652 dnsmasq 37r FIFO 0,13 0t0 2169865 pipe dnsmasq 353652 dnsmasq 38r FIFO 0,13 0t0 2169870 pipe dnsmasq 353652 dnsmasq 39r FIFO 0,13 0t0 2169889 pipe dnsmasq 353652 dnsmasq 40r FIFO 0,13 0t0 2167963 pipe dnsmasq 353652 dnsmasq 41r FIFO 0,13 0t0 2169913 pipe dnsmasq 353652 dnsmasq 42r FIFO 0,13 0t0 2167986 pipe dnsmasq 353652 dnsmasq 43r FIFO 0,13 0t0 2167991 pipe dnsmasq 353652 dnsmasq 44r FIFO 0,13 0t0 2168017 pipe dnsmasq 353652 dnsmasq 45r FIFO 0,13 0t0 2195632 pipe dnsmasq 353652 dnsmasq 46r FIFO 0,13 0t0 2169959 pipe dnsmasq 353652 dnsmasq 47r FIFO 0,13 0t0 2169039 pipe dnsmasq 353652 dnsmasq 48r FIFO 0,13 0t0 2169044 pipe dnsmasq 353652 dnsmasq 49r FIFO 0,13 0t0 2169049 pipe dnsmasq 353652 dnsmasq 50r FIFO 0,13 0t0 2168260 pipe dnsmasq 353652 dnsmasq 51r FIFO 0,13 0t0 2168265 pipe dnsmasq 353652 dnsmasq 52r FIFO 0,13 0t0 2168291 pipe dnsmasq 353652 dnsmasq 53r FIFO 0,13 0t0 2166675 pipe dnsmasq 353652 dnsmasq 54r FIFO 0,13 0t0 2168424 pipe dnsmasq 353652 dnsmasq 55r FIFO 0,13 0t0 2172006 pipe dnsmasq 353652 dnsmasq 56r FIFO 0,13 0t0 2172011 pipe dnsmasq 353652 dnsmasq 57r FIFO 0,13 0t0 2172016 pipe dnsmasq 353652 dnsmasq 58r FIFO 0,13 0t0 2172021 pipe dnsmasq 353652 dnsmasq 59r FIFO 0,13 0t0 2168447 pipe dnsmasq 353652 dnsmasq 60r FIFO 0,13 0t0 2168437 pipe dnsmasq 353652 dnsmasq 61r FIFO 0,13 0t0 2168442 pipe dnsmasq 353652 dnsmasq 62r FIFO 0,13 0t0 2181519 pipe dnsmasq 353652 dnsmasq 63r FIFO 0,13 0t0 2168452 pipe dnsmasq 353652 dnsmasq 64r FIFO 0,13 0t0 2168457 pipe dnsmasq 353652 dnsmasq 65r FIFO 0,13 0t0 2168462 pipe dnsmasq 353652 dnsmasq 66r FIFO 0,13 0t0 2168467 pipe dnsmasq 353652 dnsmasq 67r FIFO 0,13 0t0 2169697 pipe dnsmasq 353652 dnsmasq 68r FIFO 0,13 0t0 2168570 pipe dnsmasq 353652 dnsmasq 69r FIFO 0,13 0t0 2177237 pipe dnsmasq 353652 dnsmasq 70r FIFO 0,13 0t0 2175137 pipe dnsmasq 353652 dnsmasq 71r FIFO 0,13 0t0 2175142 pipe dnsmasq 353652 dnsmasq 72r FIFO 0,13 0t0 2175522 pipe dnsmasq 353652 dnsmasq 73r FIFO 0,13 0t0 2175671 pipe dnsmasq 353652 dnsmasq 74r FIFO 0,13 0t0 2174377 pipe dnsmasq 353652 dnsmasq 75r FIFO 0,13 0t0 2174382 pipe dnsmasq 353652 dnsmasq 76r FIFO 0,13 0t0 2174387 pipe dnsmasq 353652 dnsmasq 77r FIFO 0,13 0t0 2173734 pipe dnsmasq 353652 dnsmasq 78r FIFO 0,13 0t0 2171677 pipe dnsmasq 353652 dnsmasq 79r FIFO 0,13 0t0 2174735 pipe dnsmasq 353652 dnsmasq 80r FIFO 0,13 0t0 2171690 pipe dnsmasq 353652 dnsmasq 81r FIFO 0,13 0t0 2173854 pipe dnsmasq 353652 dnsmasq 82r FIFO 0,13 0t0 2173849 pipe dnsmasq 353652 dnsmasq 83r FIFO 0,13 0t0 2193852 pipe dnsmasq 353652 dnsmasq 84r FIFO 0,13 0t0 2171869 pipe dnsmasq 353652 dnsmasq 85r FIFO 0,13 0t0 2174654 pipe dnsmasq 353652 dnsmasq 86r FIFO 0,13 0t0 2174659 pipe dnsmasq 353652 dnsmasq 87r FIFO 0,13 0t0 2174664 pipe dnsmasq 353652 dnsmasq 88r FIFO 0,13 0t0 2174669 pipe dnsmasq 353652 dnsmasq 89r FIFO 0,13 0t0 2174690 pipe dnsmasq 353652 dnsmasq 90r FIFO 0,13 0t0 2174695 pipe dnsmasq 353652 dnsmasq 91r FIFO 0,13 0t0 2174700 pipe dnsmasq 353652 dnsmasq 92r FIFO 0,13 0t0 2174705 pipe dnsmasq 353652 dnsmasq 93r FIFO 0,13 0t0 2177141 pipe dnsmasq 353652 dnsmasq 94r FIFO 0,13 0t0 2190900 pipe dnsmasq 353652 dnsmasq 95r FIFO 0,13 0t0 2189406 pipe dnsmasq 353652 dnsmasq 96r FIFO 0,13 0t0 2176478 pipe dnsmasq 353652 dnsmasq 97r FIFO 0,13 0t0 2177156 pipe dnsmasq 353652 dnsmasq 98r FIFO 0,13 0t0 2177161 pipe dnsmasq 353652 dnsmasq 99r FIFO 0,13 0t0 2177166 pipe dnsmasq 353652 dnsmasq 100r FIFO 0,13 0t0 2177171 pipe dnsmasq 353652 dnsmasq 101r FIFO 0,13 0t0 2193924 pipe dnsmasq 353652 dnsmasq 102r FIFO 0,13 0t0 2177226 pipe dnsmasq 353652 dnsmasq 103r FIFO 0,13 0t0 2177242 pipe dnsmasq 353652 dnsmasq 104r FIFO 0,13 0t0 2177232 pipe dnsmasq 353652 dnsmasq 105r FIFO 0,13 0t0 2177247 pipe dnsmasq 353652 dnsmasq 106r FIFO 0,13 0t0 2177252 pipe dnsmasq 353652 dnsmasq 107r FIFO 0,13 0t0 2174804 pipe dnsmasq 353652 dnsmasq 108r FIFO 0,13 0t0 2174809 pipe dnsmasq 353652 dnsmasq 109r FIFO 0,13 0t0 2178304 pipe dnsmasq 353652 dnsmasq 110r FIFO 0,13 0t0 2178349 pipe dnsmasq 353652 dnsmasq 111r FIFO 0,13 0t0 2179221 pipe dnsmasq 353652 dnsmasq 112r FIFO 0,13 0t0 2179226 pipe dnsmasq 353652 dnsmasq 113r FIFO 0,13 0t0 2179260 pipe dnsmasq 353652 dnsmasq 114r FIFO 0,13 0t0 2179265 pipe dnsmasq 353652 dnsmasq 115r FIFO 0,13 0t0 2177542 pipe dnsmasq 353652 dnsmasq 116r FIFO 0,13 0t0 2176950 pipe dnsmasq 353652 dnsmasq 117r FIFO 0,13 0t0 2176955 pipe dnsmasq 353652 dnsmasq 118r FIFO 0,13 0t0 2180533 pipe dnsmasq 353652 dnsmasq 119r FIFO 0,13 0t0 2178566 pipe dnsmasq 353652 dnsmasq 120r FIFO 0,13 0t0 2177772 pipe dnsmasq 353652 dnsmasq 121r FIFO 0,13 0t0 2192777 pipe dnsmasq 353652 dnsmasq 122r FIFO 0,13 0t0 2178570 pipe dnsmasq 353652 dnsmasq 123r FIFO 0,13 0t0 2180560 pipe dnsmasq 353652 dnsmasq 124r FIFO 0,13 0t0 2181014 pipe dnsmasq 353652 dnsmasq 125r FIFO 0,13 0t0 2178835 pipe dnsmasq 353652 dnsmasq 126r FIFO 0,13 0t0 2178872 pipe dnsmasq 353652 dnsmasq 127r FIFO 0,13 0t0 2195635 pipe dnsmasq 353652 dnsmasq 128r FIFO 0,13 0t0 2190638 pipe dnsmasq 353652 dnsmasq 129r FIFO 0,13 0t0 2183249 pipe dnsmasq 353652 dnsmasq 130r FIFO 0,13 0t0 2181886 pipe dnsmasq 353652 dnsmasq 131r FIFO 0,13 0t0 2181891 pipe dnsmasq 353652 dnsmasq 132r FIFO 0,13 0t0 2181896 pipe dnsmasq 353652 dnsmasq 133r FIFO 0,13 0t0 2181901 pipe dnsmasq 353652 dnsmasq 134r FIFO 0,13 0t0 2181906 pipe dnsmasq 353652 dnsmasq 135r FIFO 0,13 0t0 2181911 pipe dnsmasq 353652 dnsmasq 136r FIFO 0,13 0t0 2181916 pipe dnsmasq 353652 dnsmasq 137r FIFO 0,13 0t0 2181921 pipe dnsmasq 353652 dnsmasq 138r FIFO 0,13 0t0 2181926 pipe dnsmasq 353652 dnsmasq 139r FIFO 0,13 0t0 2181950 pipe dnsmasq 353652 dnsmasq 140r FIFO 0,13 0t0 2181955 pipe dnsmasq 353652 dnsmasq 141r FIFO 0,13 0t0 2181960 pipe dnsmasq 353652 dnsmasq 142r FIFO 0,13 0t0 2181965 pipe dnsmasq 353652 dnsmasq 143r FIFO 0,13 0t0 2179942 pipe dnsmasq 353652 dnsmasq 144r FIFO 0,13 0t0 2181972 pipe dnsmasq 353652 dnsmasq 145r FIFO 0,13 0t0 2181977 pipe dnsmasq 353652 dnsmasq 146r FIFO 0,13 0t0 2181982 pipe dnsmasq 353652 dnsmasq 147r FIFO 0,13 0t0 2179947 pipe dnsmasq 353652 dnsmasq 148r FIFO 0,13 0t0 2179952 pipe dnsmasq 353652 dnsmasq 149r FIFO 0,13 0t0 2179957 pipe dnsmasq 353652 dnsmasq 150r FIFO 0,13 0t0 2179962 pipe dnsmasq 353652 dnsmasq 151r FIFO 0,13 0t0 2191441 pipe dnsmasq 353652 dnsmasq 152r FIFO 0,13 0t0 2182788 pipe dnsmasq 353652 dnsmasq 153r FIFO 0,13 0t0 2182832 pipe dnsmasq 353652 dnsmasq 154r FIFO 0,13 0t0 2182837 pipe dnsmasq 353652 dnsmasq 155r FIFO 0,13 0t0 2182842 pipe dnsmasq 353652 dnsmasq 156r FIFO 0,13 0t0 2182847 pipe dnsmasq 353652 dnsmasq 157r FIFO 0,13 0t0 2182852 pipe dnsmasq 353652 dnsmasq 158r FIFO 0,13 0t0 2182857 pipe dnsmasq 353652 dnsmasq 159r FIFO 0,13 0t0 2182862 pipe dnsmasq 353652 dnsmasq 160r FIFO 0,13 0t0 2182867 pipe dnsmasq 353652 dnsmasq 161r FIFO 0,13 0t0 2184771 pipe dnsmasq 353652 dnsmasq 162r FIFO 0,13 0t0 2184776 pipe dnsmasq 353652 dnsmasq 163r FIFO 0,13 0t0 2184781 pipe dnsmasq 353652 dnsmasq 164r FIFO 0,13 0t0 2184786 pipe dnsmasq 353652 dnsmasq 165r FIFO 0,13 0t0 2184791 pipe dnsmasq 353652 dnsmasq 166r FIFO 0,13 0t0 2184796 pipe dnsmasq 353652 dnsmasq 167r FIFO 0,13 0t0 2184801 pipe dnsmasq 353652 dnsmasq 168r FIFO 0,13 0t0 2186319 pipe dnsmasq 353652 dnsmasq 169r FIFO 0,13 0t0 2185620 pipe dnsmasq 353652 dnsmasq 170r FIFO 0,13 0t0 2185622 pipe dnsmasq 353652 dnsmasq 171r FIFO 0,13 0t0 2188284 pipe dnsmasq 353652 dnsmasq 172r FIFO 0,13 0t0 2188289 pipe dnsmasq 353652 dnsmasq 173r FIFO 0,13 0t0 2188294 pipe dnsmasq 353652 dnsmasq 174r FIFO 0,13 0t0 2188299 pipe dnsmasq 353652 dnsmasq 175r FIFO 0,13 0t0 2188304 pipe dnsmasq 353652 dnsmasq 176r FIFO 0,13 0t0 2184088 pipe dnsmasq 353652 dnsmasq 177r FIFO 0,13 0t0 2184093 pipe dnsmasq 353652 dnsmasq 178r FIFO 0,13 0t0 2197656 pipe dnsmasq 353652 dnsmasq 179r FIFO 0,13 0t0 2184103 pipe dnsmasq 353652 dnsmasq 180r FIFO 0,13 0t0 2184108 pipe dnsmasq 353652 dnsmasq 181r FIFO 0,13 0t0 2184113 pipe dnsmasq 353652 dnsmasq 182r FIFO 0,13 0t0 2186668 pipe dnsmasq 353652 dnsmasq 183r FIFO 0,13 0t0 2186673 pipe dnsmasq 353652 dnsmasq 184r FIFO 0,13 0t0 2194155 pipe dnsmasq 353652 dnsmasq 185r FIFO 0,13 0t0 2189411 pipe dnsmasq 353652 dnsmasq 186r FIFO 0,13 0t0 2186013 pipe dnsmasq 353652 dnsmasq 187r FIFO 0,13 0t0 2188672 pipe dnsmasq 353652 dnsmasq 188r FIFO 0,13 0t0 2189921 pipe dnsmasq 353652 dnsmasq 189r FIFO 0,13 0t0 2189926 pipe dnsmasq 353652 dnsmasq 190r FIFO 0,13 0t0 2187053 pipe dnsmasq 353652 dnsmasq 191r FIFO 0,13 0t0 2186973 pipe dnsmasq 353652 dnsmasq 192r FIFO 0,13 0t0 2186978 pipe dnsmasq 353652 dnsmasq 193r FIFO 0,13 0t0 2186983 pipe dnsmasq 353652 dnsmasq 194r FIFO 0,13 0t0 2190089 pipe dnsmasq 353652 dnsmasq 195r FIFO 0,13 0t0 2195645 pipe dnsmasq 353652 dnsmasq 196r FIFO 0,13 0t0 2190094 pipe dnsmasq 353652 dnsmasq 197r FIFO 0,13 0t0 2190419 pipe dnsmasq 353652 dnsmasq 198r FIFO 0,13 0t0 2190422 pipe dnsmasq 353652 dnsmasq 199r FIFO 0,13 0t0 2197653 pipe dnsmasq 353652 dnsmasq 200r FIFO 0,13 0t0 2190197 pipe dnsmasq 353652 dnsmasq 201r FIFO 0,13 0t0 2187162 pipe dnsmasq 353652 dnsmasq 202r FIFO 0,13 0t0 2191364 pipe dnsmasq 353652 dnsmasq 203r FIFO 0,13 0t0 2190559 pipe dnsmasq 353652 dnsmasq 204r FIFO 0,13 0t0 2187180 pipe dnsmasq 353652 dnsmasq 205r FIFO 0,13 0t0 2190574 pipe dnsmasq 353652 dnsmasq 206r FIFO 0,13 0t0 2190266 pipe dnsmasq 353652 dnsmasq 207r FIFO 0,13 0t0 2190650 pipe dnsmasq 353652 dnsmasq 208r FIFO 0,13 0t0 2191946 pipe dnsmasq 353652 dnsmasq 209r FIFO 0,13 0t0 2190644 pipe dnsmasq 353652 dnsmasq 210r FIFO 0,13 0t0 2190655 pipe dnsmasq 353652 dnsmasq 211r FIFO 0,13 0t0 2190660 pipe dnsmasq 353652 dnsmasq 212r FIFO 0,13 0t0 2190665 pipe dnsmasq 353652 dnsmasq 213r FIFO 0,13 0t0 2192069 pipe dnsmasq 353652 dnsmasq 214r FIFO 0,13 0t0 2191446 pipe dnsmasq 353652 dnsmasq 215r FIFO 0,13 0t0 2191451 pipe dnsmasq 353652 dnsmasq 216r FIFO 0,13 0t0 2192566 pipe dnsmasq 353652 dnsmasq 217r FIFO 0,13 0t0 2192568 pipe dnsmasq 353652 dnsmasq 218r FIFO 0,13 0t0 2195514 pipe dnsmasq 353652 dnsmasq 219r FIFO 0,13 0t0 2190910 pipe dnsmasq 353652 dnsmasq 220r FIFO 0,13 0t0 2191853 pipe dnsmasq 353652 dnsmasq 221r FIFO 0,13 0t0 2191858 pipe dnsmasq 353652 dnsmasq 222r FIFO 0,13 0t0 2191863 pipe dnsmasq 353652 dnsmasq 223r FIFO 0,13 0t0 2191868 pipe dnsmasq 353652 dnsmasq 224r FIFO 0,13 0t0 2191873 pipe dnsmasq 353652 dnsmasq 225r FIFO 0,13 0t0 2191878 pipe dnsmasq 353652 dnsmasq 226r FIFO 0,13 0t0 2191883 pipe dnsmasq 353652 dnsmasq 227r FIFO 0,13 0t0 2191888 pipe dnsmasq 353652 dnsmasq 228r FIFO 0,13 0t0 2191936 pipe dnsmasq 353652 dnsmasq 229r FIFO 0,13 0t0 2191941 pipe dnsmasq 353652 dnsmasq 230r FIFO 0,13 0t0 2195652 pipe dnsmasq 353652 dnsmasq 231r FIFO 0,13 0t0 2192000 pipe dnsmasq 353652 dnsmasq 232r FIFO 0,13 0t0 2192005 pipe dnsmasq 353652 dnsmasq 233r FIFO 0,13 0t0 2192010 pipe dnsmasq 353652 dnsmasq 234r FIFO 0,13 0t0 2192015 pipe dnsmasq 353652 dnsmasq 235r FIFO 0,13 0t0 2192020 pipe dnsmasq 353652 dnsmasq 236r FIFO 0,13 0t0 2193647 pipe dnsmasq 353652 dnsmasq 237r FIFO 0,13 0t0 2193652 pipe dnsmasq 353652 dnsmasq 238r FIFO 0,13 0t0 2196732 pipe dnsmasq 353652 dnsmasq 239r FIFO 0,13 0t0 2192105 pipe dnsmasq 353652 dnsmasq 240r FIFO 0,13 0t0 2195655 pipe dnsmasq 353652 dnsmasq 241r FIFO 0,13 0t0 2192167 pipe dnsmasq 353652 dnsmasq 242r FIFO 0,13 0t0 2195192 pipe dnsmasq 353652 dnsmasq 243r FIFO 0,13 0t0 2193857 pipe dnsmasq 353652 dnsmasq 244r FIFO 0,13 0t0 2195658 pipe dnsmasq 353652 dnsmasq 245r FIFO 0,13 0t0 2193929 pipe dnsmasq 353652 dnsmasq 246r FIFO 0,13 0t0 2193934 pipe dnsmasq 353652 dnsmasq 247r FIFO 0,13 0t0 2194999 pipe dnsmasq 353652 dnsmasq 248r FIFO 0,13 0t0 2195004 pipe dnsmasq 353652 dnsmasq 249r FIFO 0,13 0t0 2195661 pipe dnsmasq 353652 dnsmasq 250r FIFO 0,13 0t0 2193014 pipe dnsmasq 353652 dnsmasq 251r FIFO 0,13 0t0 2196735 pipe dnsmasq 353652 dnsmasq 252r FIFO 0,13 0t0 2195197 pipe dnsmasq 353652 dnsmasq 253r FIFO 0,13 0t0 2195279 pipe dnsmasq 353652 dnsmasq 254r FIFO 0,13 0t0 2195598 pipe dnsmasq 353652 dnsmasq 255r FIFO 0,13 0t0 2194160 pipe dnsmasq 353652 dnsmasq 256r FIFO 0,13 0t0 2195346 pipe dnsmasq 353652 dnsmasq 257r FIFO 0,13 0t0 2195351 pipe dnsmasq 353652 dnsmasq 258r FIFO 0,13 0t0 2195396 pipe dnsmasq 353652 dnsmasq 260r FIFO 0,13 0t0 2195519 pipe dnsmasq 353652 dnsmasq 261r FIFO 0,13 0t0 2195525 pipe dnsmasq 353652 dnsmasq 262r FIFO 0,13 0t0 2195530 pipe dnsmasq 353652 dnsmasq 263r FIFO 0,13 0t0 2196603 pipe dnsmasq 353652 dnsmasq 264r FIFO 0,13 0t0 2195564 pipe dnsmasq 353652 dnsmasq 266r FIFO 0,13 0t0 2195603 pipe Created attachment 1959510 [details] bz2188443-dnsmasqconf-24-april.txt Understood on restarting every 3 hours, I am doing that presently :) Here's my dnsmasq.conf. Happy to share the sub configurations also, they are simply address and DHCP-host records, not much more. Interesting, there are not too many network sockets, but too many FIFO pipes. I thought there should be always just two sockets of pipe to send messages back from helper or maybe from TCP forks. Not yet sure why they are not closed properly and why those pipes are even opened. It might be related to tftp, which is not usually enabled in common configurations. Have you tried disabling tftp support in dnsmasq and using tftp-server instead? Would it help preventing those leaks? Does not seem to help. Commented out: #enable-tftp #tftp-root=/var/lib/tftpboot Back up to 44 FIFO after 15 minutes. Any other suggestions? Should I try upstream 2.89 to see if the same behavior exists? I guess they could be TCP worker pipes, which are used from child forked process to insert cache records into master process.
You could rule that out by using -d parameter. That would make quite bad TCP performance, but avoids spawning tcp_pipes in dnsmasq_daemon->tcp_pipes. I just guess attaching strace to main dnsmasq process might reveal what calls are done after pipe() calls. I am not sure how to identify the place, where those pipes are created and why they are not closed. Not sure why I haven't ever saw so high number of pipes in my testing.
Do you change ulimit -a of the dnsmasq? Is it possible fork() is failing and such failure is causing pipe file descriptor leak?
tcp_pipes size is limited to 20. So your number of pipes is far higher in that. I haven't found candidate place by looking at the code.
Can you try to capture output of:
# strace -p ${PID} -e t=pipe2,close,write,writev -e decode-fd=socket
Whether it would hint what happens close to pipe calls, whether descriptors are closed and in which case it might leave it open.
Not sure how else to find the cause of those leaks.
I may have found some fix missing in our version after all. Upstream commit ad90eb07 [1] fixes case when too many TCP connections are received. It makes sure it accepts the connection only if it can find free TCP slot first. That might help you, in your strace were pipe failed followed by BADF error. I think that is related to that case. It tried to use pipefd in else branch on dnsmasq.c:1969, but that pipefd failed to be set when pipe returned -1. Since it is not properly initialized, I think it can contain random numbers, which are then closed. I will try to prepare fixed candidate for testing next week. [1] https://thekelleys.org.uk/gitweb/?p=dnsmasq.git;h=ad90eb075dfeeb1936e8bc0f323fcc23f89364d4 Good to hear Petr. Happy to test, just let me know. Can you please try MR #18 [1], whether the issue is still there? 1 https://gitlab.com/redhat/centos-stream/rpms/dnsmasq/-/merge_requests/18 tried throwing just higher number of tcp queries on dnsmasq, but that does not reproduce leaking descriptors
# for I in {1..200}; do echo $RANDOM.home.arpa a; echo $RANDOM.home.arpa aaaa; done > queries2
# dnsperf -d queries -s 127.0.0.2 -m tcp -l 10
Used bind listening on 127.0.0.1 to answer those queries. As expected, TCP performance of dnsmasq is quite tragic:
tatistics:
Queries sent: 500
Queries completed: 300 (60.00%)
Queries lost: 200 (40.00%)
Response codes: NXDOMAIN 300 (100.00%)
Average packet size: request 32, response 76
Run time (s): 10.000089
Queries per second: 29.999733
Average Latency (s): 0.001131 (min 0.000060, max 0.004839)
Latency StdDev (s): 0.001306
Connection Statistics:
Reconnections: 2 (66.67% of 3 connections)
Average Latency (s): 0.000261 (min 0.000020, max 0.000416)
Latency StdDev (s): 0.000212
Compared to just bind directly:
Statistics:
Queries sent: 310695
Queries completed: 310695 (100.00%)
Queries lost: 0 (0.00%)
Response codes: NXDOMAIN 310695 (100.00%)
Average packet size: request 32, response 76
Run time (s): 10.040069
Queries per second: 30945.504458
Average Latency (s): 0.003192 (min 0.000032, max 0.042978)
Latency StdDev (s): 0.000693
Connection Statistics:
Reconnections: 0 (0.00% of 1 connections)
Average Latency (s): 0.000173 (min 0.000173, max 0.000173)
But this does not cause dnsmasq to leak pipe descriptors.
dnsmasq 5186 dnsmasq 0u CHR 1,3 0t0 4 /dev/null
dnsmasq 5186 dnsmasq 1u CHR 1,3 0t0 4 /dev/null
dnsmasq 5186 dnsmasq 2u CHR 1,3 0t0 4 /dev/null
dnsmasq 5186 dnsmasq 3u netlink 0t0 32908 ROUTE
dnsmasq 5186 dnsmasq 4u IPv4 32910 0t0 UDP 127.0.0.2:domain
dnsmasq 5186 dnsmasq 5u IPv4 32911 0t0 TCP 127.0.0.2:domain (LISTEN)
dnsmasq 5186 dnsmasq 6r FIFO 0,13 0t0 32916 pipe
dnsmasq 5186 dnsmasq 7w FIFO 0,13 0t0 32916 pipe
dnsmasq 5186 dnsmasq 8u unix 0xff29a7048326bb80 0t0 32918 type=DGRAM (CONNECTED)
Hi Petr, After running this build (dnsmasq-2.85-11.el9.x86_64) over the weekend, I see at maximum now 18 pipe processes associated to the master pid of dnsmasq. This looks to be massively improved over the previous behavior, I would say this addresses my original problem reported. I'll keep running this, and am happy to gather any further diagnostic or logging data. Just let me know what you may need. Hi Petr, Just to report this has been stable for me after 2 weeks, no issues. Can we merge this fix? Upstream explanation how that happens: https://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2021q2/014976.html I were not able create reproducer, but reading that explanation. Maybe I would have to run 2 dnsperf instances, one on IPv4, one on IPv6. Dunno. Is it okay to merge it without working reproducer? Unfortunately this won't be fixed in 9.3 unless z-stream is requested. I have not been able to create working automated reproducer (yet). Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug. This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there. Due to differences in account names between systems, some fields were not replicated. Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information. To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer. You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like: "Bugzilla Bug" = 1234567 In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information. |
Description of problem: I'm seeing dnsmasq service exit after a period of time (minimum 6 hours) on a RHEL 9.1 box. This is undesired, and manual intervention is required (systemctl restart dnsmasq) in order for service to be restored. I'm not seeing anything of significance in logs or via journalctl. I added DumpCore=yes in /etc/systemd/system.conf in an attempt to get any useful logs, but none seem to be generated. In running strace on the master PID, I notice 1 EMFILE (Too many > open files) and -1 EBADF (Bad file descriptor) at the end of the output right before the service goes down. I see a climbing number of file descriptors also when I restart the service. # ls /proc/249121/fd | wc -l 403 This value started at around 30. In my other lab with RHEL8 / dnsmasq, I see 23. Version-Release number of selected component (if applicable): dnsmasq-2.85-5.el9.x86_64 Steps to Reproduce: 1. Have dnsmasq service running, where service is acting as both a DHCP server and DNS resolver 2. Wait 6 hours 3. service shuts down on its own / dies, requires manual intervention Additional info: RHEL 9.1, 5.14.0-162.23.1.el9_1.x86_64 kernel.