This bug has been migrated to another issue tracking site. It has been closed here and may no longer be being monitored.

If you would like to get updates for this issue, or to participate in it, you may do so at Red Hat Issue Tracker .
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2188443 - dnsmasq: dying after some time, file descriptors and too many open files
Summary: dnsmasq: dying after some time, file descriptors and too many open files
Keywords:
Status: CLOSED MIGRATED
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: dnsmasq
Version: 9.1
Hardware: Unspecified
OS: Linux
unspecified
high
Target Milestone: rc
: ---
Assignee: Petr Menšík
QA Contact: rhel-cs-infra-services-qe
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-04-20 18:58 UTC by Dave Cain
Modified: 2023-09-21 19:06 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-09-21 19:06:13 UTC
Type: Bug
Target Upstream Version:
Embargoed:
pm-rhel: mirror+

Attachments (Terms of Use)
bz2188443-dnsmasqconf-24-april.txt (818 bytes, text/plain)
2023-04-24 11:17 UTC, Dave Cain 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Gitlab redhat/centos-stream/rpms dnsmasq merge_requests 18 0 None opened Fix bug in TCP process handling 2023-06-09 14:19:53 UTC
Red Hat Issue Tracker   RHEL-6584 0 None Migrated None 2023-09-21 19:06:07 UTC
Red Hat Issue Tracker RHELPLAN-155334 0 None None None 2023-04-20 19:00:01 UTC

Description Dave Cain 2023-04-20 18:58:58 UTC 
Description of problem:
I'm seeing dnsmasq service exit after a period of time (minimum 6 hours) on a RHEL 9.1 box.  This is undesired, and manual intervention is required (systemctl restart dnsmasq) in order for service to be restored.

I'm not seeing anything of significance in logs or via journalctl.  I added DumpCore=yes in /etc/systemd/system.conf in an attempt to get any useful logs, but none seem to be generated.  In running strace on the master PID, I notice 1 EMFILE (Too many
> open files) and -1 EBADF (Bad file descriptor) at the end of the output right before the service goes down.

I see a climbing number of file descriptors also when I restart the service.

# ls /proc/249121/fd | wc -l
403

This value started at around 30.  In my other lab with RHEL8 / dnsmasq, I see 23.

Version-Release number of selected component (if applicable):
dnsmasq-2.85-5.el9.x86_64

Steps to Reproduce:
1. Have dnsmasq service running, where service is acting as both a DHCP server and DNS resolver
2. Wait 6 hours
3. service shuts down on its own / dies, requires manual intervention

Additional info:
RHEL 9.1, 5.14.0-162.23.1.el9_1.x86_64 kernel.
Comment 2 Petr Menšík 2023-04-22 08:39:48 UTC 
Hmm, interesting. Can you use lsof -np $(pidof dnsmasq) to list those descriptors? Are they related to DHCP or DNS?
Comment 4 Petr Menšík 2023-04-22 08:49:48 UTC 
Is there anything notable about the configuration? Can you share all uncommented lines, possibly with replaced domains to example.{com,net,org}? I have not seen similar report, so there has to be something unusual for your configuration, which others do not need often.

I am afraid the only workaround until this is fixes would be periodic restarts, let's say each 3 hours.
Comment 5 Dave Cain 2023-04-24 11:10:40 UTC 
Output as requested, this is after an hour of uptime on the service.  Looks to be DNS query related looking at a few of the FDs:

$ sudo lsof -np 353652
COMMAND    PID    USER   FD      TYPE             DEVICE SIZE/OFF      NODE NAME
dnsmasq 353652 dnsmasq  cwd       DIR              253,0      235       128 /
dnsmasq 353652 dnsmasq  rtd       DIR              253,0      235       128 /
dnsmasq 353652 dnsmasq  txt       REG              253,0   434560 537834766 /usr/sbin/dnsmasq
dnsmasq 353652 dnsmasq  mem       REG              253,0    53416 537568532 /usr/lib64/libnss_sss.so.2
dnsmasq 353652 dnsmasq  mem       REG              253,0    26988 537517833 /usr/lib64/gconv/gconv-modules.cache
dnsmasq 353652 dnsmasq  mem       REG              253,0   153600 537517810 /usr/lib64/libgpg-error.so.0.32.0
dnsmasq 353652 dnsmasq  mem       REG              253,0   108144 540216284 /usr/lib64/libgcc_s-11-20220421.so.1
dnsmasq 353652 dnsmasq  mem       REG              253,0  1288512 537744235 /usr/lib64/libgcrypt.so.20.4.0
dnsmasq 353652 dnsmasq  mem       REG              253,0    36944 537517653 /usr/lib64/libcap.so.2.48
dnsmasq 353652 dnsmasq  mem       REG              253,0   144120 537517837 /usr/lib64/liblz4.so.1.9.3
dnsmasq 353652 dnsmasq  mem       REG              253,0   882376 537517664 /usr/lib64/libzstd.so.1.5.1
dnsmasq 353652 dnsmasq  mem       REG              253,0   178744 537517647 /usr/lib64/liblzma.so.5.2.5
dnsmasq 353652 dnsmasq  mem       REG              253,0  1591920 537517747 /usr/lib64/libunistring.so.2.1.0
dnsmasq 353652 dnsmasq  mem       REG              253,0   919336 537745891 /usr/lib64/libsystemd.so.0.33.0
dnsmasq 353652 dnsmasq  mem       REG              253,0  2386744 537517426 /usr/lib64/libc.so.6
dnsmasq 353652 dnsmasq  mem       REG              253,0   679080 537517743 /usr/lib64/libgmp.so.10.4.0
dnsmasq 353652 dnsmasq  mem       REG              253,0   627264 537517646 /usr/lib64/libhogweed.so.6.5
dnsmasq 353652 dnsmasq  mem       REG              253,0   341648 537517872 /usr/lib64/libnettle.so.8.5
dnsmasq 353652 dnsmasq  mem       REG              253,0   130952 537517841 /usr/lib64/libidn2.so.0.3.7
dnsmasq 353652 dnsmasq  mem       REG              253,0   343760 537745896 /usr/lib64/libdbus-1.so.3.19.13
dnsmasq 353652 dnsmasq  mem       REG              253,0   918104 537517416 /usr/lib64/ld-linux-x86-64.so.2
dnsmasq 353652 dnsmasq    0u      CHR                1,3      0t0         4 /dev/null
dnsmasq 353652 dnsmasq    1u      CHR                1,3      0t0         4 /dev/null
dnsmasq 353652 dnsmasq    2u      CHR                1,3      0t0         4 /dev/null
dnsmasq 353652 dnsmasq    3u      REG              253,0      568  45139367 /var/lib/dnsmasq/dnsmasq.leases
dnsmasq 353652 dnsmasq    4u     IPv4            2168941      0t0       UDP *:bootps 
dnsmasq 353652 dnsmasq    5u  netlink                         0t0   2168942 ROUTE
dnsmasq 353652 dnsmasq    6u     IPv4            2168944      0t0       UDP *:domain 
dnsmasq 353652 dnsmasq    7u     IPv4            2168945      0t0       TCP *:domain (LISTEN)
dnsmasq 353652 dnsmasq    8u     IPv4            2168946      0t0       UDP *:tftp 
dnsmasq 353652 dnsmasq    9u     IPv6            2168947      0t0       UDP *:domain 
dnsmasq 353652 dnsmasq   10u     IPv6            2168948      0t0       TCP *:domain (LISTEN)
dnsmasq 353652 dnsmasq   11u     IPv6            2168949      0t0       UDP *:tftp 
dnsmasq 353652 dnsmasq   12r     FIFO               0,13      0t0   2168956 pipe
dnsmasq 353652 dnsmasq   13w     FIFO               0,13      0t0   2168956 pipe
dnsmasq 353652 dnsmasq   14u     unix 0xff2fdbcac7d6c400      0t0   2166214 type=DGRAM (CONNECTED)
dnsmasq 353652 dnsmasq   16r     FIFO               0,13      0t0   2195666 pipe
dnsmasq 353652 dnsmasq   17r     FIFO               0,13      0t0   2180555 pipe
dnsmasq 353652 dnsmasq   18r     FIFO               0,13      0t0   2195669 pipe
dnsmasq 353652 dnsmasq   19r     FIFO               0,13      0t0   2196716 pipe
dnsmasq 353652 dnsmasq   20r     FIFO               0,13      0t0   2183425 pipe
dnsmasq 353652 dnsmasq   21r     FIFO               0,13      0t0   2196719 pipe
dnsmasq 353652 dnsmasq   22r     FIFO               0,13      0t0   2190192 pipe
dnsmasq 353652 dnsmasq   23r     FIFO               0,13      0t0   2168432 pipe
dnsmasq 353652 dnsmasq   24r     FIFO               0,13      0t0   2182506 pipe
dnsmasq 353652 dnsmasq   25r     FIFO               0,13      0t0   2178054 pipe
dnsmasq 353652 dnsmasq   26r     FIFO               0,13      0t0   2184098 pipe
dnsmasq 353652 dnsmasq   27r     FIFO               0,13      0t0   2175961 pipe
dnsmasq 353652 dnsmasq   28r     FIFO               0,13      0t0   2195622 pipe
dnsmasq 353652 dnsmasq   29r     FIFO               0,13      0t0   2173844 pipe
dnsmasq 353652 dnsmasq   30r     FIFO               0,13      0t0   2195642 pipe
dnsmasq 353652 dnsmasq   31r     FIFO               0,13      0t0   2186968 pipe
dnsmasq 353652 dnsmasq   32r     FIFO               0,13      0t0   2195625 pipe
dnsmasq 353652 dnsmasq   33r     FIFO               0,13      0t0   2193009 pipe
dnsmasq 353652 dnsmasq   34r     FIFO               0,13      0t0   2175132 pipe
dnsmasq 353652 dnsmasq   35r     FIFO               0,13      0t0   2169954 pipe
dnsmasq 353652 dnsmasq   36r     FIFO               0,13      0t0   2169860 pipe
dnsmasq 353652 dnsmasq   37r     FIFO               0,13      0t0   2169865 pipe
dnsmasq 353652 dnsmasq   38r     FIFO               0,13      0t0   2169870 pipe
dnsmasq 353652 dnsmasq   39r     FIFO               0,13      0t0   2169889 pipe
dnsmasq 353652 dnsmasq   40r     FIFO               0,13      0t0   2167963 pipe
dnsmasq 353652 dnsmasq   41r     FIFO               0,13      0t0   2169913 pipe
dnsmasq 353652 dnsmasq   42r     FIFO               0,13      0t0   2167986 pipe
dnsmasq 353652 dnsmasq   43r     FIFO               0,13      0t0   2167991 pipe
dnsmasq 353652 dnsmasq   44r     FIFO               0,13      0t0   2168017 pipe
dnsmasq 353652 dnsmasq   45r     FIFO               0,13      0t0   2195632 pipe
dnsmasq 353652 dnsmasq   46r     FIFO               0,13      0t0   2169959 pipe
dnsmasq 353652 dnsmasq   47r     FIFO               0,13      0t0   2169039 pipe
dnsmasq 353652 dnsmasq   48r     FIFO               0,13      0t0   2169044 pipe
dnsmasq 353652 dnsmasq   49r     FIFO               0,13      0t0   2169049 pipe
dnsmasq 353652 dnsmasq   50r     FIFO               0,13      0t0   2168260 pipe
dnsmasq 353652 dnsmasq   51r     FIFO               0,13      0t0   2168265 pipe
dnsmasq 353652 dnsmasq   52r     FIFO               0,13      0t0   2168291 pipe
dnsmasq 353652 dnsmasq   53r     FIFO               0,13      0t0   2166675 pipe
dnsmasq 353652 dnsmasq   54r     FIFO               0,13      0t0   2168424 pipe
dnsmasq 353652 dnsmasq   55r     FIFO               0,13      0t0   2172006 pipe
dnsmasq 353652 dnsmasq   56r     FIFO               0,13      0t0   2172011 pipe
dnsmasq 353652 dnsmasq   57r     FIFO               0,13      0t0   2172016 pipe
dnsmasq 353652 dnsmasq   58r     FIFO               0,13      0t0   2172021 pipe
dnsmasq 353652 dnsmasq   59r     FIFO               0,13      0t0   2168447 pipe
dnsmasq 353652 dnsmasq   60r     FIFO               0,13      0t0   2168437 pipe
dnsmasq 353652 dnsmasq   61r     FIFO               0,13      0t0   2168442 pipe
dnsmasq 353652 dnsmasq   62r     FIFO               0,13      0t0   2181519 pipe
dnsmasq 353652 dnsmasq   63r     FIFO               0,13      0t0   2168452 pipe
dnsmasq 353652 dnsmasq   64r     FIFO               0,13      0t0   2168457 pipe
dnsmasq 353652 dnsmasq   65r     FIFO               0,13      0t0   2168462 pipe
dnsmasq 353652 dnsmasq   66r     FIFO               0,13      0t0   2168467 pipe
dnsmasq 353652 dnsmasq   67r     FIFO               0,13      0t0   2169697 pipe
dnsmasq 353652 dnsmasq   68r     FIFO               0,13      0t0   2168570 pipe
dnsmasq 353652 dnsmasq   69r     FIFO               0,13      0t0   2177237 pipe
dnsmasq 353652 dnsmasq   70r     FIFO               0,13      0t0   2175137 pipe
dnsmasq 353652 dnsmasq   71r     FIFO               0,13      0t0   2175142 pipe
dnsmasq 353652 dnsmasq   72r     FIFO               0,13      0t0   2175522 pipe
dnsmasq 353652 dnsmasq   73r     FIFO               0,13      0t0   2175671 pipe
dnsmasq 353652 dnsmasq   74r     FIFO               0,13      0t0   2174377 pipe
dnsmasq 353652 dnsmasq   75r     FIFO               0,13      0t0   2174382 pipe
dnsmasq 353652 dnsmasq   76r     FIFO               0,13      0t0   2174387 pipe
dnsmasq 353652 dnsmasq   77r     FIFO               0,13      0t0   2173734 pipe
dnsmasq 353652 dnsmasq   78r     FIFO               0,13      0t0   2171677 pipe
dnsmasq 353652 dnsmasq   79r     FIFO               0,13      0t0   2174735 pipe
dnsmasq 353652 dnsmasq   80r     FIFO               0,13      0t0   2171690 pipe
dnsmasq 353652 dnsmasq   81r     FIFO               0,13      0t0   2173854 pipe
dnsmasq 353652 dnsmasq   82r     FIFO               0,13      0t0   2173849 pipe
dnsmasq 353652 dnsmasq   83r     FIFO               0,13      0t0   2193852 pipe
dnsmasq 353652 dnsmasq   84r     FIFO               0,13      0t0   2171869 pipe
dnsmasq 353652 dnsmasq   85r     FIFO               0,13      0t0   2174654 pipe
dnsmasq 353652 dnsmasq   86r     FIFO               0,13      0t0   2174659 pipe
dnsmasq 353652 dnsmasq   87r     FIFO               0,13      0t0   2174664 pipe
dnsmasq 353652 dnsmasq   88r     FIFO               0,13      0t0   2174669 pipe
dnsmasq 353652 dnsmasq   89r     FIFO               0,13      0t0   2174690 pipe
dnsmasq 353652 dnsmasq   90r     FIFO               0,13      0t0   2174695 pipe
dnsmasq 353652 dnsmasq   91r     FIFO               0,13      0t0   2174700 pipe
dnsmasq 353652 dnsmasq   92r     FIFO               0,13      0t0   2174705 pipe
dnsmasq 353652 dnsmasq   93r     FIFO               0,13      0t0   2177141 pipe
dnsmasq 353652 dnsmasq   94r     FIFO               0,13      0t0   2190900 pipe
dnsmasq 353652 dnsmasq   95r     FIFO               0,13      0t0   2189406 pipe
dnsmasq 353652 dnsmasq   96r     FIFO               0,13      0t0   2176478 pipe
dnsmasq 353652 dnsmasq   97r     FIFO               0,13      0t0   2177156 pipe
dnsmasq 353652 dnsmasq   98r     FIFO               0,13      0t0   2177161 pipe
dnsmasq 353652 dnsmasq   99r     FIFO               0,13      0t0   2177166 pipe
dnsmasq 353652 dnsmasq  100r     FIFO               0,13      0t0   2177171 pipe
dnsmasq 353652 dnsmasq  101r     FIFO               0,13      0t0   2193924 pipe
dnsmasq 353652 dnsmasq  102r     FIFO               0,13      0t0   2177226 pipe
dnsmasq 353652 dnsmasq  103r     FIFO               0,13      0t0   2177242 pipe
dnsmasq 353652 dnsmasq  104r     FIFO               0,13      0t0   2177232 pipe
dnsmasq 353652 dnsmasq  105r     FIFO               0,13      0t0   2177247 pipe
dnsmasq 353652 dnsmasq  106r     FIFO               0,13      0t0   2177252 pipe
dnsmasq 353652 dnsmasq  107r     FIFO               0,13      0t0   2174804 pipe
dnsmasq 353652 dnsmasq  108r     FIFO               0,13      0t0   2174809 pipe
dnsmasq 353652 dnsmasq  109r     FIFO               0,13      0t0   2178304 pipe
dnsmasq 353652 dnsmasq  110r     FIFO               0,13      0t0   2178349 pipe
dnsmasq 353652 dnsmasq  111r     FIFO               0,13      0t0   2179221 pipe
dnsmasq 353652 dnsmasq  112r     FIFO               0,13      0t0   2179226 pipe
dnsmasq 353652 dnsmasq  113r     FIFO               0,13      0t0   2179260 pipe
dnsmasq 353652 dnsmasq  114r     FIFO               0,13      0t0   2179265 pipe
dnsmasq 353652 dnsmasq  115r     FIFO               0,13      0t0   2177542 pipe
dnsmasq 353652 dnsmasq  116r     FIFO               0,13      0t0   2176950 pipe
dnsmasq 353652 dnsmasq  117r     FIFO               0,13      0t0   2176955 pipe
dnsmasq 353652 dnsmasq  118r     FIFO               0,13      0t0   2180533 pipe
dnsmasq 353652 dnsmasq  119r     FIFO               0,13      0t0   2178566 pipe
dnsmasq 353652 dnsmasq  120r     FIFO               0,13      0t0   2177772 pipe
dnsmasq 353652 dnsmasq  121r     FIFO               0,13      0t0   2192777 pipe
dnsmasq 353652 dnsmasq  122r     FIFO               0,13      0t0   2178570 pipe
dnsmasq 353652 dnsmasq  123r     FIFO               0,13      0t0   2180560 pipe
dnsmasq 353652 dnsmasq  124r     FIFO               0,13      0t0   2181014 pipe
dnsmasq 353652 dnsmasq  125r     FIFO               0,13      0t0   2178835 pipe
dnsmasq 353652 dnsmasq  126r     FIFO               0,13      0t0   2178872 pipe
dnsmasq 353652 dnsmasq  127r     FIFO               0,13      0t0   2195635 pipe
dnsmasq 353652 dnsmasq  128r     FIFO               0,13      0t0   2190638 pipe
dnsmasq 353652 dnsmasq  129r     FIFO               0,13      0t0   2183249 pipe
dnsmasq 353652 dnsmasq  130r     FIFO               0,13      0t0   2181886 pipe
dnsmasq 353652 dnsmasq  131r     FIFO               0,13      0t0   2181891 pipe
dnsmasq 353652 dnsmasq  132r     FIFO               0,13      0t0   2181896 pipe
dnsmasq 353652 dnsmasq  133r     FIFO               0,13      0t0   2181901 pipe
dnsmasq 353652 dnsmasq  134r     FIFO               0,13      0t0   2181906 pipe
dnsmasq 353652 dnsmasq  135r     FIFO               0,13      0t0   2181911 pipe
dnsmasq 353652 dnsmasq  136r     FIFO               0,13      0t0   2181916 pipe
dnsmasq 353652 dnsmasq  137r     FIFO               0,13      0t0   2181921 pipe
dnsmasq 353652 dnsmasq  138r     FIFO               0,13      0t0   2181926 pipe
dnsmasq 353652 dnsmasq  139r     FIFO               0,13      0t0   2181950 pipe
dnsmasq 353652 dnsmasq  140r     FIFO               0,13      0t0   2181955 pipe
dnsmasq 353652 dnsmasq  141r     FIFO               0,13      0t0   2181960 pipe
dnsmasq 353652 dnsmasq  142r     FIFO               0,13      0t0   2181965 pipe
dnsmasq 353652 dnsmasq  143r     FIFO               0,13      0t0   2179942 pipe
dnsmasq 353652 dnsmasq  144r     FIFO               0,13      0t0   2181972 pipe
dnsmasq 353652 dnsmasq  145r     FIFO               0,13      0t0   2181977 pipe
dnsmasq 353652 dnsmasq  146r     FIFO               0,13      0t0   2181982 pipe
dnsmasq 353652 dnsmasq  147r     FIFO               0,13      0t0   2179947 pipe
dnsmasq 353652 dnsmasq  148r     FIFO               0,13      0t0   2179952 pipe
dnsmasq 353652 dnsmasq  149r     FIFO               0,13      0t0   2179957 pipe
dnsmasq 353652 dnsmasq  150r     FIFO               0,13      0t0   2179962 pipe
dnsmasq 353652 dnsmasq  151r     FIFO               0,13      0t0   2191441 pipe
dnsmasq 353652 dnsmasq  152r     FIFO               0,13      0t0   2182788 pipe
dnsmasq 353652 dnsmasq  153r     FIFO               0,13      0t0   2182832 pipe
dnsmasq 353652 dnsmasq  154r     FIFO               0,13      0t0   2182837 pipe
dnsmasq 353652 dnsmasq  155r     FIFO               0,13      0t0   2182842 pipe
dnsmasq 353652 dnsmasq  156r     FIFO               0,13      0t0   2182847 pipe
dnsmasq 353652 dnsmasq  157r     FIFO               0,13      0t0   2182852 pipe
dnsmasq 353652 dnsmasq  158r     FIFO               0,13      0t0   2182857 pipe
dnsmasq 353652 dnsmasq  159r     FIFO               0,13      0t0   2182862 pipe
dnsmasq 353652 dnsmasq  160r     FIFO               0,13      0t0   2182867 pipe
dnsmasq 353652 dnsmasq  161r     FIFO               0,13      0t0   2184771 pipe
dnsmasq 353652 dnsmasq  162r     FIFO               0,13      0t0   2184776 pipe
dnsmasq 353652 dnsmasq  163r     FIFO               0,13      0t0   2184781 pipe
dnsmasq 353652 dnsmasq  164r     FIFO               0,13      0t0   2184786 pipe
dnsmasq 353652 dnsmasq  165r     FIFO               0,13      0t0   2184791 pipe
dnsmasq 353652 dnsmasq  166r     FIFO               0,13      0t0   2184796 pipe
dnsmasq 353652 dnsmasq  167r     FIFO               0,13      0t0   2184801 pipe
dnsmasq 353652 dnsmasq  168r     FIFO               0,13      0t0   2186319 pipe
dnsmasq 353652 dnsmasq  169r     FIFO               0,13      0t0   2185620 pipe
dnsmasq 353652 dnsmasq  170r     FIFO               0,13      0t0   2185622 pipe
dnsmasq 353652 dnsmasq  171r     FIFO               0,13      0t0   2188284 pipe
dnsmasq 353652 dnsmasq  172r     FIFO               0,13      0t0   2188289 pipe
dnsmasq 353652 dnsmasq  173r     FIFO               0,13      0t0   2188294 pipe
dnsmasq 353652 dnsmasq  174r     FIFO               0,13      0t0   2188299 pipe
dnsmasq 353652 dnsmasq  175r     FIFO               0,13      0t0   2188304 pipe
dnsmasq 353652 dnsmasq  176r     FIFO               0,13      0t0   2184088 pipe
dnsmasq 353652 dnsmasq  177r     FIFO               0,13      0t0   2184093 pipe
dnsmasq 353652 dnsmasq  178r     FIFO               0,13      0t0   2197656 pipe
dnsmasq 353652 dnsmasq  179r     FIFO               0,13      0t0   2184103 pipe
dnsmasq 353652 dnsmasq  180r     FIFO               0,13      0t0   2184108 pipe
dnsmasq 353652 dnsmasq  181r     FIFO               0,13      0t0   2184113 pipe
dnsmasq 353652 dnsmasq  182r     FIFO               0,13      0t0   2186668 pipe
dnsmasq 353652 dnsmasq  183r     FIFO               0,13      0t0   2186673 pipe
dnsmasq 353652 dnsmasq  184r     FIFO               0,13      0t0   2194155 pipe
dnsmasq 353652 dnsmasq  185r     FIFO               0,13      0t0   2189411 pipe
dnsmasq 353652 dnsmasq  186r     FIFO               0,13      0t0   2186013 pipe
dnsmasq 353652 dnsmasq  187r     FIFO               0,13      0t0   2188672 pipe
dnsmasq 353652 dnsmasq  188r     FIFO               0,13      0t0   2189921 pipe
dnsmasq 353652 dnsmasq  189r     FIFO               0,13      0t0   2189926 pipe
dnsmasq 353652 dnsmasq  190r     FIFO               0,13      0t0   2187053 pipe
dnsmasq 353652 dnsmasq  191r     FIFO               0,13      0t0   2186973 pipe
dnsmasq 353652 dnsmasq  192r     FIFO               0,13      0t0   2186978 pipe
dnsmasq 353652 dnsmasq  193r     FIFO               0,13      0t0   2186983 pipe
dnsmasq 353652 dnsmasq  194r     FIFO               0,13      0t0   2190089 pipe
dnsmasq 353652 dnsmasq  195r     FIFO               0,13      0t0   2195645 pipe
dnsmasq 353652 dnsmasq  196r     FIFO               0,13      0t0   2190094 pipe
dnsmasq 353652 dnsmasq  197r     FIFO               0,13      0t0   2190419 pipe
dnsmasq 353652 dnsmasq  198r     FIFO               0,13      0t0   2190422 pipe
dnsmasq 353652 dnsmasq  199r     FIFO               0,13      0t0   2197653 pipe
dnsmasq 353652 dnsmasq  200r     FIFO               0,13      0t0   2190197 pipe
dnsmasq 353652 dnsmasq  201r     FIFO               0,13      0t0   2187162 pipe
dnsmasq 353652 dnsmasq  202r     FIFO               0,13      0t0   2191364 pipe
dnsmasq 353652 dnsmasq  203r     FIFO               0,13      0t0   2190559 pipe
dnsmasq 353652 dnsmasq  204r     FIFO               0,13      0t0   2187180 pipe
dnsmasq 353652 dnsmasq  205r     FIFO               0,13      0t0   2190574 pipe
dnsmasq 353652 dnsmasq  206r     FIFO               0,13      0t0   2190266 pipe
dnsmasq 353652 dnsmasq  207r     FIFO               0,13      0t0   2190650 pipe
dnsmasq 353652 dnsmasq  208r     FIFO               0,13      0t0   2191946 pipe
dnsmasq 353652 dnsmasq  209r     FIFO               0,13      0t0   2190644 pipe
dnsmasq 353652 dnsmasq  210r     FIFO               0,13      0t0   2190655 pipe
dnsmasq 353652 dnsmasq  211r     FIFO               0,13      0t0   2190660 pipe
dnsmasq 353652 dnsmasq  212r     FIFO               0,13      0t0   2190665 pipe
dnsmasq 353652 dnsmasq  213r     FIFO               0,13      0t0   2192069 pipe
dnsmasq 353652 dnsmasq  214r     FIFO               0,13      0t0   2191446 pipe
dnsmasq 353652 dnsmasq  215r     FIFO               0,13      0t0   2191451 pipe
dnsmasq 353652 dnsmasq  216r     FIFO               0,13      0t0   2192566 pipe
dnsmasq 353652 dnsmasq  217r     FIFO               0,13      0t0   2192568 pipe
dnsmasq 353652 dnsmasq  218r     FIFO               0,13      0t0   2195514 pipe
dnsmasq 353652 dnsmasq  219r     FIFO               0,13      0t0   2190910 pipe
dnsmasq 353652 dnsmasq  220r     FIFO               0,13      0t0   2191853 pipe
dnsmasq 353652 dnsmasq  221r     FIFO               0,13      0t0   2191858 pipe
dnsmasq 353652 dnsmasq  222r     FIFO               0,13      0t0   2191863 pipe
dnsmasq 353652 dnsmasq  223r     FIFO               0,13      0t0   2191868 pipe
dnsmasq 353652 dnsmasq  224r     FIFO               0,13      0t0   2191873 pipe
dnsmasq 353652 dnsmasq  225r     FIFO               0,13      0t0   2191878 pipe
dnsmasq 353652 dnsmasq  226r     FIFO               0,13      0t0   2191883 pipe
dnsmasq 353652 dnsmasq  227r     FIFO               0,13      0t0   2191888 pipe
dnsmasq 353652 dnsmasq  228r     FIFO               0,13      0t0   2191936 pipe
dnsmasq 353652 dnsmasq  229r     FIFO               0,13      0t0   2191941 pipe
dnsmasq 353652 dnsmasq  230r     FIFO               0,13      0t0   2195652 pipe
dnsmasq 353652 dnsmasq  231r     FIFO               0,13      0t0   2192000 pipe
dnsmasq 353652 dnsmasq  232r     FIFO               0,13      0t0   2192005 pipe
dnsmasq 353652 dnsmasq  233r     FIFO               0,13      0t0   2192010 pipe
dnsmasq 353652 dnsmasq  234r     FIFO               0,13      0t0   2192015 pipe
dnsmasq 353652 dnsmasq  235r     FIFO               0,13      0t0   2192020 pipe
dnsmasq 353652 dnsmasq  236r     FIFO               0,13      0t0   2193647 pipe
dnsmasq 353652 dnsmasq  237r     FIFO               0,13      0t0   2193652 pipe
dnsmasq 353652 dnsmasq  238r     FIFO               0,13      0t0   2196732 pipe
dnsmasq 353652 dnsmasq  239r     FIFO               0,13      0t0   2192105 pipe
dnsmasq 353652 dnsmasq  240r     FIFO               0,13      0t0   2195655 pipe
dnsmasq 353652 dnsmasq  241r     FIFO               0,13      0t0   2192167 pipe
dnsmasq 353652 dnsmasq  242r     FIFO               0,13      0t0   2195192 pipe
dnsmasq 353652 dnsmasq  243r     FIFO               0,13      0t0   2193857 pipe
dnsmasq 353652 dnsmasq  244r     FIFO               0,13      0t0   2195658 pipe
dnsmasq 353652 dnsmasq  245r     FIFO               0,13      0t0   2193929 pipe
dnsmasq 353652 dnsmasq  246r     FIFO               0,13      0t0   2193934 pipe
dnsmasq 353652 dnsmasq  247r     FIFO               0,13      0t0   2194999 pipe
dnsmasq 353652 dnsmasq  248r     FIFO               0,13      0t0   2195004 pipe
dnsmasq 353652 dnsmasq  249r     FIFO               0,13      0t0   2195661 pipe
dnsmasq 353652 dnsmasq  250r     FIFO               0,13      0t0   2193014 pipe
dnsmasq 353652 dnsmasq  251r     FIFO               0,13      0t0   2196735 pipe
dnsmasq 353652 dnsmasq  252r     FIFO               0,13      0t0   2195197 pipe
dnsmasq 353652 dnsmasq  253r     FIFO               0,13      0t0   2195279 pipe
dnsmasq 353652 dnsmasq  254r     FIFO               0,13      0t0   2195598 pipe
dnsmasq 353652 dnsmasq  255r     FIFO               0,13      0t0   2194160 pipe
dnsmasq 353652 dnsmasq  256r     FIFO               0,13      0t0   2195346 pipe
dnsmasq 353652 dnsmasq  257r     FIFO               0,13      0t0   2195351 pipe
dnsmasq 353652 dnsmasq  258r     FIFO               0,13      0t0   2195396 pipe
dnsmasq 353652 dnsmasq  260r     FIFO               0,13      0t0   2195519 pipe
dnsmasq 353652 dnsmasq  261r     FIFO               0,13      0t0   2195525 pipe
dnsmasq 353652 dnsmasq  262r     FIFO               0,13      0t0   2195530 pipe
dnsmasq 353652 dnsmasq  263r     FIFO               0,13      0t0   2196603 pipe
dnsmasq 353652 dnsmasq  264r     FIFO               0,13      0t0   2195564 pipe
dnsmasq 353652 dnsmasq  266r     FIFO               0,13      0t0   2195603 pipe
Comment 6 Dave Cain 2023-04-24 11:17:19 UTC 
Created attachment 1959510 [details]
bz2188443-dnsmasqconf-24-april.txt

Understood on restarting every 3 hours, I am doing that presently :)

Here's my dnsmasq.conf.  Happy to share the sub configurations also, they are simply address and DHCP-host records, not much more.
Comment 7 Petr Menšík 2023-05-04 10:56:51 UTC 
Interesting, there are not too many network sockets, but too many FIFO pipes. I thought there should be always just two sockets of pipe to send messages back from helper or maybe from TCP forks. Not yet sure why they are not closed properly and why those pipes are even opened. It might be related to tftp, which is not usually enabled in common configurations.

Have you tried disabling tftp support in dnsmasq and using tftp-server instead? Would it help preventing those leaks?
Comment 8 Dave Cain 2023-05-05 02:26:02 UTC 
Does not seem to help.  Commented out:
#enable-tftp
#tftp-root=/var/lib/tftpboot

Back up to 44 FIFO  after 15 minutes.
Comment 9 Dave Cain 2023-06-02 21:09:07 UTC 
Any other suggestions?  Should I try upstream 2.89 to see if the same behavior exists?
Comment 10 Petr Menšík 2023-06-03 14:29:47 UTC 
I guess they could be TCP worker pipes, which are used from child forked process to insert cache records into master process.

You could rule that out by using -d parameter. That would make quite bad TCP performance, but avoids spawning tcp_pipes in dnsmasq_daemon->tcp_pipes. I just guess attaching strace to main dnsmasq process might reveal what calls are done after pipe() calls. I am not sure how to identify the place, where those pipes are created and why they are not closed. Not sure why I haven't ever saw so high number of pipes in my testing.

Do you change ulimit -a of the dnsmasq? Is it possible fork() is failing and such failure is causing pipe file descriptor leak?

tcp_pipes size is limited to 20. So your number of pipes is far higher in that. I haven't found candidate place by looking at the code.

Can you try to capture output of:

# strace -p ${PID} -e t=pipe2,close,write,writev -e decode-fd=socket

Whether it would hint what happens close to pipe calls, whether descriptors are closed and in which case it might leave it open.
Not sure how else to find the cause of those leaks.
Comment 11 Petr Menšík 2023-06-03 15:19:00 UTC 
I may have found some fix missing in our version after all. Upstream commit ad90eb07 [1] fixes case when too many TCP connections are received. It makes sure it accepts the connection only if it can find free TCP slot first. That might help you, in your strace were pipe failed followed by BADF error. I think that is related to that case.

It tried to use pipefd in else branch on dnsmasq.c:1969, but that pipefd failed to be set when pipe returned -1. Since it is not properly initialized, I think it can contain random numbers, which are then closed. I will try to prepare fixed candidate for testing next week.

[1] https://thekelleys.org.uk/gitweb/?p=dnsmasq.git;h=ad90eb075dfeeb1936e8bc0f323fcc23f89364d4
Comment 12 Dave Cain 2023-06-06 02:50:48 UTC 
Good to hear Petr.  Happy to test, just let me know.
Comment 13 Petr Menšík 2023-06-09 14:19:53 UTC 
Can you please try MR #18 [1], whether the issue is still there?

1 https://gitlab.com/redhat/centos-stream/rpms/dnsmasq/-/merge_requests/18
Comment 14 Petr Menšík 2023-06-09 15:52:12 UTC 
tried throwing just higher number of tcp queries on dnsmasq, but that does not reproduce leaking descriptors

# for I in {1..200}; do echo $RANDOM.home.arpa a; echo $RANDOM.home.arpa aaaa; done > queries2

# dnsperf -d queries -s 127.0.0.2 -m tcp -l 10

Used bind listening on 127.0.0.1 to answer those queries. As expected, TCP performance of dnsmasq is quite tragic:

tatistics:

  Queries sent:         500
  Queries completed:    300 (60.00%)
  Queries lost:         200 (40.00%)

  Response codes:       NXDOMAIN 300 (100.00%)
  Average packet size:  request 32, response 76
  Run time (s):         10.000089
  Queries per second:   29.999733

  Average Latency (s):  0.001131 (min 0.000060, max 0.004839)
  Latency StdDev (s):   0.001306

Connection Statistics:

  Reconnections:        2 (66.67% of 3 connections)

  Average Latency (s):  0.000261 (min 0.000020, max 0.000416)
  Latency StdDev (s):   0.000212

Compared to just bind directly:
Statistics:

  Queries sent:         310695
  Queries completed:    310695 (100.00%)
  Queries lost:         0 (0.00%)

  Response codes:       NXDOMAIN 310695 (100.00%)
  Average packet size:  request 32, response 76
  Run time (s):         10.040069
  Queries per second:   30945.504458

  Average Latency (s):  0.003192 (min 0.000032, max 0.042978)
  Latency StdDev (s):   0.000693

Connection Statistics:

  Reconnections:        0 (0.00% of 1 connections)

  Average Latency (s):  0.000173 (min 0.000173, max 0.000173)

But this does not cause dnsmasq to leak pipe descriptors.
dnsmasq 5186 dnsmasq    0u      CHR                1,3      0t0       4 /dev/null
dnsmasq 5186 dnsmasq    1u      CHR                1,3      0t0       4 /dev/null
dnsmasq 5186 dnsmasq    2u      CHR                1,3      0t0       4 /dev/null
dnsmasq 5186 dnsmasq    3u  netlink                         0t0   32908 ROUTE
dnsmasq 5186 dnsmasq    4u     IPv4              32910      0t0     UDP 127.0.0.2:domain 
dnsmasq 5186 dnsmasq    5u     IPv4              32911      0t0     TCP 127.0.0.2:domain (LISTEN)
dnsmasq 5186 dnsmasq    6r     FIFO               0,13      0t0   32916 pipe
dnsmasq 5186 dnsmasq    7w     FIFO               0,13      0t0   32916 pipe
dnsmasq 5186 dnsmasq    8u     unix 0xff29a7048326bb80      0t0   32918 type=DGRAM (CONNECTED)
Comment 15 Dave Cain 2023-06-26 10:28:18 UTC 
Hi Petr,

After running this build (dnsmasq-2.85-11.el9.x86_64) over the weekend, I see at maximum now 18 pipe processes associated to the master pid of dnsmasq.  This looks to be massively improved over the previous behavior, I would say this addresses my original problem reported.  I'll keep running this, and am happy to gather any further diagnostic or logging data.  Just let me know what you may need.
Comment 16 Dave Cain 2023-07-14 02:28:31 UTC 
Hi Petr,

Just to report this has been stable for me after 2 weeks, no issues.  Can we merge this fix?
Comment 17 Petr Menšík 2023-07-17 17:00:54 UTC 
Upstream explanation how that happens:
https://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2021q2/014976.html

I were not able create reproducer, but reading that explanation. Maybe I would have to run 2 dnsperf instances, one on IPv4, one on IPv6. Dunno.

Is it okay to merge it without working reproducer?
Comment 21 Petr Menšík 2023-09-07 15:20:41 UTC 
Unfortunately this won't be fixed in 9.3 unless z-stream is requested. I have not been able to create working automated reproducer (yet).
Comment 22 RHEL Program Management 2023-09-21 18:52:18 UTC 
Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.
Comment 23 RHEL Program Management 2023-09-21 19:06:13 UTC 
This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.
Note You need to log in before you can comment on or make changes to this bug.