Bug 534117 - dhcpd memory leak when failover configured
Summary: dhcpd memory leak when failover configured
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: dhcp
Version: 5.4
Hardware: All
OS: Linux
urgent
high
Target Milestone: rc
: ---
Assignee: Jiri Popelka
QA Contact: Alexander Todorov
URL:
Whiteboard:
Depends On:
Blocks: 552210 552211
TreeView+ depends on / blocked
 
Reported: 2009-11-10 16:10 UTC by Martin Poole
Modified: 2018-10-27 15:21 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2010-03-30 08:18:48 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2010:0223 0 normal SHIPPED_LIVE dhcp bug fix update 2010-03-29 12:33:41 UTC

Description Martin Poole 2009-11-10 16:10:03 UTC
Description of problem:

dhcpd leaks memory for each DISCOVER when configured for failover


.............................................
DHCP Server 1 - Before restart
top - 08:00:03 up 20 days, 23:07,  1 user,  load average: 0.08, 0.08, 0.12
Tasks:  80 total,   3 running,  77 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.5%us,  1.3%sy,  0.0%ni, 96.8%id,  0.2%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:   2075356k total,  1523968k used,   551388k free,   156696k buffers
Swap:  3068372k total,    17860k used,  3050512k free,   522792k cached

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
26731 root      15   0  773m 771m  616 R  4.7 38.1  33:51.36 dhcpd

DHCP Server 1 - immediately after restart
top - 08:03:38 up 20 days, 23:10,  1 user,  load average: 0.16, 0.10, 0.11
Tasks:  80 total,   1 running,  79 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.1%us,  0.1%sy,  0.0%ni, 99.5%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   2075356k total,   741272k used,  1334084k free,   156828k buffers
Swap:  3068372k total,    17860k used,  3050512k free,   497184k cached

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
29230 root      15   0 35032  32m  584 S  0.4  1.6   0:01.84 dhcpd
..............................................



Version-Release number of selected component (if applicable):

dhcp-3.0.5-21.el5-i386

How reproducible:

Always.

Steps to Reproduce:

Configure failover

failover peer "dhcp-failover" {
 primary; # declare this to be the primary server
 address 10.20.121.225;
 port 520;
 peer address 10.20.121.226;
 peer port 520;
 max-response-delay 30;
 max-unacked-updates 10;
 # load balance max seconds 3;
 load balance max seconds 6;
 mclt 1800;
 split 128;
}

-Start dhcpd
-Observe memory consumption with top or ps


  

Additional info:


......................................
diff -up dhcp-3.0.5/server/failover.c.orig dhcp-3.0.5/server/failover.c
--- dhcp-3.0.5/server/failover.c.orig 2009-11-02 13:23:33.000000000 -0500
+++ dhcp-3.0.5/server/failover.c 2009-11-02 13:23:50.000000000 -0500
@@ -5040,6 +5040,8 @@ int load_balance_mine (struct packet *pa
  packet -> options, (struct option_state *)0,
  &global_scope, oc, MDL)) {
hbaix = loadb_p_hash (ds.data, ds.len);
+
+                data_string_forget(&ds, MDL);
} else {
hbaix = loadb_p_hash (packet -> raw -> chaddr,
     packet -> raw -> hlen);
.......................................

This patch comes from the upstream 3.13 version and the description of this fix is:

 "Memory leak in the load_balance_mine() function is fixed. This would leak ~20-30 octets per DHCPDISCOVER packet while failover was in use and in normal state."

This matches the customer environment and configuration. A test package with this patch has been tried and it did correct the issue.  The upstream changelog is at:

 http://oldwww.isc.org/sw/dhcp/dhcp_rel2.php?noframes=1

Comment 10 Alexander Todorov 2010-01-05 14:31:36 UTC
(In reply to comment #9)
> Event posted on 01-04-2010 09:18am EST by jruemker
> 
> Customer was satisfied with hotfix and allowed CRM to close.  Thanks!
> 
> John
> 

Can we get more information about the customer's environment/setup and how they have verified the hotfix? 

In our testing dhcpd still seems to be leaking memory although that looks like a different bug which is fixed upstream. (There are several memleak fixes upstream indeed). QE can't verify a single memleak fix if the whole program is still leaking.

Comment 11 John Ruemker 2010-01-06 16:16:03 UTC
The customer was able to reproduce the issue simply by letting dhcpd run for more than a day and they would monitor the memory usage slowly climb in top/ps until all available memory would be consumed by it.  They installed the hotfix package and this no longer occurred. 

To reproduce the problem, they used a failover configuration as seen in the above description.  

Let me know if there's any other info I can provide.

Comment 12 Alexander Todorov 2010-01-08 11:35:57 UTC
QE has performed some more testing and the results are:

We can confirm the leakage in dhcp-3.0.5-21.el5: tested for 15 min and the leakage was some 17 MB. We didn't notice any considerable leak in case of the patched package dhcp-3.0.5-22.el5. Just some additional 3 or 4 megs in 30 minutes were used, but that doesn't need to be a dhcpd's memleak.

We believe that the leak in load_balance_mine() function has been properly
fixed and we're moving this bug to VERIFIED. 

There however may be other memory leaks depending on your setup and patches for
them may not have been pulled from upstream. If you experience further issues
please file a separate bug.

Comment 15 errata-xmlrpc 2010-03-30 08:18:48 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2010-0223.html


Note You need to log in before you can comment on or make changes to this bug.