Bug 491172 - RH4.7: glibc-related CGI segmentation faults in httpd and lots of mem errors in valgrind
RH4.7: glibc-related CGI segmentation faults in httpd and lots of mem errors ...
Status: CLOSED CANTFIX
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: glibc (Show other bugs)
4.7
i386 Linux
low Severity urgent
: ---
: ---
Assigned To: Jakub Jelinek
BaseOS QE
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-03-19 13:21 EDT by justin
Modified: 2009-04-18 04:34 EDT (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-04-18 04:34:04 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description justin 2009-03-19 13:21:48 EDT
Description of problem: 

When calling CGI files a segmentation fault started happening sporadically about 1 week ago. We have to keep restoring our monitoring server configuration from backup every 48 hours just to keep it on life support. To address this, I updated the OS from RH 4.6 RH to 4.7 and from nagios 3.03 to 3.06 (I've posted this to the nagios developers mailing list also.) Here is the diagnostics data I have collected so far:


[root@nagios01 nagmin]# uname -a
Linux nagios01 2.6.9-78.0.13.ELsmp #1 SMP Wed Jan 7 17:52:47 EST 2009 i686 i686 i386 GNU/Linux
[root@nagios01 nagmin]# rpm -qa | grep glibc
glibc-headers-2.3.4-2.41.el4_7.1
glibc-kernheaders-2.4-9.1.103.EL
glibc-common-2.3.4-2.41.el4_7.1
glibc-2.3.4-2.41.el4_7.1
glibc-devel-2.3.4-2.41.el4_7.1


Errors:

From httpd/error_log, this repeats over and over:

[Thu Mar 19 03:12:02 2009] [error] [client 123.45.67.89] *** glibc detected *** free(): invalid next size (normal): 0x08cac4b8 ***, referer: http://123.45.67.89/nagios/cgi-bin/tac.cgi
[Thu Mar 19 03:12:02 2009] [error] [client 123.45.67.89] Premature end of script headers: status.cgi, referer: http://123.45.67.89/nagios/cgi-bin/tac.cgi
[Thu Mar 19 03:12:07 2009] [error] [client 123.45.67.89] *** glibc detected *** free(): invalid next size (fast): 0x097a44f0 ***

[root@nagios01 nagmin]# export REQUEST_METHOD="GET"
[root@nagios01 nagmin]# /usr/local/nagios/sbin/status.cgi
When it doesn’t work it outputs this single line: Segmentation fault

Now some output from valgrind:

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2009.03.19 09:44:35 =~=~=~=~=~=~=~=~=~=~=~=
valgrind --tool=memcheck /usr/local/nagios/sbin/status.cgi
==23834== Memcheck, a memory error detector.
==23834== Copyright (C) 2002-2005, and GNU GPL'd, by Julian Seward et al.
==23834== Using LibVEX rev 1575, a library for dynamic binary translation.
==23834== Copyright (C) 2004-2005, and GNU GPL'd, by OpenWorks LLP.
==23834== Using valgrind-3.1.1, a dynamic binary instrumentation framework.
==23834== Copyright (C) 2000-2005, and GNU GPL'd, by Julian Seward et al.
==23834== For more details, rerun with: -v
==23834== 
==23834== Invalid read of size 1
==23834==    at 0x8054D9D: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x80552D3: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x8052C63: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x24BDF2: (below main) (in /lib/tls/libc-2.3.4.so)
==23834==  Address 0x4019A4F is 1 bytes before a block of size 2 alloc'd
==23834==    at 0x4004405: malloc (vg_replace_malloc.c:149)
==23834==    by 0x8054DC9: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x80552D3: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x8052C63: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x24BDF2: (below main) (in /lib/tls/libc-2.3.4.so)
==23834== 
==23834== Invalid read of size 1
==23834==    at 0x8054D9D: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x8055596: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x8052C63: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x24BDF2: (below main) (in /lib/tls/libc-2.3.4.so)
==23834==  Address 0x4019F8F is 1 bytes before a block of size 2 alloc'd
==23834==    at 0x4004405: malloc (vg_replace_malloc.c:149)
==23834==    by 0x8054DC9: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x8055596: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x8052C63: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x24BDF2: (below main) (in /lib/tls/libc-2.3.4.so)
==23834== 
==23834== Invalid read of size 1
==23834==    at 0x8054D9D: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x8054E51: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x8052C7C: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x24BDF2: (below main) (in /lib/tls/libc-2.3.4.so)
==23834==  Address 0x4025BDF is 1 bytes before a block of size 2 alloc'd
==23834==    at 0x4004405: malloc (vg_replace_malloc.c:149)
==23834==    by 0x8054DC9: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x8054E51: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x8052C7C: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x24BDF2: (below main) (in /lib/tls/libc-2.3.4.so)
==23834== 
==23834== Invalid read of size 1
==23834==    at 0x8054D9D: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x8060076: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x806D4ED: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x805B36F: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x8054194: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x8052C9A: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x24BDF2: (below main) (in /lib/tls/libc-2.3.4.so)
==23834==  Address 0x404F817 is 1 bytes before a block of size 2 alloc'd
==23834==    at 0x4004405: malloc (vg_replace_malloc.c:149)
==23834==    by 0x8054DC9: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x8060076: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x806D4ED: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x805B36F: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x8054194: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x8052C9A: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x24BDF2: (below main) (in /lib/tls/libc-2.3.4.so)
==23834== 
==23834== Invalid read of size 1
==23834==    at 0x8054D9D: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x8069ECB: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x806D5D4: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x805B36F: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x8054194: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x8052C9A: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x24BDF2: (below main) (in /lib/tls/libc-2.3.4.so)
==23834==  Address 0x407A177 is 1 bytes before a block of size 2 alloc'd
==23834==    at 0x4004405: malloc (vg_replace_malloc.c:149)
==23834==    by 0x8054DC9: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x8069ECB: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x806D5D4: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x805B36F: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x8054194: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x8052C9A: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x24BDF2: (below main) (in /lib/tls/libc-2.3.4.so)
==23834== 
==23834== Invalid read of size 1
==23834==    at 0x8054D9D: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x806E600: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x806E842: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x80541CC: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x8052CB9: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x24BDF2: (below main) (in /lib/tls/libc-2.3.4.so)
==23834==  Address 0x473172F is 1 bytes before a block of size 2 alloc'd
==23834==    at 0x4004405: malloc (vg_replace_malloc.c:149)
==23834==    by 0x8054DC9: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x806E600: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x806E842: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x80541CC: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x8052CB9: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x24BDF2: (below main) (in /lib/tls/libc-2.3.4.so)
==23834== 
==23834== Invalid read of size 1
==23834==    at 0x8054D9D: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x806E68E: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x806E842: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x80541CC: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x8052CB9: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x24BDF2: (below main) (in /lib/tls/libc-2.3.4.so)
==23834==  Address 0x483B91F is 1 bytes before a block of size 2 alloc'd
==23834==    at 0x4004405: malloc (vg_replace_malloc.c:149)
==23834==    by 0x8054DC9: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x806E68E: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x806E842: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x80541CC: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x8052CB9: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x24BDF2: (below main) (in /lib/tls/libc-2.3.4.so)
Cache-Control: no-store
Pragma: no-cache
Refresh: 90
Last-Modified: Thu, 19 Mar 2009 16:44:39 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-type: text/html

<html>
<head>
<link rel="shortcut icon" href="/nagios/images/favicon.ico" type="image/ico">
<title>
Current Network Status
</title>
<LINK REL='stylesheet' TYPE='text/css' HREF='/nagios/stylesheets/common.css'><LINK REL='stylesheet' TYPE='text/css' HREF='/nagios/stylesheets/status.css'></head>
<body CLASS='status'>

<!-- Produced by Nagios (http://www.nagios.org).  Copyright (c) 1999-2007 Ethan Galstad. -->
==23834== 
==23834== Invalid read of size 1
==23834==    at 0x8054D9D: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x80571AD: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x8052CE0: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x24BDF2: (below main) (in /lib/tls/libc-2.3.4.so)
==23834==  Address 0x4557717 is 1 bytes before a block of size 2 alloc'd
==23834==    at 0x4004405: malloc (vg_replace_malloc.c:149)
==23834==    by 0x8054DC9: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x80571AD: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x8052CE0: (within /usr/local/nagios/sbin/status.cgi)
==23834==    by 0x24BDF2: (below main) (in /lib/tls/libc-2.3.4.so)
<table border=0 width=100% cellspacing=0 cellpadding=0>
<tr>
<td align=left valign=top width=33%>
<TABLE CLASS='infoBox' BORDER=1 CELLSPACING=0 CELLPADDING=0>
<TR><TD CLASS='infoBox'>
<DIV CLASS='infoBoxTitle'>Current Network Status</DIV>
Last Updated: Thu Mar 19 09:44:57 PDT 2009<BR>
Updated every 90 seconds<br>
Nagios&reg; 3.0.3 - <A HREF='http://www.nagios.org' TARGET='_new' CLASS='homepageURL'>www.nagios.org</A><BR>
Logged in as <i>?</i><BR>
</TD></TR>
</TABLE>
<TABLE BORDER=1 CELLPADDING=0 CELLSPACING=0 CLASS='linkBox'>
<TR><TD CLASS='linkBox'>
<a href='history.cgi?host=all'>View History For all hosts</a><br>
<a href='notifications.cgi?host=all'>View Notifications For All Hosts</a>
<br><a href='status.cgi?hostgroup=all&style=hostdetail'>View Host Status Detail For All Hosts</a>
</TD></TR>
</TABLE>
</td>
<td align=center valign=top width=33%>
<DIV CLASS='hostTotals'>Host Status Totals</DIV>
<TABLE BORDER=0 CELLSPACING=0 CELLPADDING=0>
<TR><TD>
<TABLE BORDER=1 CLASS='hostTotals'>
<TR>
<TH CLASS='hostTotals'><A CLASS='hostTotals' HREF='status.cgi?host=(null)&hoststatustypes=2'>Up</A></TH>
<TH CLASS='hostTotals'><A CLASS='hostTotals' HREF='status.cgi?host=(null)&hoststatustypes=4'>Down</A></TH>
<TH CLASS='hostTotals'><A CLASS='hostTotals' HREF='status.cgi?host=(null)&hoststatustypes=8'>Unreachable</A></TH>
<TH CLASS='hostTotals'><A CLASS='hostTotals' HREF='status.cgi?host=(null)&hoststatustypes=1'>Pending</A></TH>
</TR>
<TR>
<TD CLASS='hostTotals'>0</TD>
<TD CLASS='hostTotals'>0</TD>
<TD CLASS='hostTotals'>0</TD>
<TD CLASS='hostTotals'>0</TD>
</TR>
</TABLE>
</TD></TR><TR><TD ALIGN=CENTER>
<TABLE BORDER=1 CLASS='hostTotals'>
<TR>
<TH CLASS='hostTotals'><A CLASS='hostTotals' HREF='status.cgi?host=(null)&hoststatustypes=12'><I>All Problems</I></A></TH>
<TH CLASS='hostTotals'><A CLASS='hostTotals' HREF='status.cgi?host=(null)'><I>All Types</I></A></TH>
</TR><TR>
<TD CLASS='hostTotals'>0</TD>
<TD CLASS='hostTotals'>0</TD>
</TR>
</TABLE>
</TD></TR>
</TABLE>
</DIV>
</td>
<td align=center valign=top width=33%>
<DIV CLASS='serviceTotals'>Service Status Totals</DIV>
<TABLE BORDER=0 CELLSPACING=0 CELLPADDING=0>
<TR><TD>
<TABLE BORDER=1 CLASS='serviceTotals'>
<TR>
<TH CLASS='serviceTotals'><A CLASS='serviceTotals' HREF='status.cgi?host=(null)&servicestatustypes=2&hoststatustypes=15'>Ok</A></TH>
<TH CLASS='serviceTotals'><A CLASS='serviceTotals' HREF='status.cgi?host=(null)&servicestatustypes=4&hoststatustypes=15'>Warning</A></TH>
<TH CLASS='serviceTotals'><A CLASS='serviceTotals' HREF='status.cgi?host=(null)&servicestatustypes=8&hoststatustypes=15'>Unknown</A></TH>
<TH CLASS='serviceTotals'><A CLASS='serviceTotals' HREF='status.cgi?host=(null)&servicestatustypes=16&hoststatustypes=15'>Critical</A></TH>
<TH CLASS='serviceTotals'><A CLASS='serviceTotals' HREF='status.cgi?host=(null)&servicestatustypes=1&hoststatustypes=15'>Pending</A></TH>
</TR>
<TR>
<TD CLASS='serviceTotals'>0</TD>
<TD CLASS='serviceTotals'>0</TD>
<TD CLASS='serviceTotals'>0</TD>
<TD CLASS='serviceTotals'>0</TD>
<TD CLASS='serviceTotals'>0</TD>
</TR>
</TABLE>
</TD></TR><TR><TD ALIGN=CENTER>
<TABLE BORDER=1 CLASS='serviceTotals'>
<TR>
<TH CLASS='serviceTotals'><A CLASS='serviceTotals' HREF='status.cgi?host=(null)&servicestatustypes=28&hoststatustypes=15'><I>All Problems</I></A></TH>
<TH CLASS='serviceTotals'><A CLASS='serviceTotals' HREF='status.cgi?host=(null)&hoststatustypes=15'><I>All Types</I></A></TH>
</TR><TR>
<TD CLASS='serviceTotals'>0</TD>
<TD CLASS='serviceTotals'>0</TD>
</TR>
</TABLE>
</TD></TR>
</TABLE>
</DIV>
</td>
<td align=right valign=bottom>
</td>
</tr>
</table>
<P>
<table border=0 width=100%>
<tr>
<td valign=top align=left width=33%>
</td><td valign=top align=center width=33%>
<DIV ALIGN=CENTER CLASS='statusTitle'>Service Status Details For All Hosts</DIV>
<br></td>
<td valign=top align=right width=33%></td>
</tr>
</table>
<TABLE BORDER=0 width=100% CLASS='status'>
<TR>
<TH CLASS='status'>Host&nbsp;<A HREF='status.cgi?host=(null)&sorttype=1&sortoption=1'><IMG SRC='/nagios/images/up.gif' BORDER=0 ALT='Sort by host name (ascending)' TITLE='Sort by host name (ascending)'></A><A HREF='status.cgi?host=(null)&sorttype=2&sortoption=1'><IMG SRC='/nagios/images/down.gif' BORDER=0 ALT='Sort by host name (descending)' TITLE='Sort by host name (descending)'></A></TH><TH CLASS='status'>Service&nbsp;<A HREF='status.cgi?host=(null)&sorttype=1&sortoption=2'><IMG SRC='/nagios/images/up.gif' BORDER=0 ALT='Sort by service name (ascending)' TITLE='Sort by service name (ascending)'></A><A HREF='status.cgi?host=(null)&sorttype=2&sortoption=2'><IMG SRC='/nagios/images/down.gif' BORDER=0 ALT='Sort by service name (descending)' TITLE='Sort by service name (descending)'></A></TH><TH CLASS='status'>Status&nbsp;<A HREF='status.cgi?host=(null)&sorttype=1&sortoption=3'><IMG SRC='/nagios/images/up.gif' BORDER=0 ALT='Sort by service status (ascending)' TITLE='Sort by service status (ascending)'></A><A HREF='status.cgi?host=(null)&sorttype=2&sortoption=3'><IMG SRC='/nagios/images/down.gif' BORDER=0 ALT='Sort by service status (descending)' TITLE='Sort by service status (descending)'></A></TH><TH CLASS='status'>Last Check&nbsp;<A HREF='status.cgi?host=(null)&sorttype=1&sortoption=4'><IMG SRC='/nagios/images/up.gif' BORDER=0 ALT='Sort by last check time (ascending)' TITLE='Sort by last check time (ascending)'></A><A HREF='status.cgi?host=(null)&sorttype=2&sortoption=4'><IMG SRC='/nagios/images/down.gif' BORDER=0 ALT='Sort by last check time (descending)' TITLE='Sort by last check time (descending)'></A></TH><TH CLASS='status'>Duration&nbsp;<A HREF='status.cgi?host=(null)&sorttype=1&sortoption=6'><IMG SRC='/nagios/images/up.gif' BORDER=0 ALT='Sort by state duration (ascending)' TITLE='Sort by state duration (ascending)'></A><A HREF='status.cgi?host=(null)&sorttype=2&sortoption=6'><IMG SRC='/nagios/images/down.gif' BORDER=0 ALT='Sort by state duration time (descending)' TITLE='Sort by state duration time (descending)'></A></TH><TH CLASS='status'>Attempt&nbsp;<A HREF='status.cgi?host=(null)&sorttype=1&sortoption=5'><IMG SRC='/nagios/images/up.gif' BORDER=0 ALT='Sort by current attempt (ascending)' TITLE='Sort by current attempt (ascending)'></A><A HREF='status.cgi?host=(null)&sorttype=2&sortoption=5'><IMG SRC='/nagios/images/down.gif' BORDER=0 ALT='Sort by current attempt (descending)' TITLE='Sort by current attempt (descending)'></A></TH><TH CLASS='status'>Status Information</TH>
</TR>
</TABLE>
<P><DIV CLASS='errorMessage'>It appears as though you do not have permission to view information for any of the services you requested...</DIV></P>
<P><DIV CLASS='errorDescription'>If you believe this is an error, check the HTTP server authentication requirements for accessing this CGI<br>and check the authorization options in your CGI configuration file.</DIV></P>

<!-- Produced by Nagios (http://www.nagios.org).  Copyright (c) 1999-2007 Ethan Galstad. -->
</body>
</html>
==23834== 
==23834== ERROR SUMMARY: 4899 errors from 8 contexts (suppressed: 13 from 1)
==23834== malloc/free: in use at exit: 2,856 bytes in 162 blocks.
==23834== malloc/free: 768,529 allocs, 768,367 frees, 21,168,610 bytes allocated.
==23834== For counts of detected errors, rerun with: -v
==23834== searching for pointers to 162 not-freed blocks.
==23834== checked 65,564 bytes.
==23834== 
==23834== LEAK SUMMARY:
==23834==    definitely lost: 262 bytes in 9 blocks.
==23834==      possibly lost: 0 bytes in 0 blocks.
==23834==    still reachable: 2,594 bytes in 153 blocks.
==23834==         suppressed: 0 bytes in 0 blocks.
==23834== Use --leak-check=full to see details of leaked memory.

--------------------- END -----------------------------


Though the CGI is working *for now*, look at all of the 4899 errors. In 48 hours it’s bound to start failing again.

After updating the kernel/rh packages/nagios to the latest stable versions, I’m at my wit’s end here. What should I do next to try to address this problem? To whoever can please assist me with this, I thank you for your help and expertise. 

Regards,

Justin
Comment 1 Ulrich Drepper 2009-04-18 04:34:04 EDT
The error message clearly says that glibc *detected* the problem.  Not that this is a problem in glibc.  Your program has a memory handling error.  Since this is no code which ships with RHEL there is no reason to file the problem here.  Go and talk to the maintainer of that package.

Note You need to log in before you can comment on or make changes to this bug.