Bug 124268 - Starting httpd results in Segmentation fault
Starting httpd results in Segmentation fault
Status: CLOSED WORKSFORME
Product: Red Hat Enterprise Linux 3
Classification: Red Hat
Component: httpd (Show other bugs)
3.0
i686 Linux
medium Severity medium
: ---
: ---
Assigned To: Joe Orton
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2004-05-25 07:43 EDT by Jean-Philippe Cassar
Modified: 2007-11-30 17:07 EST (History)
0 users

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2004-05-26 07:10:50 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
http config file (34.34 KB, text/plain)
2004-05-25 10:40 EDT, Jean-Philippe Cassar
no flags Details
core dump (/usr/sbin/httpd -X) (174.94 KB, text/plain)
2004-05-25 11:58 EDT, Jean-Philippe Cassar
no flags Details
strace -o /tmp/httpd.trace /usr/sbin/httpd (35.24 KB, text/plain)
2004-05-25 11:58 EDT, Jean-Philippe Cassar
no flags Details

  None (edit)
Description Jean-Philippe Cassar 2004-05-25 07:43:30 EDT
Description of problem:
Env: RHES-3AS Linux viper 2.4.21-15.EL #1 Thu Apr 22 00:27:41 EDT 
2004 i686 i686 i386 GNU/Linux
http: httpd-2.0.46-32.ent

Running /usr/sbin/httpd -V will produced the following:
Server version: Apache/2.0.46                      
Server built:   May 25 2004 05:32:19               
Server's Module Magic Number: 20020903:4           
Architecture:   32-bit                             
Server compiled with....                           
 -D APACHE_MPM_DIR="server/mpm/prefork"            
 -D APR_HAS_SENDFILE                               
 -D APR_HAS_MMAP                                   
 -D APR_HAVE_IPV6 (IPv4-mapped addresses disabled) 
 -D APR_USE_SYSVSEM_SERIALIZE                      
 -D APR_USE_PTHREAD_SERIALIZE                      
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT              
 -D APR_HAS_OTHER_CHILD                            
 -D AP_HAVE_RELIABLE_PIPED_LOGS                    
 -D HTTPD_ROOT="/etc/httpd"                        
 -D SUEXEC_BIN="/usr/sbin/suexec"                  
 -D DEFAULT_PIDLOG="logs/httpd.pid"                
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="logs/accept.lock"            
 -D DEFAULT_ERRORLOG="logs/error_log"              
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"         
 -D SERVER_CONFIG_FILE="conf/httpd.conf"           

Running /usr/sbin/httpd -t will produced:
Syntax OK

running service httpd start will always produced:
Starting httpd:                                            [FAILED]

Running /usr/sbin/httpd
Segmentation fault


Version-Release number of selected component (if applicable):
httpd-2.0.46-32.ent

How reproducible:
Every time

Steps to Reproduce:
1.service httpd start
2./usr/sbin/httpd
3.
  
Actual results:
Segmentation fault

Expected results:
running process serving requests on ports 80 and 443

Additional info:
Running gdb /usr/sbin/httpd
GNU gdb Red Hat Linux (6.0post-
0.20040223.17rh)                                 
Copyright 2004 Free Software Foundation, 
Inc.                                   
GDB is free software, covered by the GNU General Public License, and 
you are    
welcome to change it and/or distribute copies of it under certain 
conditions.   
Type "show copying" to see the 
conditions.                                      
There is absolutely no warranty for GDB.  Type "show warranty" for 
details.     
This GDB was configured as "i386-redhat-linux-gnu"...Using host 
libthread_db lib
rary "/lib/tls/libthread_db.so.1".                                    
          
                                                                      
          
(gdb) 
run                                                                   
    
Starting 
program: /usr/sbin/httpd                                              
 
[Thread debugging using libthread_db 
enabled]                                   
[New Thread -1223441760 (LWP 
23602)]                                            
                                                                      
          
Program received signal SIGSEGV, Segmentation 
fault.                            
[Switching to Thread -1223441760 (LWP 
23602)]                                   
apr_os_proc_mutex_get (ospmutex=0xbfffcc08, 
pmutex=0x8212090)                   
    at 
proc_mutex.c:865                                                      
   
865         ospmutex->crossproc = pmutex->interproc-
>filedes;                   



Any idea on how to resolve this problem will be greatly appreciated
Comment 1 Joe Orton 2004-05-25 07:49:23 EDT
Thanks for the report.  Can you attach the httpd.conf and any modified
/etc/httpd/conf.d/*.conf files in use?  Are any third-party modules
being loaded?

If you can attach the complete backtrace from gdb that would be most
useful.
Comment 2 Jean-Philippe Cassar 2004-05-25 10:40:43 EDT
Created attachment 100540 [details]
http config file

As requested, httpd.conf file.
Comment 3 Jean-Philippe Cassar 2004-05-25 10:41:24 EDT
Thanks for the quick reply. I am using right now the 
unmodified /etc/httpd/conf/ttpd.conf and the Include conf.d/*.conf 
statement is commented. The files are standards unmodified and came 
with the httpd-2.0.46-32.ent rpm files (httpd, httpd-devel and mod-
ssl). I attached the httpd.conf to this record.
For the gdb, I also load the httpd-debug for the symbols. the bt is 
below:

865         ospmutex->crossproc = pmutex->interproc-
>filedes;               
(gdb) 
bt                                                                    
#0  apr_os_proc_mutex_get (ospmutex=0xbfffa448, 
pmutex=0x8134228)           
    at 
proc_mutex.c:865                                                     
#1  0x08083267 in unixd_set_proc_mutex_perms 
(pmutex=0x0)                   
    at /usr/src/debug/httpd-
2.0.46/os/unix/unixd.c:439                      
#2  0x080832d3 in unixd_set_global_mutex_perms 
(gmutex=0x0)                 
    at /usr/src/debug/httpd-
2.0.46/os/unix/unixd.c:457                      
#3  0xb70adfde in post_config (p=0x80980a8, plog=0x80c2150, 
ptemp=0x80c4158,
    
s=0x80b2260)                                                          
  
    at /usr/src/debug/httpd-
2.0.46/modules/mappers/mod_rewrite.c:1021       
#4  0x080680da in ap_run_post_config (pconf=0x80980a8, 
plog=0x80c2150,      
    ptemp=0x80c4158, 
s=0x80b2260)                                           
    at /usr/src/debug/httpd-
2.0.46/server/config.c:132                      
#5  0x0806d8da in main (argc=1, 
argv=0xbfffa624)                            
    at /usr/src/debug/httpd-
2.0.46/server/main.c:608                        
(gdb)

Please let me know if you need more info.
Comment 4 Joe Orton 2004-05-25 10:57:24 EDT
I notice from your httpd -V output that this is a custom-built RPM.  
If using the Red Hat-supplied httpd and mod_ssl etc packages, is the
behaviour still reproducible?

"Server built:   May 25 2004 05:32:19"

vs

# rpm -q httpd
httpd-2.0.46-32.ent
# httpd -V
Server version: Apache/2.0.46
Server built:   Mar  1 2004 12:14:32
Comment 5 Jean-Philippe Cassar 2004-05-25 11:19:30 EDT
not really a custom build, I obtain the rpm from the ftp updates 
directory and issue an rpmbuild -bb httpd.spec 
Silly me and have now the CD, so just did rpm -Uvh --replacefiles --
replacepkgs httpd httpd-devel and mod_ssl
so now:
/usr/sbin/httpd -V              
Server version: Apache/2.0.46                      
Server built:   Mar  1 2004 12:14:32               
Server's Module Magic Number: 20020903:4           
Architecture:   32-bit                             
Server compiled with....                           
 -D APACHE_MPM_DIR="server/mpm/prefork"            
 -D APR_HAS_SENDFILE                               
 -D APR_HAS_MMAP                                   
 -D APR_HAVE_IPV6 (IPv4-mapped addresses disabled) 
 -D APR_USE_SYSVSEM_SERIALIZE                      
 -D APR_USE_PTHREAD_SERIALIZE                      
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT              
 -D APR_HAS_OTHER_CHILD                            
 -D AP_HAVE_RELIABLE_PIPED_LOGS                    
 -D HTTPD_ROOT="/etc/httpd"                        
 -D SUEXEC_BIN="/usr/sbin/suexec"                  
 -D DEFAULT_PIDLOG="logs/httpd.pid"                
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="logs/accept.lock"            
 -D DEFAULT_ERRORLOG="logs/error_log"              
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"         
 -D SERVER_CONFIG_FILE="conf/httpd.conf"           

/usr/sbin/httpd -t                                           
Syntax OK

[root@viper conf]# 
gdb /usr/sbin/httpd                                          
GNU gdb Red Hat Linux (6.0post-
0.20040223.17rh)                                 
Copyright 2004 Free Software Foundation, 
Inc.                                   
GDB is free software, covered by the GNU General Public License, and 
you are    
welcome to change it and/or distribute copies of it under certain 
conditions.   
Type "show copying" to see the 
conditions.                                      
There is absolutely no warranty for GDB.  Type "show warranty" for 
details.     
This GDB was configured as "i386-redhat-linux-gnu"...(no debugging 
symbols found
)...Using host libthread_db 
library "/lib/tls/libthread_db.so.1".               
                                                                      
          
(gdb) 
run                                                                   
    
Starting 
program: /usr/sbin/httpd                                              
 
(no debugging symbols found)...(no debugging symbols found)...(no 
debugging symb
ols found)...(no debugging symbols found)...(no debugging symbols 
found)...(no d
ebugging symbols found)...(no debugging symbols found)...(no 
debugging symbols f
ound)...(no debugging symbols found)...(no debugging symbols found)...
(no debugg
ing symbols found)...[Thread debugging using libthread_db 
enabled]              
[New Thread -1223409536 (LWP 
10340)]                                            
                                                                      
          
Program received signal SIGSEGV, Segmentation 
fault.                            
[Switching to Thread -1223409536 (LWP 
10340)]                                   
apr_os_proc_mutex_get (ospmutex=0xbfffc0e8, 
pmutex=0x8134230)                   
    at 
proc_mutex.c:865                                                      
   
865         ospmutex->crossproc = pmutex->interproc-
>filedes;                   
(gdb) bt                                                         
#0  apr_os_proc_mutex_get (ospmutex=0xbfffc0e8, pmutex=0x8134230)
    at proc_mutex.c:865                                          
#1  0x08083257 in unixd_set_proc_mutex_perms ()                  
#2  0x080832c3 in unixd_set_global_mutex_perms ()                
#3  0xb70b5fde in ?? () from /etc/httpd/modules/mod_rewrite.so   
#4  0x08134220 in ?? ()                                          
#5  0x00000000 in ?? ()                                          
(gdb)                                                            

so, we still at the same point with the original rpms from RH-EL AS 
3.0 Disk 3.

Thanks again.
Comment 6 Joe Orton 2004-05-25 11:40:51 EDT
OK, thanks. Since we can't reproduce the problem here, can you do:

# ulimit -c unlimited
# /usr/sbin/httpd -X
...

and attach the core dump produced to this bug report? Also, attaching
the output of:

# strace -o /tmp/httpd.trace /usr/sbin/httpd

may be useful.
Comment 7 Jean-Philippe Cassar 2004-05-25 11:58:06 EDT
Created attachment 100543 [details]
core dump (/usr/sbin/httpd -X)

Sorry, greater than 1000Kb, so gzip it.
Comment 8 Jean-Philippe Cassar 2004-05-25 11:58:36 EDT
Created attachment 100544 [details]
strace -o /tmp/httpd.trace /usr/sbin/httpd
Comment 9 Joe Orton 2004-05-25 12:09:24 EDT
Again, can you eliminate non-standard library builds from the issue:
the strace shows that e.g. LD_LIBRARY_PATH is forcing a different
libdb-4.1.so to be used:

open("/usr/local/BerkeleyDB.4.1/lib/libdb-4.1.so", O_RDONLY) = 3

try, e.g.

# env -i /usr/sbin/httpd
Comment 10 Jean-Philippe Cassar 2004-05-25 12:35:29 EDT
Same with environment.
# env -i /usr/sbin/httpd                                       
Segmentation fault (core 
dumped)                                                
You have new mail 
in /var/spool/mail/root                                       
# whereis libdb-4.1.so                                         
libdb-4.1: /lib/libdb-4.1.so /lib/libdb-4.1.a /usr/lib/libdb-
4.1.so /usr/lib/lib
db-4.1.a /usr/lib/libdb-4.1.la
I also removed usr/local/BerkeleyDB.4.1/lib from ld.so.conf and then 
ldconfig and issue the same test. Same result: Segmentation fault 
Comment 11 Joe Orton 2004-05-26 05:21:17 EDT
The core dump looks like random memory corruption; since we can't
reproduce the problem here, we need to work out what is different
about your environment.  Did the problem manifest from a fresh
RHEL3/AS U2 install, or else, when did it start occuring?

Can you try:

# rpm -V httpd mod_ssl glibc

to verify that the packages have not become corrupt.
Comment 12 Jean-Philippe Cassar 2004-05-26 06:42:33 EDT
ok, this is what I got:
# rpm -V httpd mod_ssl glibc
S.5....T c /etc/httpd/conf/httpd.conf        
S.5....T c /etc/rc.d/init.d/httpd            
....L...   /usr/lib/libapr-0.so.0            

Not really familar with the output and what all of this means. 
However I understand that the output shows something that is not 
correct but what?
Comment 13 Joe Orton 2004-05-26 06:50:50 EDT
This:

....L...   /usr/lib/libapr-0.so.0

means that the libapr-0.so.0 symlink has been changed.  It should
point to libapr-0.so.0.9.4.

# ls -lo /usr/lib/libapr-0.so*
lrwxr-xr-x    1 root           17 May 12 12:51 /usr/lib/libapr-0.so ->
libapr-0.so.0.9.4
lrwxr-xr-x    1 root           17 May 12 12:51 /usr/lib/libapr-0.so.0
-> libapr-0.so.0.9.4
-rwxr-xr-x    1 root       125756 Mar  1 17:20 /usr/lib/libapr-0.so.0.9.4

Comment 14 Jean-Philippe Cassar 2004-05-26 07:10:50 EDT
Thank you,
just found the problem.
]# ls -lo /usr/lib/libapr-0.so*                              
lrwxrwxrwx    1 root           17 May 25 10:08 /usr/lib/libapr-0.so -
> libapr-0.
so.0.9.4                                                              
          
lrwxrwxrwx    1 root           17 May 25 10:08 /usr/lib/libapr-
0.so.0 -> libapr-
0.so.0.9.5                                                            
          
-rwxr-xr-x    1 root       125756 Mar  1 11:20 /usr/lib/libapr-
0.so.0.9.4       
-rwxr-xr-x    1 root       381120 May  1 16:58 /usr/lib/libapr-
0.so.0.9.5       
# rpm -q --whatprovides /usr/lib/libapr-0.so.0.9.5          
file /usr/lib/libapr-0.so.0.9.5 is not owned by any 
package                     
# rpm -q --whatprovides /usr/lib/libapr-0.so.0.9.4          
httpd-2.0.46-
32.ent                                                             
[root@viper xitiami]# rm -f /usr/lib/libapr-
0.so.0.9.5                          
[root@viper xitiami]# 
ldconfig /usr/lib                                         
[root@viper xitiami]# ls -lo /usr/lib/libapr-
0.so*                              
lrwxrwxrwx    1 root           17 May 25 10:08 /usr/lib/libapr-0.so -
> libapr-0.
so.0.9.4                                                              
          
lrwxrwxrwx    1 root           17 May 26 05:59 /usr/lib/libapr-
0.so.0 -> libapr-
0.so.0.9.4                                                            
          
-rwxr-xr-x    1 root       125756 Mar  1 11:20 /usr/lib/libapr-
0.so.0.9.4

# service httpd start                             
Starting httpd:                                            [  OK  ]
# service httpd status                              
httpd (pid 9109 9108 9107 9106 9105 9104 9103 9102 9099) is running..
#                                                   

Thank you so much, back in business. Very good throubleshooting.
I change the status to Worksforme since it is resolved.
Have a wonderful day.

Note You need to log in before you can comment on or make changes to this bug.