Description of problem: Env: RHES-3AS Linux viper 2.4.21-15.EL #1 Thu Apr 22 00:27:41 EDT 2004 i686 i686 i386 GNU/Linux http: httpd-2.0.46-32.ent Running /usr/sbin/httpd -V will produced the following: Server version: Apache/2.0.46 Server built: May 25 2004 05:32:19 Server's Module Magic Number: 20020903:4 Architecture: 32-bit Server compiled with.... -D APACHE_MPM_DIR="server/mpm/prefork" -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses disabled) -D APR_USE_SYSVSEM_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D HTTPD_ROOT="/etc/httpd" -D SUEXEC_BIN="/usr/sbin/suexec" -D DEFAULT_PIDLOG="logs/httpd.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_LOCKFILE="logs/accept.lock" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="conf/mime.types" -D SERVER_CONFIG_FILE="conf/httpd.conf" Running /usr/sbin/httpd -t will produced: Syntax OK running service httpd start will always produced: Starting httpd: [FAILED] Running /usr/sbin/httpd Segmentation fault Version-Release number of selected component (if applicable): httpd-2.0.46-32.ent How reproducible: Every time Steps to Reproduce: 1.service httpd start 2./usr/sbin/httpd 3. Actual results: Segmentation fault Expected results: running process serving requests on ports 80 and 443 Additional info: Running gdb /usr/sbin/httpd GNU gdb Red Hat Linux (6.0post- 0.20040223.17rh) Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db lib rary "/lib/tls/libthread_db.so.1". (gdb) run Starting program: /usr/sbin/httpd [Thread debugging using libthread_db enabled] [New Thread -1223441760 (LWP 23602)] Program received signal SIGSEGV, Segmentation fault. [Switching to Thread -1223441760 (LWP 23602)] apr_os_proc_mutex_get (ospmutex=0xbfffcc08, pmutex=0x8212090) at proc_mutex.c:865 865 ospmutex->crossproc = pmutex->interproc- >filedes; Any idea on how to resolve this problem will be greatly appreciated
Thanks for the report. Can you attach the httpd.conf and any modified /etc/httpd/conf.d/*.conf files in use? Are any third-party modules being loaded? If you can attach the complete backtrace from gdb that would be most useful.
Created attachment 100540 [details] http config file As requested, httpd.conf file.
Thanks for the quick reply. I am using right now the unmodified /etc/httpd/conf/ttpd.conf and the Include conf.d/*.conf statement is commented. The files are standards unmodified and came with the httpd-2.0.46-32.ent rpm files (httpd, httpd-devel and mod- ssl). I attached the httpd.conf to this record. For the gdb, I also load the httpd-debug for the symbols. the bt is below: 865 ospmutex->crossproc = pmutex->interproc- >filedes; (gdb) bt #0 apr_os_proc_mutex_get (ospmutex=0xbfffa448, pmutex=0x8134228) at proc_mutex.c:865 #1 0x08083267 in unixd_set_proc_mutex_perms (pmutex=0x0) at /usr/src/debug/httpd- 2.0.46/os/unix/unixd.c:439 #2 0x080832d3 in unixd_set_global_mutex_perms (gmutex=0x0) at /usr/src/debug/httpd- 2.0.46/os/unix/unixd.c:457 #3 0xb70adfde in post_config (p=0x80980a8, plog=0x80c2150, ptemp=0x80c4158, s=0x80b2260) at /usr/src/debug/httpd- 2.0.46/modules/mappers/mod_rewrite.c:1021 #4 0x080680da in ap_run_post_config (pconf=0x80980a8, plog=0x80c2150, ptemp=0x80c4158, s=0x80b2260) at /usr/src/debug/httpd- 2.0.46/server/config.c:132 #5 0x0806d8da in main (argc=1, argv=0xbfffa624) at /usr/src/debug/httpd- 2.0.46/server/main.c:608 (gdb) Please let me know if you need more info.
I notice from your httpd -V output that this is a custom-built RPM. If using the Red Hat-supplied httpd and mod_ssl etc packages, is the behaviour still reproducible? "Server built: May 25 2004 05:32:19" vs # rpm -q httpd httpd-2.0.46-32.ent # httpd -V Server version: Apache/2.0.46 Server built: Mar 1 2004 12:14:32
not really a custom build, I obtain the rpm from the ftp updates directory and issue an rpmbuild -bb httpd.spec Silly me and have now the CD, so just did rpm -Uvh --replacefiles -- replacepkgs httpd httpd-devel and mod_ssl so now: /usr/sbin/httpd -V Server version: Apache/2.0.46 Server built: Mar 1 2004 12:14:32 Server's Module Magic Number: 20020903:4 Architecture: 32-bit Server compiled with.... -D APACHE_MPM_DIR="server/mpm/prefork" -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses disabled) -D APR_USE_SYSVSEM_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D HTTPD_ROOT="/etc/httpd" -D SUEXEC_BIN="/usr/sbin/suexec" -D DEFAULT_PIDLOG="logs/httpd.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_LOCKFILE="logs/accept.lock" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="conf/mime.types" -D SERVER_CONFIG_FILE="conf/httpd.conf" /usr/sbin/httpd -t Syntax OK [root@viper conf]# gdb /usr/sbin/httpd GNU gdb Red Hat Linux (6.0post- 0.20040223.17rh) Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu"...(no debugging symbols found )...Using host libthread_db library "/lib/tls/libthread_db.so.1". (gdb) run Starting program: /usr/sbin/httpd (no debugging symbols found)...(no debugging symbols found)...(no debugging symb ols found)...(no debugging symbols found)...(no debugging symbols found)...(no d ebugging symbols found)...(no debugging symbols found)...(no debugging symbols f ound)...(no debugging symbols found)...(no debugging symbols found)... (no debugg ing symbols found)...[Thread debugging using libthread_db enabled] [New Thread -1223409536 (LWP 10340)] Program received signal SIGSEGV, Segmentation fault. [Switching to Thread -1223409536 (LWP 10340)] apr_os_proc_mutex_get (ospmutex=0xbfffc0e8, pmutex=0x8134230) at proc_mutex.c:865 865 ospmutex->crossproc = pmutex->interproc- >filedes; (gdb) bt #0 apr_os_proc_mutex_get (ospmutex=0xbfffc0e8, pmutex=0x8134230) at proc_mutex.c:865 #1 0x08083257 in unixd_set_proc_mutex_perms () #2 0x080832c3 in unixd_set_global_mutex_perms () #3 0xb70b5fde in ?? () from /etc/httpd/modules/mod_rewrite.so #4 0x08134220 in ?? () #5 0x00000000 in ?? () (gdb) so, we still at the same point with the original rpms from RH-EL AS 3.0 Disk 3. Thanks again.
OK, thanks. Since we can't reproduce the problem here, can you do: # ulimit -c unlimited # /usr/sbin/httpd -X ... and attach the core dump produced to this bug report? Also, attaching the output of: # strace -o /tmp/httpd.trace /usr/sbin/httpd may be useful.
Created attachment 100543 [details] core dump (/usr/sbin/httpd -X) Sorry, greater than 1000Kb, so gzip it.
Created attachment 100544 [details] strace -o /tmp/httpd.trace /usr/sbin/httpd
Again, can you eliminate non-standard library builds from the issue: the strace shows that e.g. LD_LIBRARY_PATH is forcing a different libdb-4.1.so to be used: open("/usr/local/BerkeleyDB.4.1/lib/libdb-4.1.so", O_RDONLY) = 3 try, e.g. # env -i /usr/sbin/httpd
Same with environment. # env -i /usr/sbin/httpd Segmentation fault (core dumped) You have new mail in /var/spool/mail/root # whereis libdb-4.1.so libdb-4.1: /lib/libdb-4.1.so /lib/libdb-4.1.a /usr/lib/libdb- 4.1.so /usr/lib/lib db-4.1.a /usr/lib/libdb-4.1.la I also removed usr/local/BerkeleyDB.4.1/lib from ld.so.conf and then ldconfig and issue the same test. Same result: Segmentation fault
The core dump looks like random memory corruption; since we can't reproduce the problem here, we need to work out what is different about your environment. Did the problem manifest from a fresh RHEL3/AS U2 install, or else, when did it start occuring? Can you try: # rpm -V httpd mod_ssl glibc to verify that the packages have not become corrupt.
ok, this is what I got: # rpm -V httpd mod_ssl glibc S.5....T c /etc/httpd/conf/httpd.conf S.5....T c /etc/rc.d/init.d/httpd ....L... /usr/lib/libapr-0.so.0 Not really familar with the output and what all of this means. However I understand that the output shows something that is not correct but what?
This: ....L... /usr/lib/libapr-0.so.0 means that the libapr-0.so.0 symlink has been changed. It should point to libapr-0.so.0.9.4. # ls -lo /usr/lib/libapr-0.so* lrwxr-xr-x 1 root 17 May 12 12:51 /usr/lib/libapr-0.so -> libapr-0.so.0.9.4 lrwxr-xr-x 1 root 17 May 12 12:51 /usr/lib/libapr-0.so.0 -> libapr-0.so.0.9.4 -rwxr-xr-x 1 root 125756 Mar 1 17:20 /usr/lib/libapr-0.so.0.9.4
Thank you, just found the problem. ]# ls -lo /usr/lib/libapr-0.so* lrwxrwxrwx 1 root 17 May 25 10:08 /usr/lib/libapr-0.so - > libapr-0. so.0.9.4 lrwxrwxrwx 1 root 17 May 25 10:08 /usr/lib/libapr- 0.so.0 -> libapr- 0.so.0.9.5 -rwxr-xr-x 1 root 125756 Mar 1 11:20 /usr/lib/libapr- 0.so.0.9.4 -rwxr-xr-x 1 root 381120 May 1 16:58 /usr/lib/libapr- 0.so.0.9.5 # rpm -q --whatprovides /usr/lib/libapr-0.so.0.9.5 file /usr/lib/libapr-0.so.0.9.5 is not owned by any package # rpm -q --whatprovides /usr/lib/libapr-0.so.0.9.4 httpd-2.0.46- 32.ent [root@viper xitiami]# rm -f /usr/lib/libapr- 0.so.0.9.5 [root@viper xitiami]# ldconfig /usr/lib [root@viper xitiami]# ls -lo /usr/lib/libapr- 0.so* lrwxrwxrwx 1 root 17 May 25 10:08 /usr/lib/libapr-0.so - > libapr-0. so.0.9.4 lrwxrwxrwx 1 root 17 May 26 05:59 /usr/lib/libapr- 0.so.0 -> libapr- 0.so.0.9.4 -rwxr-xr-x 1 root 125756 Mar 1 11:20 /usr/lib/libapr- 0.so.0.9.4 # service httpd start Starting httpd: [ OK ] # service httpd status httpd (pid 9109 9108 9107 9106 9105 9104 9103 9102 9099) is running.. # Thank you so much, back in business. Very good throubleshooting. I change the status to Worksforme since it is resolved. Have a wonderful day.