Bug 98979 - Apache 2.0.40 hangs on heavy load
Summary: Apache 2.0.40 hangs on heavy load
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Linux
Classification: Retired
Component: httpd
Version: 9
Hardware: i686
OS: Linux
high
high
Target Milestone: ---
Assignee: Joe Orton
QA Contact: Brian Brock
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2003-07-11 09:57 UTC by Syed Buhari
Modified: 2005-10-31 22:00 UTC (History)
0 users

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2003-07-15 07:33:15 UTC
Embargoed:


Attachments (Terms of Use)
httpd config file (12.34 KB, text/plain)
2003-07-14 14:28 UTC, Syed Buhari
no flags Details
latest http config file (34.30 KB, text/plain)
2003-07-14 15:10 UTC, Syed Buhari
no flags Details

Description Syed Buhari 2003-07-11 09:57:27 UTC
Description of problem:
I have tried Apache 2.0.40 on Redhat 9.0 (installed by rpm package from 
redhat). Apache web server hangs, when i hit the web server multiple times 
simultaneously. This behaviour does not happen with source verison (i have 
downloaded source code of 2.0.40 and build on RH 9.0).

Version-Release number of selected component (if applicable):
Redhat 9.0, Apache httpd 2.0.45

How reproducible:


Steps to Reproduce:
1.
2.
3.
    
Actual results:


Expected results:


Additional info:
I have Apache DSO module which connects to remote application and returning 
html page. I am getting html page when single request is submitted and web 
server hangs when i simultaneously hit web server with different request. This 
does not occurrs with source version of apache httpd 2.0.40 ( downloaded from 
apache.org). Same appplication working fine in RedHat 8.0 (both binary and 
source installation).

Comment 1 Joe Orton 2003-07-11 10:06:20 UTC
Did you recompile your module on Red Hat Linux 9?  This report is too vague to
be able to determine that the problem isn't in your custom module.

Try running "strace" on an httpd child and see where the hang occurs.

Comment 2 Syed Buhari 2003-07-11 11:46:02 UTC
Yes, I have recompiled my DSO module on RH 9.0. Moreover, I have tried with cgi 
module (that does same operation as our DSO module) and it's working without 
any issues. I am getting this problem only through DSO module.

Comment 3 Joe Orton 2003-07-11 13:41:44 UTC
Since the hang is in your custom module you need to work out what is causing the
hang.   Once you have reproduced the hang, find an httpd child which is hung
(via "ps axf", for instance), and run

  # strace -p 12354

on it to see where it is blocked.  Enabling the server-status page may help. 
Please attach your httpd.conf too.

Comment 4 Syed Buhari 2003-07-14 10:02:32 UTC
I could not able to trace the child process. It says

trace: ptrace(PTRACE_SYSCALL, ...): Operation not permitted
detach: ptrace(PTRACE_DETACH, ...): Operation not permitted

any other way to trace it.

Comment 5 Joe Orton 2003-07-14 10:04:09 UTC
You need to get the latest kernel from up2date and reboot into it to fix
strace'ing httpd children, unfortunately.

Comment 6 Syed Buhari 2003-07-14 14:28:07 UTC
Created attachment 92912 [details]
httpd config file

Comment 7 Joe Orton 2003-07-14 14:32:38 UTC
Can you confirm that the attached httpd.conf hasn't been truncated, the last two
lines are:

# 
# Disable autoindex for the root direc

Comment 8 Syed Buhari 2003-07-14 15:10:52 UTC
Created attachment 92913 [details]
latest http config file

I have attached latest httpd config file.

Comment 9 Syed Buhari 2003-07-14 15:11:36 UTC
I have upgraded the kernel to 2.4.20-18.9. I have applied strace to child 
process which hangs and got the following out put.

futex(0x405ac520, FUTEX_WAIT, 5, NULL

I have noticed that, child process is hanged in the following scenario

I have dso module which process the incoming request. I am hitting this module  
and stopped intermediately (I am forcing to stop the request by pressing stop 
button in the web browser) before request is fully processed. Child process is 
hanged when i tried to access same module again.

Comment 10 Joe Orton 2003-07-14 15:13:37 UTC
I presume you have not configured use of the httpd.worker threaded MPM?  This is
a hang when calling pthread_mutex_lock.  Does your custom module use threads or
mutexes?

Comment 11 Syed Buhari 2003-07-14 15:31:04 UTC
My web server MPM is configured to use pre-fork not worker. I am using the web 
server which installed along with RH 9.0 and it is configured to use pre-fork 
by default. 
I am using same configuration in RH 8.0 and it is working with out any issues. 
Also I have installed apache source and built a binaries on RH 9.0. This source 
version of apache is configured with pre-fork model and it is working with out 
any issues.
Is the apache binaries shipped with RH 9.0 is compiled with new pthread (NPTL) 
in RH 9.0 ?

Comment 12 Joe Orton 2003-07-14 15:35:59 UTC
Yes, the default threading library on RHL9 is NPTL. You can try starting httpd
using:

# service httpd stop # wait for shutdown
# LD_ASSUME_KERNEL=2.2.9 /usr/sbin/httpd

to use the old thread library. Does this make a difference?  Can you confirm
that your custom module does use threads and mutexes?


Comment 13 Syed Buhari 2003-07-15 05:36:16 UTC
Yes, my custom module uses threads and mutexes. I have tried with 

# LD_ASSUME_KERNEL=2.2.9 /usr/sbin/httpd

Now, child process does not hang and everything works as expected.


Comment 14 Syed Buhari 2003-07-15 07:27:07 UTC
Regarding source version of Apache 2.0.40 

Source version of apache (MDM is prefork) also hangs. I have configured apache 
source MDM to use worker and still the web server is hangs. I have given the 
strace output for web server.

8933 ?        S      0:00 /usr/local/apache2/bin/httpd -k start
8934 ?        S      0:00  \_ /usr/local/apache2/bin/httpd -k start
8935 ?        S      0:00  \_ /usr/local/apache2/bin/httpd -k start
8937 ?        S      0:00  \_ /usr/local/apache2/bin/httpd -k start

strace -p 8934
accept(8,

strace -p 8935
read(5,

strace -p 8937
read(5,

This is happens only when the request is stopped intermediately. I couldn't 
able to run using LD_ASSUME_KERNEL with my source version of apache (httpd does 
not start and no error is returned).

Binary apache is working with out any issues with 
#LD_ASSUME_KERNEL=2.2.9 /usr/sbin/httpd






Comment 15 Joe Orton 2003-07-15 07:33:15 UTC
If that's worker then you need to use "ps axmf" to see all the threads.  Syed,
if your module is hanging in versions of httpd compiled from source too, it's
hard to say that this is a bug in our httpd package - it's more likely that this
is some problem in the thread or mutex handling in your module which is exposed
by NPTL.  I suggest you review the source of your module carefully looking for
deadlocks etc - I'm going to close this bug.

Comment 16 Syed Buhari 2003-07-15 08:48:21 UTC
Any idea, How this is working in RH 8.0 and working in RH 9.0 with 
LD_ASSUME_KERNEL=2.2.9 ?

Comment 17 Joe Orton 2003-07-15 08:52:08 UTC
On 9 if you use LD_ASSUME_KERNEL, it uses the same threading library as 8.0.  On
9, if you don't use LD_ASSUME_KERNEL, it uses NPTL.


Note You need to log in before you can comment on or make changes to this bug.