Bug 92114 - threaded Apache hangs using loopback
threaded Apache hangs using loopback
Product: Red Hat Linux
Classification: Retired
Component: kernel (Show other bugs)
i686 Linux
medium Severity low
: ---
: ---
Assigned To: Arjan van de Ven
Brian Brock
Depends On:
  Show dependency treegraph
Reported: 2003-06-02 16:04 EDT by Greg Ames
Modified: 2007-04-18 12:54 EDT (History)
0 users

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2004-09-30 11:41:02 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Greg Ames 2003-06-02 16:04:13 EDT
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0.1) Gecko/20021003

Description of problem:
When testing Apache using the worker MPM I get very intermittent hangs using the
loopback interface under high loads.  If I edit /etc/hosts so that all my
virtual hosts use an ethernet interface, the problem disappears.  I never see
the problem using Apache's prefork MPM (not threaded), nor do I see it with the
worker MPM using less than 4 threads per process.

Apache's server-status shows the hung thread is in "R" state, i.e, trying to
read the http request line.  Apache times out the hung condition after 5
minutes, then issues close(), then the client reports:

for fd 37 (after reading 0 bytes): read: Connection reset by peer

So both sides of the connection were trying to read!  gdb shows that the hung
Apache worker thread is stuck in a poll() syscall when there are bytes available
to be read according to netstat -at:

(gdb) bt
#0  0x420db1a7 in poll () from /lib/i686/libc.so.6
#1  0x4005d084 in apr_poll ()
   from /home/gregames/apache/2.0.46/built/lib/libapr-0.so.0
#2  0x4005d496 in apr_wait_for_io_or_timeout ()
   from /home/gregames/apache/2.0.46/built/lib/libapr-0.so.0
#3  0x4005470b in uapr_socket_recv ()
   from /home/gregames/apache/2.0.46/built/lib/libapr-0.so.0
#4  0x40053d11 in apr_socket_recv ()
   from /home/gregames/apache/2.0.46/built/lib/libapr-0.so.0
#5  0x40019b04 in socket_bucket_read ()
   from /home/gregames/apache/2.0.46/built/lib/libaprutil-0.so.0
#6  0x4001a2ba in apr_brigade_split_line ()
   from /home/gregames/apache/2.0.46/built/lib/libaprutil-0.so.0
#7  0x080788b3 in core_input_filter ()
#8  0x08072022 in ap_get_brigade ()
#9  0x08072022 in ap_get_brigade ()
#10 0x08072e28 in ap_rgetline_core ()
#11 0x080732e2 in read_request_line ()

netstat -st shows that TCPAbortOnClose is incremented after Apache times out and
closes the connection.  kernel source says that this happens because there is
unread data.

It seems like there might be a race condition between the loopback driver
sending data from the client's process and the poll() for readability from the
Apache worker thread, where new data arrives just after tcp_poll checks for it,
but before the poll sleeps.  Looking at the kernel source, I can't see how this
is serialized/locked, but I'm a kernel newbie.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Run Apache with the worker MPM and ThreadsPerChild set to 8 
2. (this part is hard) On the same machine, run a client that simulates a
production web site's workload with a mixture of tiny, medium, and huge files
and CGIs.  SPECWeb99 might work (not confirmed).  I'm using a custom client. 
3.  Make sure that all http traffic flows thru the loopback interface.


Expected Results:  There shouldn't be race conditions between poll() for
readability and the loopback driver.

Additional info:
Comment 1 Arjan van de Ven 2003-06-02 16:10:19 EDT
could you try the current erratum kernel for RHL8?
At least it doesn't have remote exploits etc etc and has lots of bugfixes...
Comment 2 Greg Ames 2003-06-04 10:18:43 EDT
OK, I'm running kernel 2.4.20-13.8 now.  The bug is either gone or a lot more
elusive now, but I think I hit it once or twice yesterday.  The external symtoms
looked the same anyway.  Today I haven't been able to hit it at all and collect
the netstats & backtrace etc to verify it's the same thing.

It looks like file caching is working better in this kernel.  Almost all of my
files are served out of the cache now after the first run, but they weren't with
2.4.18.  I believe this is decreasing the interrupt rate and might make this bug
harder to catch, whatever it is.  I've been running grep -r thru /usr to add
some interrupts.

If I figure out how to recreate this more reliably, I will report back.

Comment 3 Bugzilla owner 2004-09-30 11:41:02 EDT
Thanks for the bug report. However, Red Hat no longer maintains this version of
the product. Please upgrade to the latest version and open a new bug if the problem

The Fedora Legacy project (http://fedoralegacy.org/) maintains some older releases, 
and if you believe this bug is interesting to them, please report the problem in
the bug tracker at: http://bugzilla.fedora.us/

Note You need to log in before you can comment on or make changes to this bug.