Bug 90918

Summary: gdb segfaults when httpd.worker breakpoint reached
Product: [Retired] Red Hat Linux Reporter: Jeff Trawick <trawick>
Component: gdbAssignee: Elena Zannoni <ezannoni>
Status: CLOSED CANTFIX QA Contact: Jay Turner <jturner>
Severity: medium Docs Contact:
Priority: medium    
Version: 9CC: ian, jorton, mattdm, srevivo
Target Milestone: ---   
Target Release: ---   
Hardware: i686   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2007-01-02 18:29:32 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jeff Trawick 2003-05-15 11:40:53 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.3) Gecko/20030312

Description of problem:
gdb segfaults when a breakpoint in threaded Apache (httpd.worker) child process
is hit, after attaching to the process with gdb and setting a breakpoint.

The segfault occurs at the time that gdb should gain control over the thread
that is handling the HTTP request.

This is seen when running httpd.worker as a normal user.  Different problems
occur when doing this as root.  (See PR 90900.)

Version-Release number of selected component (if applicable):
gdb-5.3post-0.20021129.18

How reproducible:
Always

Steps to Reproduce:
As normal user, run httpd.worker from RH9 httpd package.  This requires a very
minimal setup like this:
[trawick@feather trawick]$ pwd
/home/trawick
[trawick@feather trawick]$ find rhpr
rhpr
rhpr/conf
rhpr/conf/httpd.conf
rhpr/logs
rhpr/logs/error_log   (created when httpd.worker starts)
rhpr/logs/httpd.pid   (created when httpd.worker starts)
[trawick@feather trawick]$ cat rhpr/conf/httpd.conf
listen 8080

[trawick@feather trawick]$

Start httpd.worker like this:

$ /usr/sbin/httpd.worker -d /home/trawick/rhpr

Attach to a child process, set a breakpoint, and continue:

$ gdb /usr/sbin/httpd.worker 13446
(logs of output about shared libs)
0xffffe002 in ?? ()
(gdb) break ap_process_connection
Breakpoint 1 at 0x8071296
(gdb) c
Continuing.

Now hit the listening socket with HTTP requests until some thread in that child
process handles the connection.  It may take several requests.

Segmentation fault
[trawick@feather trawick]$

(no other output but "Segmentation fault")

I assume that it occurred when a thread in the child controlled by gdb hit the
desired breakpoint.


Actual Results:  gdb segfaulted

Expected Results:  gdb should have gained control of thread execution and
allowed the user to step through the code and examine data and perform other
debugging tasks.

Additional info:

Comment 1 Joe Orton 2003-05-15 14:01:44 UTC
Can you post the httpd.conf you used to reproduce this as well? With a minimal
config, this actually seems to work OK.

$ gdb /usr/sbin/httpd.worker 3106
...
(gdb) break ap_process_connection
Breakpoint 1 at 0x8071292: file /usr/src/debug/httpd-2.0.40/server/connection.c,
line 202.
(gdb) cont
Continuing.
[Switching to Thread 1094487344 (LWP 3108)]
 
Breakpoint 1, ap_process_connection (c=0x813ac78, csd=0x813aba0) at
/usr/src/debug/httpd-2.0.40/server/connection.c:202
202         ap_update_vhost_given_ip(c);
(gdb) print *c
$1 = {pool = 0x813ab68, base_server = 0x809fdb0, vhost_lookup_data = 0x0, 
...


Comment 2 Jeff Trawick 2003-05-15 14:30:57 UTC
Joe, the minimal config (directory structure, httpd.conf contents, httpd.worker
command-line parameters) is in my original post.  The config file is one line
("Listen 8080"), so you probably missed that.

I will first update to the latest kernel, reboot, and try again.

If that fails, I'll then update to the lastest glibc, reboot, and try again.

Then I'll report back.

Thanks!

Comment 3 Jeff Trawick 2003-05-15 16:57:06 UTC
Status so far: 

upgraded to latest kernel...  now running with 2.4.20-13.9

I can hit breakpoints in an Apache child process.  When the process is running
and I attempt to regain control with the interrupt key (Ctrl-C) I am unable to
regain control and must kill gdb.


Comment 4 Jeff Trawick 2003-05-15 20:53:19 UTC
Some more data:

updated to latest glibc rpms, rebooted, built prefork flavor of Apache but APR
is still thread-enabled

attach to child process, set a breakpoint, hit it, statement step for a while
but statement step loses control:

$ gdb ~/apacheinst/bin/httpd 29897
GNU gdb Red Hat Linux (5.3post-0.20021129.18rh)
Copyright 2003 Free Software Foundation, Inc.
(lots of output)
Reading symbols from /lib/tls/libpthread.so.0...done.
[New Thread 1075583328 (LWP 29897)]
(what's up with that?)
0xffffe002 in ?? ()
(gdb) break ap_die
Breakpoint 1 at 0x807838f: file http_request.c, line 118.
(gdb) c
Continuing.
[Switching to Thread 1075583328 (LWP 29897)]

Breakpoint 1, ap_die (type=404, r=0x81959d8) at http_request.c:118
118         int error_index = ap_index_of_response(type);
(gdb) n
keep doing "n" okay for a while then:
(gdb)
232         ap_send_error_response(r_1st_err, recursive_error);

it hangs at this point...  gdb doesn't regain control when
ap_send_error_response() finishes

now do Ctrl-C
Program received signal SIGINT, Interrupt.
0xffffe002 in ?? ()
(gdb) where
#0  0xffffe002 in ?? ()
#1  0x4013eb8f in proc_mutex_proc_pthread_acquire (mutex=0x0) at proc_mutex.c:412
#2  0x4013f1d8 in apr_proc_mutex_lock (mutex=0xfffffffc) at proc_mutex.c:857
#3  0x0809cc01 in accept_mutex_on () at prefork.c:245
#4  0x0809d414 in child_main (child_num_arg=-4) at prefork.c:580
#5  0x0809d514 in make_child (s=0x80e8c18, slot=1) at prefork.c:747
#6  0x0809d5b3 in startup_children (number_to_start=4) at prefork.c:765
#7  0x0809dca5 in ap_mpm_run (_pconf=0x1, plog=0x81247b0, s=0x80e8c18) at
prefork.c:982
#8  0x080a2a77 in main (argc=3, argv=0xbfffe2c4) at main.c:671
#9  0x420156a4 in __libc_start_main () from /lib/tls/libc.so.6

The parameters as displayed to apr_proc_mutex_lock() and
proc_mutex_proc_pthread_acquire() are not correct.  Is it because the main
thread is in a pthread mutex?

Those functions had to access storage via those ptrs, and they didn't
segfault while doing so.

Comment 5 Jeff Trawick 2003-05-16 01:39:36 UTC
Some more problems in a different debug session:

gdb bin/httpd          (prefork MPM, thread-enabled APR)
(gdb) r -DONE_PROCESS

let it run for a while and break
(gdb) where
#0  0xffffe002 in ?? ()
#1  0x08089c87 in connect_to_daemon (sdptr=0xfffffdfe, r=0x8193588,
conf=0x80e9928) at mod_cgid.c:1085
...
(gdb) n
Cannot find bounds of current function
(gdb) n
Cannot find bounds of current function
(gdb) up
#1  0x08089c87 in connect_to_daemon (sdptr=0xfffffdfe, r=0x8193588,
conf=0x80e9928) at mod_cgid.c:1085
1085                    apr_sleep(sliding_timer);
(gdb) n
Cannot find bounds of current function


Comment 6 Bill Nottingham 2006-08-05 03:15:33 UTC
Red Hat apologizes that these issues have not been resolved yet. We do want to
make sure that no important bugs slip through the cracks.

Red Hat Linux 7.3 and Red Hat Linux 9 are no longer supported by Red Hat, Inc.
They are maintained by the Fedora Legacy project (http://www.fedoralegacy.org/)
for security updates only. If this is a security issue, please reassign to the
'Fedora Legacy' product in bugzilla. Please note that Legacy security update
support for these products will stop on December 31st, 2006.

If this is not a security issue, please check if this issue is still present
in a current Fedora Core release. If so, please change the product and version
to match, and check the box indicating that the requested information has been
provided.

If you are currently still running Red Hat Linux 7.3 or 9, please note that
Fedora Legacy security update support for these products will stop on December
31st, 2006. You are strongly advised to upgrade to a current Fedora Core release
or Red Hat Enterprise Linux or comparable. Some information on which option may
be right for you is available at http://www.redhat.com/rhel/migrate/redhatlinux/.

Any bug still open against Red Hat Linux 7.3 or 9 at the end of 2006 will be
closed 'CANTFIX'. Again, if this bug still exists in a current release, or is a
security issue, please change the product as necessary. We thank you for your
help, and apologize again that we haven't handled these issues to this point.


Comment 8 Bill Nottingham 2007-01-02 18:29:32 UTC
Red Hat Linux 7.3 and Red Hat Linux 9 are no longer supported by Red Hat, Inc.
f you are currently still running Red Hat Linux 7.3 or 9, you are strongly
advised to upgrade to a current Fedora Core release or Red Hat Enterprise Linux
or comparable. Some information on which option may be right for you is
available at http://www.redhat.com/rhel/migrate/redhatlinux/.

Closing as CANTFIX.