Bug 88010 - GDB unable to debug apache 2.0.44
GDB unable to debug apache 2.0.44
Status: CLOSED NOTABUG
Product: Red Hat Linux
Classification: Retired
Component: gdb (Show other bugs)
9
athlon Linux
medium Severity medium
: ---
: ---
Assigned To: Elena Zannoni
Jay Turner
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2003-04-04 11:19 EST by Shawn Walker
Modified: 2015-01-07 19:04 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2003-04-22 17:47:03 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Shawn Walker 2003-04-04 11:19:57 EST
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1) Gecko/20030225

Description of problem:
I recently compiled Apache 2.0.44 from source, installed it, and then compiled
and installed mod_perl CVS from source, and Perl 5.8.0 from source (using:
DEBUGGING MULTIPLICITY USE_ITHREADS USE_LARGE_FILES PERL_IMPLICIT_CONTEXT).
Apache was compiled with the following:

Server version: Apache/2.0.44
Server built:   Apr  1 2003 00:26:43
Server's Module Magic Number: 20020903:0
Architecture:   32-bit
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/worker"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6
 -D APR_USE_PROC_PTHREAD_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D HTTPD_ROOT="/home/swalker/bin/apache"
 -D SUEXEC_BIN="/home/swalker/bin/apache/bin/suexec"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

Attempting to attach to any of the apache processes after they've started causes
them to immediately exit after continuing after the initial break, making
debugging absolutely impossible. Starting apache in "single threaded" mode with -X

This did not happen under RedHat 8.0 with the exact same setup and sources.

GDB session:
(gdb) file httpd
Reading symbols from httpd...done.
(gdb) run -X
Starting program: /home/swalker/bin/apache/bin/httpd -X
[New Thread 1077339392 (LWP 25915)]
[New Thread 1099852992 (LWP 25919)]
[New Thread 1116941376 (LWP 25920)]
[New Thread 1125329856 (LWP 25921)]
[New Thread 1133718336 (LWP 25922)]
[New Thread 1142106816 (LWP 25923)]
[New Thread 1150495296 (LWP 25924)]
[New Thread 1158883776 (LWP 25925)]
[New Thread 1167272256 (LWP 25926)]
[New Thread 1175660736 (LWP 25927)]
[New Thread 1184049216 (LWP 25928)]
[New Thread 1192437696 (LWP 25929)]
[New Thread 1200826176 (LWP 25930)]
[New Thread 1209214656 (LWP 25931)]
[New Thread 1217603136 (LWP 25932)]
[New Thread 1225991616 (LWP 25933)]
[New Thread 1234380096 (LWP 25934)]
[New Thread 1242768576 (LWP 25935)]
[New Thread 1251157056 (LWP 25936)]
[New Thread 1259545536 (LWP 25937)]
[New Thread 1267934016 (LWP 25938)]
[New Thread 1276322496 (LWP 25939)]
[New Thread 1284710976 (LWP 25940)]
[New Thread 1293099456 (LWP 25941)]
[New Thread 1301487936 (LWP 25942)]
[New Thread 1309876416 (LWP 25943)]
[New Thread 1318264896 (LWP 25944)]
[New Thread 1326653376 (LWP 25945)]
[Thread 1099852992 (LWP 25919) exited]
[Thread 1116941376 (LWP 25920) exited]
[Thread 1125329856 (LWP 25921) exited]
 
Program received signal SIGHUP, Hangup.
[Switching to Thread 1326653376 (LWP 25945)]
0xffffe002 in ?? ()
(gdb) bt
#0  0xffffe002 in ?? ()
#1  0x40257fe5 in apr_accept (new=0x4f131b3c, sock=0x80e05a8,
connection_context=0xa88d8c8) at sockets.c:420
#2  0x080b4d62 in unixd_accept (accepted=0x4f131b6c, lr=0xfffffe00,
ptrans=0xa88d8c8) at unixd.c:457
#3  0x0809c7e2 in listener_thread (thd=0x9a428c0, dummy=0x80e0540) at worker.c:808
#4  0x40259518 in dummy_worker (opaque=0xfffffe00) at thread.c:127
#5  0x4021d2b6 in start_thread () from /lib/tls/libpthread.so.0
(gdb) c
Continuing.
[Thread 1326653376 (LWP 25945) exited]
[Thread 1133718336 (LWP 25922) exited]
[Thread 1142106816 (LWP 25923) exited]
[Thread 1150495296 (LWP 25924) exited]
[Thread 1158883776 (LWP 25925) exited]
[Thread 1167272256 (zombie) exited]
[Thread 1175660736 (zombie) exited]
[Thread 1184049216 (LWP 25928) exited]
[Thread 1192437696 (zombie) exited]
[Thread 1200826176 (LWP 25930) exited]
[Thread 1209214656 (zombie) exited]
[Thread 1217603136 (LWP 25932) exited]
[Thread 1225991616 (LWP 25933) exited]
[Thread 1234380096 (LWP 25934) exited]
[Thread 1242768576 (LWP 25935) exited]
[Thread 1251157056 (LWP 25936) exited]
[Thread 1259545536 (LWP 25937) exited]
[Thread 1267934016 (LWP 25938) exited]
[Thread 1276322496 (zombie) exited]
[Thread 1284710976 (LWP 25940) exited]
[Thread 1293099456 (zombie) exited]
[Thread 1301487936 (LWP 25942) exited]
[Thread 1309876416 (LWP 25943) exited]
[Thread 1318264896 (zombie) exited]
 
Program exited normally.


Version-Release number of selected component (if applicable):
GNU gdb Red Hat Linux (5.3post-0.20021129.18rh)

How reproducible:
Always

Steps to Reproduce:
1. See Description

Additional info:
Comment 1 Elena Zannoni 2003-04-04 13:19:05 EST
There are still a few problems with the kernel, glibc and gdb when using the new
NPTL thread model.

Could you try the same but with setting
LD_ASSUME_KERNEL=2.4.1
or LD_ASSUME_KERNEL=2.2.5 ?
Comment 2 Roland McGrath 2003-04-04 17:14:06 EST
This report needs more information.  It mentions a scenario using attach
but does not show the details of that.  It shows a scenario using run but
does not describe what is going on there.  Can you demonstrate the problem
without downloading and building a large software package?  It will be much
easier for us to figure out if you can show the same problem debugging a binary
that comes with RHL, or a small program.  Please address the failure mode using
"run" and the failure mode using "attach" separately with complete details on
each.  The situations are rather different and might not be the same problem.

Also, please show your kernel version info, i.e. uname -a; and rpm -q glibc.
Comment 3 Shawn Walker 2003-04-05 15:41:21 EST
In answer to part of Roland's question:
glibc-2.3.2-24.9 (glibc errata package I was told to try, however the problem
also occurs with the original glibc in RedHat 9)

Linux swalker.theiqgroup.com 2.4.20-8 #1 Thu Mar 13 17:18:24 EST 2003 i686
athlon i386 GNU/Linux

The scenario is starting apache 2.0.44 with mod_perl CVS, and then trying to
attach to any process that it started.

I will attempt to show the same process with the Apache that RedHat ships. The
reason I do not use the Apache that RedHat ships is because it is not new enough
to work properly with the newest mod_perl which contains fixes necessary to
develop the application that I work on.
Comment 4 Shawn Walker 2003-04-05 15:50:39 EST
Ok, here's an example *attach* session with the httpd that's included with RedHat 9:

[root@swalker root]# /etc/init.d/httpd start
Starting httpd:                                            [  OK  ]
[root@swalker root]# which httpd
/usr/sbin/httpd
[root@swalker root]# ps -elf | grep -i httpd
1 S root      4629     1  0  77   0    -  1636 schedu 14:47 ?        00:00:00
/usr/sbin/httpd
5 S apache    4632  4629  0  84   0    -  1642 schedu 14:47 ?        00:00:00
[httpd]
5 S apache    4633  4629  0  84   0    -  1642 schedu 14:47 ?        00:00:00
[httpd]
5 S apache    4634  4629  0  84   0    -  1642 schedu 14:47 ?        00:00:00
[httpd]
5 S apache    4635  4629  0  84   0    -  1642 schedu 14:47 ?        00:00:00
[httpd]
5 S apache    4636  4629  0  84   0    -  1642 schedu 14:47 ?        00:00:00
[httpd]
5 S apache    4637  4629  0  84   0    -  1642 schedu 14:47 ?        00:00:00
[httpd]
5 S apache    4638  4629  0  84   0    -  1642 schedu 14:47 ?        00:00:00
[httpd]
5 S apache    4639  4629  0  84   0    -  1642 schedu 14:47 ?        00:00:00
[httpd]
0 S root      4643  4572  0  75   0    -   894 pipe_w 14:47 pts/0    00:00:00
grep -i httpd
[root@swalker root]# gdb
GNU gdb Red Hat Linux (5.3post-0.20021129.18rh)
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu".
(gdb) attach 4632
Attaching to process 4632
Couldn't get registers: Operation not permitted.
(gdb) c
Continuing.
Couldn't get registers: Operation not permitted.
(gdb) q
The program is running.  Quit anyway (and detach it)? (y or n) y
Detaching from program: , process 4632
ptrace: Operation not permitted.
(gdb) q
The program is running.  Quit anyway (and detach it)? (y or n) y
Detaching from program: , process 4632
ptrace: Operation not permitted.
(gdb)

Note how I can't even quit GDB.
Comment 5 Shawn Walker 2003-04-05 15:57:37 EST
Here's a few examples of a *run* session with the Apache that comes with RedHat 9:

[root@swalker sbin]# gdb
GNU gdb Red Hat Linux (5.3post-0.20021129.18rh)
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu".
(gdb) file ./httpd
Reading symbols from ./httpd...(no debugging symbols found)...done.
(gdb) r
Starting program: /usr/sbin/httpd
(no debugging symbols found)...(no debugging symbols found)...(no debugging
symbols found)...[New Thread 1077361280 (LWP 4783)]
                                                                               
                                                                                   
Program exited normally.
(gdb)

Even though it says program exited normally, checking the process reveals otherwise:
1 S root      4790     1  0  75   0    -  1636 schedu 14:54 ?        00:00:00
/usr/sbin/httpd
5 S apache    4791  4790  0  81   0    -  1642 schedu 14:54 ?        00:00:00
[httpd]
5 S apache    4792  4790  0  81   0    -  1642 schedu 14:54 ?        00:00:00
[httpd]
5 S apache    4793  4790  0  81   0    -  1642 schedu 14:54 ?        00:00:00
[httpd]
5 S apache    4794  4790  0  81   0    -  1642 schedu 14:54 ?        00:00:00
[httpd]
5 S apache    4795  4790  0  81   0    -  1642 schedu 14:54 ?        00:00:00
[httpd]
5 S apache    4796  4790  0  81   0    -  1642 schedu 14:54 ?        00:00:00
[httpd]
5 S apache    4797  4790  0  81   0    -  1642 schedu 14:54 ?        00:00:00
[httpd]
5 S apache    4798  4790  0  81   0    -  1642 schedu 14:54 ?        00:00:00
[httpd]

Or this one where I attempted to start it in single threaded mode:

[root@swalker sbin]# gdb
GNU gdb Red Hat Linux (5.3post-0.20021129.18rh)
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu".
(gdb) file ./httpd
Reading symbols from ./httpd...(no debugging symbols found)...done.
(gdb) run -X
Starting program: /usr/sbin/httpd -X
(no debugging symbols found)...(no debugging symbols found)...(no debugging
symbols found)...[New Thread 1077361280 (LWP 4901)]
Couldn't get registers: Operation not permitted.
(gdb) c
Continuing.
Cannot fetch general-purpose registers for thread 1077361280: generic error
(gdb)
Comment 6 Elena Zannoni 2003-04-07 11:02:36 EDT
Have you tried the LD_ASSUME_KERNEL workaround? Does that make any difference?

Comment 7 Shawn Walker 2003-04-07 12:17:18 EDT
Using LD_ASSUME_KERNEL=2.4.1 while attaching to the child process started by
httpd -X causes "Trace/Breakpoint Trap" to be printed in the terminal where I
started httpd -X, the parent process exits and the child process that i'm
attached to remains running, and gdb to just "sit there" doing nothing in the
terminal where I'm using it.
Comment 8 Shawn Walker 2003-04-07 12:19:56 EDT
LD_ASSUME_KERNEL=2.2.5 produces the same affect as LD_ASSUME_KERNEL=2.4.1 as
mentioned above.
Comment 9 Shawn Walker 2003-04-07 12:27:06 EDT
Using LD_ASSUME_KERNEL=2.4.1, and attaching to a child process of httpd started
in normal mode (threading) and using a breakpoint and then continuing causes GDB
to seg fault upon triggering the breakpoint. The apache processes go about their
merry way.
Comment 10 Deron Meranda 2003-04-07 15:08:56 EDT
You might want to look at Bug #88047.  This might be related to the kmod-ptrace
kernel patch instead of NTLM.  Notice that your ps listing shows process 4791 as
"[httpd]".  This is because it is running suid (euid != uid) and the ptrace
patch makes most ptrace operations fail (which /proc, ps(1), and gdb(1) make use
of)....even if you are root!  This might also explain the "Couldn't get
registers: Operation not permitted." in gdb, again, even though you are root and
should have permission to do anything.  Just a guess here...

One thing you might want to try is to start apache as a non-root user on a high
unpriviledged port.  Make sure the RunAs directives specify the same user you
start apache as, so that euid = uid and egid = gid for all the child
processes/threads.
Comment 11 Shawn Walker 2003-04-07 15:27:12 EDT
Actually, when I'm running Apache 2.0.44 I run it as my user (not root) on the
local system and it binds to port 8080, so no luck with that workaround. The
examples you see above are done as root because I was asked to show the issue
when trying to do it to the apache that comes with RedHat 9. I get the "couldn't
get registers" message regardless of whether or not I'm running as root I'm
fairly certain.

But, now that you mention it, I do also have problems with doing strace on the
apache process sometimes.
Comment 12 Shawn Walker 2003-04-14 17:46:38 EDT
The good news is that glibc-2.3.2-27.9 fixed this problem, in fact debugging
threaded programs is now very pleasant compared to what it was in the past. Feel
free to close this!
Comment 13 Elena Zannoni 2003-04-22 17:47:03 EDT
Not a gdb bug after all; but a glibc problem. I am closing this.

Note You need to log in before you can comment on or make changes to this bug.