From Bugzilla Helper: User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1) Gecko/20030225 Description of problem: I recently compiled Apache 2.0.44 from source, installed it, and then compiled and installed mod_perl CVS from source, and Perl 5.8.0 from source (using: DEBUGGING MULTIPLICITY USE_ITHREADS USE_LARGE_FILES PERL_IMPLICIT_CONTEXT). Apache was compiled with the following: Server version: Apache/2.0.44 Server built: Apr 1 2003 00:26:43 Server's Module Magic Number: 20020903:0 Architecture: 32-bit Server compiled with.... -D APACHE_MPM_DIR="server/mpm/worker" -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 -D APR_USE_PROC_PTHREAD_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D HTTPD_ROOT="/home/swalker/bin/apache" -D SUEXEC_BIN="/home/swalker/bin/apache/bin/suexec" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="conf/mime.types" -D SERVER_CONFIG_FILE="conf/httpd.conf" Attempting to attach to any of the apache processes after they've started causes them to immediately exit after continuing after the initial break, making debugging absolutely impossible. Starting apache in "single threaded" mode with -X This did not happen under RedHat 8.0 with the exact same setup and sources. GDB session: (gdb) file httpd Reading symbols from httpd...done. (gdb) run -X Starting program: /home/swalker/bin/apache/bin/httpd -X [New Thread 1077339392 (LWP 25915)] [New Thread 1099852992 (LWP 25919)] [New Thread 1116941376 (LWP 25920)] [New Thread 1125329856 (LWP 25921)] [New Thread 1133718336 (LWP 25922)] [New Thread 1142106816 (LWP 25923)] [New Thread 1150495296 (LWP 25924)] [New Thread 1158883776 (LWP 25925)] [New Thread 1167272256 (LWP 25926)] [New Thread 1175660736 (LWP 25927)] [New Thread 1184049216 (LWP 25928)] [New Thread 1192437696 (LWP 25929)] [New Thread 1200826176 (LWP 25930)] [New Thread 1209214656 (LWP 25931)] [New Thread 1217603136 (LWP 25932)] [New Thread 1225991616 (LWP 25933)] [New Thread 1234380096 (LWP 25934)] [New Thread 1242768576 (LWP 25935)] [New Thread 1251157056 (LWP 25936)] [New Thread 1259545536 (LWP 25937)] [New Thread 1267934016 (LWP 25938)] [New Thread 1276322496 (LWP 25939)] [New Thread 1284710976 (LWP 25940)] [New Thread 1293099456 (LWP 25941)] [New Thread 1301487936 (LWP 25942)] [New Thread 1309876416 (LWP 25943)] [New Thread 1318264896 (LWP 25944)] [New Thread 1326653376 (LWP 25945)] [Thread 1099852992 (LWP 25919) exited] [Thread 1116941376 (LWP 25920) exited] [Thread 1125329856 (LWP 25921) exited] Program received signal SIGHUP, Hangup. [Switching to Thread 1326653376 (LWP 25945)] 0xffffe002 in ?? () (gdb) bt #0 0xffffe002 in ?? () #1 0x40257fe5 in apr_accept (new=0x4f131b3c, sock=0x80e05a8, connection_context=0xa88d8c8) at sockets.c:420 #2 0x080b4d62 in unixd_accept (accepted=0x4f131b6c, lr=0xfffffe00, ptrans=0xa88d8c8) at unixd.c:457 #3 0x0809c7e2 in listener_thread (thd=0x9a428c0, dummy=0x80e0540) at worker.c:808 #4 0x40259518 in dummy_worker (opaque=0xfffffe00) at thread.c:127 #5 0x4021d2b6 in start_thread () from /lib/tls/libpthread.so.0 (gdb) c Continuing. [Thread 1326653376 (LWP 25945) exited] [Thread 1133718336 (LWP 25922) exited] [Thread 1142106816 (LWP 25923) exited] [Thread 1150495296 (LWP 25924) exited] [Thread 1158883776 (LWP 25925) exited] [Thread 1167272256 (zombie) exited] [Thread 1175660736 (zombie) exited] [Thread 1184049216 (LWP 25928) exited] [Thread 1192437696 (zombie) exited] [Thread 1200826176 (LWP 25930) exited] [Thread 1209214656 (zombie) exited] [Thread 1217603136 (LWP 25932) exited] [Thread 1225991616 (LWP 25933) exited] [Thread 1234380096 (LWP 25934) exited] [Thread 1242768576 (LWP 25935) exited] [Thread 1251157056 (LWP 25936) exited] [Thread 1259545536 (LWP 25937) exited] [Thread 1267934016 (LWP 25938) exited] [Thread 1276322496 (zombie) exited] [Thread 1284710976 (LWP 25940) exited] [Thread 1293099456 (zombie) exited] [Thread 1301487936 (LWP 25942) exited] [Thread 1309876416 (LWP 25943) exited] [Thread 1318264896 (zombie) exited] Program exited normally. Version-Release number of selected component (if applicable): GNU gdb Red Hat Linux (5.3post-0.20021129.18rh) How reproducible: Always Steps to Reproduce: 1. See Description Additional info:
There are still a few problems with the kernel, glibc and gdb when using the new NPTL thread model. Could you try the same but with setting LD_ASSUME_KERNEL=2.4.1 or LD_ASSUME_KERNEL=2.2.5 ?
This report needs more information. It mentions a scenario using attach but does not show the details of that. It shows a scenario using run but does not describe what is going on there. Can you demonstrate the problem without downloading and building a large software package? It will be much easier for us to figure out if you can show the same problem debugging a binary that comes with RHL, or a small program. Please address the failure mode using "run" and the failure mode using "attach" separately with complete details on each. The situations are rather different and might not be the same problem. Also, please show your kernel version info, i.e. uname -a; and rpm -q glibc.
In answer to part of Roland's question: glibc-2.3.2-24.9 (glibc errata package I was told to try, however the problem also occurs with the original glibc in RedHat 9) Linux swalker.theiqgroup.com 2.4.20-8 #1 Thu Mar 13 17:18:24 EST 2003 i686 athlon i386 GNU/Linux The scenario is starting apache 2.0.44 with mod_perl CVS, and then trying to attach to any process that it started. I will attempt to show the same process with the Apache that RedHat ships. The reason I do not use the Apache that RedHat ships is because it is not new enough to work properly with the newest mod_perl which contains fixes necessary to develop the application that I work on.
Ok, here's an example *attach* session with the httpd that's included with RedHat 9: [root@swalker root]# /etc/init.d/httpd start Starting httpd: [ OK ] [root@swalker root]# which httpd /usr/sbin/httpd [root@swalker root]# ps -elf | grep -i httpd 1 S root 4629 1 0 77 0 - 1636 schedu 14:47 ? 00:00:00 /usr/sbin/httpd 5 S apache 4632 4629 0 84 0 - 1642 schedu 14:47 ? 00:00:00 [httpd] 5 S apache 4633 4629 0 84 0 - 1642 schedu 14:47 ? 00:00:00 [httpd] 5 S apache 4634 4629 0 84 0 - 1642 schedu 14:47 ? 00:00:00 [httpd] 5 S apache 4635 4629 0 84 0 - 1642 schedu 14:47 ? 00:00:00 [httpd] 5 S apache 4636 4629 0 84 0 - 1642 schedu 14:47 ? 00:00:00 [httpd] 5 S apache 4637 4629 0 84 0 - 1642 schedu 14:47 ? 00:00:00 [httpd] 5 S apache 4638 4629 0 84 0 - 1642 schedu 14:47 ? 00:00:00 [httpd] 5 S apache 4639 4629 0 84 0 - 1642 schedu 14:47 ? 00:00:00 [httpd] 0 S root 4643 4572 0 75 0 - 894 pipe_w 14:47 pts/0 00:00:00 grep -i httpd [root@swalker root]# gdb GNU gdb Red Hat Linux (5.3post-0.20021129.18rh) Copyright 2003 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu". (gdb) attach 4632 Attaching to process 4632 Couldn't get registers: Operation not permitted. (gdb) c Continuing. Couldn't get registers: Operation not permitted. (gdb) q The program is running. Quit anyway (and detach it)? (y or n) y Detaching from program: , process 4632 ptrace: Operation not permitted. (gdb) q The program is running. Quit anyway (and detach it)? (y or n) y Detaching from program: , process 4632 ptrace: Operation not permitted. (gdb) Note how I can't even quit GDB.
Here's a few examples of a *run* session with the Apache that comes with RedHat 9: [root@swalker sbin]# gdb GNU gdb Red Hat Linux (5.3post-0.20021129.18rh) Copyright 2003 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu". (gdb) file ./httpd Reading symbols from ./httpd...(no debugging symbols found)...done. (gdb) r Starting program: /usr/sbin/httpd (no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...[New Thread 1077361280 (LWP 4783)] Program exited normally. (gdb) Even though it says program exited normally, checking the process reveals otherwise: 1 S root 4790 1 0 75 0 - 1636 schedu 14:54 ? 00:00:00 /usr/sbin/httpd 5 S apache 4791 4790 0 81 0 - 1642 schedu 14:54 ? 00:00:00 [httpd] 5 S apache 4792 4790 0 81 0 - 1642 schedu 14:54 ? 00:00:00 [httpd] 5 S apache 4793 4790 0 81 0 - 1642 schedu 14:54 ? 00:00:00 [httpd] 5 S apache 4794 4790 0 81 0 - 1642 schedu 14:54 ? 00:00:00 [httpd] 5 S apache 4795 4790 0 81 0 - 1642 schedu 14:54 ? 00:00:00 [httpd] 5 S apache 4796 4790 0 81 0 - 1642 schedu 14:54 ? 00:00:00 [httpd] 5 S apache 4797 4790 0 81 0 - 1642 schedu 14:54 ? 00:00:00 [httpd] 5 S apache 4798 4790 0 81 0 - 1642 schedu 14:54 ? 00:00:00 [httpd] Or this one where I attempted to start it in single threaded mode: [root@swalker sbin]# gdb GNU gdb Red Hat Linux (5.3post-0.20021129.18rh) Copyright 2003 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu". (gdb) file ./httpd Reading symbols from ./httpd...(no debugging symbols found)...done. (gdb) run -X Starting program: /usr/sbin/httpd -X (no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...[New Thread 1077361280 (LWP 4901)] Couldn't get registers: Operation not permitted. (gdb) c Continuing. Cannot fetch general-purpose registers for thread 1077361280: generic error (gdb)
Have you tried the LD_ASSUME_KERNEL workaround? Does that make any difference?
Using LD_ASSUME_KERNEL=2.4.1 while attaching to the child process started by httpd -X causes "Trace/Breakpoint Trap" to be printed in the terminal where I started httpd -X, the parent process exits and the child process that i'm attached to remains running, and gdb to just "sit there" doing nothing in the terminal where I'm using it.
LD_ASSUME_KERNEL=2.2.5 produces the same affect as LD_ASSUME_KERNEL=2.4.1 as mentioned above.
Using LD_ASSUME_KERNEL=2.4.1, and attaching to a child process of httpd started in normal mode (threading) and using a breakpoint and then continuing causes GDB to seg fault upon triggering the breakpoint. The apache processes go about their merry way.
You might want to look at Bug #88047. This might be related to the kmod-ptrace kernel patch instead of NTLM. Notice that your ps listing shows process 4791 as "[httpd]". This is because it is running suid (euid != uid) and the ptrace patch makes most ptrace operations fail (which /proc, ps(1), and gdb(1) make use of)....even if you are root! This might also explain the "Couldn't get registers: Operation not permitted." in gdb, again, even though you are root and should have permission to do anything. Just a guess here... One thing you might want to try is to start apache as a non-root user on a high unpriviledged port. Make sure the RunAs directives specify the same user you start apache as, so that euid = uid and egid = gid for all the child processes/threads.
Actually, when I'm running Apache 2.0.44 I run it as my user (not root) on the local system and it binds to port 8080, so no luck with that workaround. The examples you see above are done as root because I was asked to show the issue when trying to do it to the apache that comes with RedHat 9. I get the "couldn't get registers" message regardless of whether or not I'm running as root I'm fairly certain. But, now that you mention it, I do also have problems with doing strace on the apache process sometimes.
The good news is that glibc-2.3.2-27.9 fixed this problem, in fact debugging threaded programs is now very pleasant compared to what it was in the past. Feel free to close this!
Not a gdb bug after all; but a glibc problem. I am closing this.