Description of problem: Apache seg faults serving any of the page. callstack: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread -1216059632 (LWP 3165)] 0x8001787a in ap_rgetline_core (s=0x80403068, n=8192, read=0xbf8e41a8, r=0x80403050, fold=0, bb=0x80403ff8) at /usr/src/debug/httpd-2.2.4/server/protocol.c:250 /usr/src/debug/httpd-2.2.4/server/protocol.c:250:8384:beg:0x8001787a (gdb) p *e $1 = {link = {next = 0x80403ffc, prev = 0x80403ffc}, type = 0xb7e8c1c0, length = 37, start = 0, data = 0x0, free = 0x80401088, list = 0xb7e7ab60} (gdb) bt #0 0x8001787a in ap_rgetline_core (s=0x80403068, n=8192, read=0xbf8e41a8, r=0x80403050, fold=0, bb=0x80403ff8) at /usr/src/debug/httpd-2.2.4/server/protocol.c:250 #1 0x80018213 in ap_read_request (conn=0x803f91c0) at /usr/src/debug/httpd-2.2.4/server/protocol.c:596 #2 0x80030fae in ap_process_http_connection (c=0x803f91c0) at /usr/src/debug/httpd-2.2.4/modules/http/http_core.c:177 #3 0x8002c80d in ap_run_process_connection (c=0x803f91c0) at /usr/src/debug/httpd-2.2.4/server/connection.c:43 #4 0x8002c90c in ap_process_connection (c=0x803f91c0, csd=0x803f9028) at /usr/src/debug/httpd-2.2.4/server/connection.c:178 #5 0x80038512 in child_main (child_num_arg=<value optimized out>) at /usr/src/debug/httpd-2.2.4/server/mpm/prefork/prefork.c:640 #6 0x80038821 in make_child (s=0x800583f0, slot=2) at /usr/src/debug/httpd-2.2.4/server/mpm/prefork/prefork.c:736 #7 0x80039223 in ap_mpm_run (_pconf=0x80056550, plog=0x80084608, s=0x800583f0) at /usr/src/debug/httpd-2.2.4/server/mpm/prefork/prefork.c:871 #8 0x8001022a in main (argc=-2147138104, argv=0x80122388) at /usr/src/debug/httpd-2.2.4/server/main.c:717 Version-Release number of selected component (if applicable): How reproducible: 100% Steps to Reproduce: 1. launch httpd 2. try to download any page the server serves Actual results: segmentation fault Additional info: rpm -qa http* httpd-tools-2.2.4-8 httpd-manual-2.2.4-8 httpd-2.2.4-8 httpd-debuginfo-2.2.4-8 Any other packages are fully up to date.
upgrade to 2.2.4-9 didn't solve the problem. callstack: #0 0xb6ea7782 in python_handler (req=0x803f9c58, phase=0xb6ea91c1 "PythonInitHandler") at mod_python.c:1549 #1 0xb6ea803a in PythonPostReadRequestHandler (req=0x803f9c58) at mod_python.c:2898 #2 0x800162de in ap_run_post_read_request () from /usr/sbin/httpd #3 0x80018784 in ap_read_request () from /usr/sbin/httpd #4 0x80030fae in ?? () from /usr/sbin/httpd #5 0x8002c80d in ap_run_process_connection () from /usr/sbin/httpd #6 0x8002c90c in ap_process_connection () from /usr/sbin/httpd #7 0x80038512 in ?? () from /usr/sbin/httpd #8 0x80038821 in ?? () from /usr/sbin/httpd #9 0x80039223 in ap_mpm_run () from /usr/sbin/httpd #10 0x8001022a in main () from /usr/sbin/httpd There is an attempt to load the module_config: /* get configuration */ conf = (py_config *) ap_get_module_config(req->per_dir_config, &python_module); but the per_dir_config field is NULL (gdb) p req->per_dir_config $2 = (struct ap_conf_vector_t *) 0x0 (gdb) p *req $4 = {pool = 0x803f9c20, connection = 0x803efdc8, server = 0x800583f0, next = 0x0, prev = 0x0, main = 0x0, the_request = 0x803fac20 "GET /mywiki/ HTTP/1.1", assbackwards = 0, proxyreq = 0, header_only = 0, protocol = 0x803faca8 "HTTP/1.1", proto_num = 1001, hostname = 0x803fb120 "localhost", request_time = 1188120322386327, status_line = 0x0, status = 200, method = 0x803fac70 "GET", method_number = 0, allowed = 0, allowed_xmethods = 0x0, allowed_methods = 0x803f9dd8, sent_bodyct = 0, bytes_sent = 0, mtime = 0, chunked = 0, range = 0x0, clength = 0, remaining = 0, read_length = -9205460337451622904, read_body = -2143312832, read_chunked = -2143313840, expecting_100 = 2151654808, headers_in = 0x0, headers_out = 0x0, err_headers_out = 0x0, subprocess_env = 0x0, notes = 0x0, content_type = 0x0, handler = 0x0, content_encoding = 0x0, content_languages = 0x0, vlist_validator = 0x803fac88 "/mywiki/", user = 0x803fac98 "/mywiki/", ap_auth_type = 0x0, no_cache = 0, no_local_copy = 0, unparsed_uri = 0x0, uri = 0x0, filename = 0x0, canonical_filename = 0x0, path_info = 0x0, args = 0x0, finfo = {pool = 0x0, valid = 0, protection = 0, filetype = APR_NOFILE, user = 0, group = 0, inode = 0, device = 0, nlink = 0, size = 0, csize = 0, atime = 0, mtime = 0, ctime = 0, fname = 0x0, name = 0x0, filehand = 0x803fac98}, parsed_uri = {scheme = 0x0, hostinfo = 0x0, user = 0x0, password = 0x10000 "Ìa\001", hostname = 0x2 <Address 0x2 out of bounds>, port_str = 0x8005af48 "X±\005\200", path = 0x803fa6f0 "à ©?\200", query = 0x0, fragment = 0x803faba0 "Ø~\005\200", hostent = 0x803fb130, port = 43936, is_initialized = 1, dns_looked_up = 1, dns_resolved = 1}, used_path_info = -2143309520, per_dir_config = 0x0, request_config = 0x0, htaccess = 0x0, output_filters = 0x803f9de8, input_filters = 0x0, proto_output_filters = 0x803f9c20, proto_input_filters = 0x4, eos_sent = 0}
Maybe this would be a little bit helpful. As you can see below, up to the frame 2 (function ap_run_post_read_request()) the per_dir_config field has still proper value (non NULL). (gdb) bt #0 0xb6e81782 in python_handler (req=0x803fed38, phase=0xb6e831c1 "PythonInitHandler") at mod_python.c:1549 #1 0xb6e8203a in PythonPostReadRequestHandler (req=0x803fed38) at mod_python.c:2898 #2 0x800162de in ap_run_post_read_request (r=0x803fed38) at /usr/src/debug/httpd-2.2.4/server/protocol.c:1636 #3 0x80018784 in ap_read_request (conn=0x803f4ea8) at /usr/src/debug/httpd-2.2.4/server/protocol.c:1011 #4 0x80030fae in ap_process_http_connection (c=0x803f4ea8) at /usr/src/debug/httpd-2.2.4/modules/http/http_core.c:177 #5 0x8002c80d in ap_run_process_connection (c=0x803f4ea8) at /usr/src/debug/httpd-2.2.4/server/connection.c:43 #6 0x8002c90c in ap_process_connection (c=0x803f4ea8, csd=0x803f4d10) at /usr/src/debug/httpd-2.2.4/server/connection.c:178 #7 0x80038512 in child_main (child_num_arg=<value optimized out>) at /usr/src/debug/httpd-2.2.4/server/mpm/prefork/prefork.c:640 #8 0x80038821 in make_child (s=0x800583f0, slot=2) at /usr/src/debug/httpd-2.2.4/server/mpm/prefork/prefork.c:736 #9 0x80039223 in ap_mpm_run (_pconf=0x80056550, plog=0x80084608, s=0x800583f0) at /usr/src/debug/httpd-2.2.4/server/mpm/prefork/prefork.c:871 #10 0x8001022a in main (argc=-2147138104, argv=0x8011ced0) at /usr/src/debug/httpd-2.2.4/server/main.c:717 (gdb) frame 0 #0 0xb6e81782 in python_handler (req=0x803fed38, phase=0xb6e831c1 "PythonInitHandler") at mod_python.c:1549 (gdb) p req $1 = (request_rec *) 0x803fed38 (gdb) p req->per_dir_config $2 = (struct ap_conf_vector_t *) 0x0 (gdb) frame 2 #2 0x800162de in ap_run_post_read_request (r=0x803fed38) at /usr/src/debug/httpd-2.2.4/server/protocol.c:1636 (gdb) p r $3 = (request_rec *) 0x803fed38 (gdb) p r->per_dir_config $4 = (struct ap_conf_vector_t *) 0x8005af48
What about the crash in ap_rgetline_core? Is that gone? I've not seen such a crash before. The mod_python crashes look like an ABI break or random memory corruption. It could be due to either the old expat/db4 getting linked into the process along with the new versions.
(In reply to comment #3) > What about the crash in ap_rgetline_core? Is that gone? I've not seen such a > crash before. That one seems to be gone. > > The mod_python crashes look like an ABI break or random memory corruption. > > It could be due to either the old expat/db4 getting linked into the process > along with the new versions. > Just to avoid such a situation in /var/cache/yum/development/packages directory I invoked: rpm -qa| awk '{print $1"*"}' | xargs -n 1 find -name | xargs -n 1 rpm -U --nodeps --force and even after reboot, I'm observing the same crash in mod_python (I'm just accessing http://localhost/).
Created attachment 177981 [details] rpm -qa output versions of all currently installed packages
e.g. the python as of -8 is still pulling in the old libexpat. We need to wait until we can get to the state where "rpm --erase compat-db compat-expat1" is possible.
Can you run "print sizeof(request_rec)" in gdb from your core dump?
Here it comes: (gdb) print sizeof(request_rec) $1 = 384
But after upgrading the following packages: Updating: python-devel i386 2.5.1-9.fc8 development 879 k python-imaging-tk i386 1.1.6-4.fc8 development 21 k python-libs i386 2.5.1-9.fc8 development 565 k python-numeric i386 24.2-6.fc8 development 739 k python-setuptools noarch 0.6c6-2.fc8 development 88 k Updating for dependencies: python i386 2.5.1-9.fc8 development 5.2 M python-imaging i386 1.1.6-4.fc8 development 420 k tkinter i386 2.5.1-9.fc8 development 216 k it changed to: (gdb) print sizeof(request_rec) $1 = 412
Thanks for that. It seems that the apr -2 package has a broken ABI :(
*** Bug 271321 has been marked as a duplicate of this bug. ***
*** Bug 272841 has been marked as a duplicate of this bug. ***
*** Bug 259561 has been marked as a duplicate of this bug. ***
apr-1.2.9-2 broke API and ABI on 32-bit platforms - it dropped large file support. Everything built against this version of APR will need to be rebuilt after APR is fixed.
*** Bug 242952 has been marked as a duplicate of this bug. ***
This should be fixed as of the following builds in dist-f8: httpd-2.2.4-10 mod_python-3.3.1-5 mod_auth_kerb-5.3-5 mod_perl-2.0.3-13 apr-1.2.9-4 apr-util-1.2.8-12 php-5.2.3-9 mod_auth_pgsql-2.0.3-6 subversion-1.4.4-7
You need to recompile mod_security as well.
This should be closed, right?
Yes indeed.