Description of problem: Apache worker thread dies (after) serving a request. The issue observed with devstack/keystone. Version-Release number of selected component (if applicable): mod_wsgi-4.4.1-1.fc21.x86_64.rpm How reproducible: always Steps to Reproduce: 1. Install keystone behind mod_wsgi as usual Actual results: $ openstack --debug --os-token 111222333444 --os-url=http://172.16.40.13:5000/v3 --os-identity-api-version=3 role add --user heat_domain_admin --domain 2e51407db9624241b93eb30b96de89c8 admin $ dmesg 00007f74de5f7b58 error 6 in libapr-1.so.0.5.1[7f74ee1ba000+32000] [ 866.252965] httpd[9680]: segfault at c ip 00007f74ee1cf030 sp 00007f74de5f7b58 error 6 in libapr-1.so.0.5.1[7f74ee1ba000+32000] [ 949.269946] httpd[9704]: segfault at c ip 00007f74ee1cf030 sp 00007f74de5f7b58 error 6 in libapr-1.so.0.5.1[7f74ee1ba000+32000] $[root@host httpd]# tail error_log pid 14839 exit signal Segmentation fault (11) [Mon Jan 05 12:37:20.749957 2015] [core:notice] [pid 9629] AH00052: child pid 14853 exit signal Segmentation fault (11) gdb: #0 apr_table_clear (t=0x0) at tables/apr_tables.c:467 #1 0x00007f38b9dbfe78 in read_chunked_trailers (b=b@entry=0x7f389c007100, merge=0, f=0x7f389c006e60, f=0x7f389c006e60, ctx=<optimized out>, ctx=<optimized out>) at http_filters.c:245 #2 0x00007f38b9dc161c in ap_http_filter (f=0x7f389c006e60, b=0x7f389c007100, mode=AP_MODE_READBYTES, block=APR_BLOCK_READ, readbytes=8192) at http_filters.c:475 #3 0x00007f38b9dc2341 in ap_discard_request_body (r=r@entry=0x7f389c004980) at http_filters.c:1455 #4 0x00007f38b9d940b9 in ap_finalize_request_protocol (r=0x7f389c004980) at protocol.c:1242 #5 0x00007f38aa8b88ae in wsgi_hook_daemon_handler (c=<optimized out>) at src/server/mod_wsgi.c:11818 #6 wsgi_process_socket (daemon=0x7f38bb9213b8, bucket_alloc=<optimized out>, sock=<optimized out>, p=<optimized out>) at src/server/mod_wsgi.c:7793 #7 wsgi_daemon_worker (thread=<optimized out>, p=<optimized out>) at src/server/mod_wsgi.c:8122 #8 wsgi_daemon_thread (thd=<optimized out>, data=<optimized out>) at src/server/mod_wsgi.c:8211 #9 0x00007f38b881f52a in start_thread (arg=0x7f38a8e72700) at pthread_create.c:310 #10 0x00007f38b835777d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 (gdb) frame #0 apr_table_clear (t=0x0) at tables/apr_tables.c:467 467 t->a.nelts = 0; (gdb) list 462 } 463 } 464 465 APR_DECLARE(void) apr_table_clear(apr_table_t *t) 466 { 467 t->a.nelts = 0; 468 t->index_initialized = 0; 469 } 470 471 APR_DECLARE(const char *) apr_table_get(const apr_table_t *t, const char *key) Expected results: - The service is working without thread crash. Additional info: - Some request succeeds even when I see SIGSEGV. - Downgrading to mod_wsgi-4.3.2-1.fc21.x86_64.rpm helps.
Commit: http://pkgs.fedoraproject.org/gitweb/?p=mod_wsgi.git;a=commitdiff;h=9daf5a45ae08748a4e8aa1b3c08d7a0b30fa1cba Package: mod_wsgi-4.4.1-2.fc21 Build: https://koji.fedoraproject.org/koji/buildinfo?buildID=602050
Can you try this build? http://koji.fedoraproject.org/koji/taskinfo?taskID=8529329 I see a possible root cause with a recent change in httpd, but I'm not sure why downgrading mod_wsgi would help.
I still have Segmentation fault(11) issue with https://kojipkgs.fedoraproject.org//packages/mod_wsgi/4.4.1/2.fc21/x86_64/mod_wsgi-4.4.1-2.fc21.x86_64.rpm. I did not see this issue with the old: https://kojipkgs.fedoraproject.org//packages/mod_wsgi/4.3.2/1.fc21/x86_64/mod_wsgi-4.3.2-1.fc21.x86_64.rpm I have these packages installed: httpd-2.4.10-15.fc21.x86_64 apr-1.5.1-3.fc21.x86_64 apr-util-1.5.4-1.fc21.x86_64
The hello world app is failing with the following config: <VirtualHost *:5000> WSGIDaemonProcess hellow WSGIProcessGroup hellow WSGIScriptAlias / /var/www/hellow </VirtualHost> Without the WSGIProcessGroup directive I do not have the issue. <VirtualHost *:5000> WSGIDaemonProcess hellow # WSGIProcessGroup hellow WSGIScriptAlias / /var/www/hellow </VirtualHost> Both cases the I get the 'hello' response, but in the first case the service thread is crashing.
Does it produce the same backtrace with the https://kojipkgs.fedoraproject.org//packages/mod_wsgi/4.4.1/2.fc21/x86_64/mod_wsgi-4.4.1-2.fc21.x86_64.rpm as in the Comment 0?
If think we have found out what was wrong, can you please try with following build? https://kojipkgs.fedoraproject.org//work/tasks/9625/8539625/mod_wsgi-4.4.1-3.fc21.x86_64.rpm
https://kojipkgs.fedoraproject.org//work/tasks/9625/8539625/mod_wsgi-4.4.1-3.fc21.x86_64.rpm is working. Thank you.
mod_wsgi-4.4.1-3.fc21 has been submitted as an update for Fedora 21. https://admin.fedoraproject.org/updates/mod_wsgi-4.4.1-3.fc21
Package mod_wsgi-4.4.1-3.fc21: * should fix your issue, * was pushed to the Fedora 21 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing mod_wsgi-4.4.1-3.fc21' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2015-0364/mod_wsgi-4.4.1-3.fc21 then log in and leave karma (feedback).
Why wasn't this fix added to Rawhide? It crashes in the same way on current Rawhide & F22. For reference the upstream fix is: https://github.com/GrahamDumpleton/mod_wsgi/commit/808e9667fdddad16f94927b9f8ad947d56ea0071 What we have in Rawhide is a couple of months old so I'm going to update it to 4.4.8 which should include the above fix.
Thanks for the fix, Richard.
mod_wsgi-4.4.1-3.fc21 has been pushed to the Fedora 21 stable repository. If problems still persist, please make note of it in this bug report.