Bug 1074406

Summary: Apache 2.4.7 core dumps
Product: [Fedora] Fedora Reporter: Daniel Steiner <daniel.steiner>
Component: httpdAssignee: Jan Kaluža <jkaluza>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 20CC: amasniko, bgamari, jkaluza, jorton, pahan
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-08-20 07:14:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Daniel Steiner 2014-03-10 06:44:05 UTC
Description of problem:
After updating https to 2.4.7 version, Apache server did not start. I get this in the log (var/log/messages), Apache does not write anything into a log:
Mar  9 13:08:37 danweb kernel: [161242.620441] /usr/sbin/httpd[8335]: segfault at 0 ip 00007f07666cfa0a sp 00007fff2bd17b10 error 4 in mod_socache_shmcb.so[7f07666cd000+5000]
Mar  9 13:08:37 danweb kernel: /usr/sbin/httpd[8335]: segfault at 0 ip 00007f07666cfa0a sp 00007fff2bd17b10 error 4 in mod_socache_shmcb.so[7f07666cd000+5000]
Mar  9 13:08:37 danweb abrt-hook-ccpp: Saved core dump of pid 8335 (/usr/sbin/httpd) to /var/tmp/abrt/ccpp-2014-03-09-13:08:37-8335 (24924160 bytes)
Mar  9 13:08:37 danweb systemd: httpd.service: main process exited, code=dumped, status=11/SEGV




Version-Release number of selected component (if applicable):
httpd-2.4.7-3.fc20.x86_64
Module:
/usr/lib64/httpd/modules/mod_socache_shmcb.so

How reproducible:
I'm not sure, but this problem occurs only on my server. I have other fedora 20 installation, and on this installation, the problem does not appear.


Steps to Reproduce:
1. 
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Jan Kaluža 2014-03-10 09:47:23 UTC
I can't reproduce this crash.

Can you check if /usr/sbin/httpd and /usr/lib64/httpd/modules/mod_socache_shmcb.so has the same creation date? (using ls -l /usr/sbin/httpd .... for example). Maybe the installation failed for some reason and you ended up with older mod_socache_shmcb.so.


You can also try to get backtrace from the crash (install httpd-debuginfo package and follow gdb part of https://httpd.apache.org/dev/debugging.html ).

Comment 2 Daniel Steiner 2014-03-10 16:43:21 UTC
Hi,

Thank you for the quick feedback. I have investigated your questions:

After installation (upgrade) of the packages:
# rpm -V httpd httpd-tools httpd-manual httpd-devel mod_ssl
S.5..UGT.  c /etc/httpd/conf.d/welcome.conf
S.5..UGT.  c /etc/httpd/conf/httpd.conf
S.5....T.  c /etc/logrotate.d/httpd
S.5..UGT.  c /etc/httpd/conf.d/ssl.conf

As I understand, the packages are properly installed the changes listed are changes I made in the configuration.


I tried to get a backtrace using gdb, here is the output:
# gdb httpd
GNU gdb (GDB) Fedora 7.6.50.20130731-19.fc20
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
..
Reading symbols from /usr/sbin/httpd...Reading symbols from /usr/lib/debug/usr/sbin/httpd.debug...done.
done.
(gdb) b ap_process_request
Breakpoint 1 at 0x54d90: file http_request.c, line 357.
(gdb) run -X -d /etc/httpd
Starting program: /usr/sbin/httpd -X -d /etc/httpd
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7fffe33d3700 (LWP 4648)]
[Thread 0x7fffe33d3700 (LWP 4648) exited]

Program received signal SIGSEGV, Segmentation fault.
socache_shmcb_init (ctx=0x0, namespace=0x7ffff461e74c "authn-socache", hints=0x7ffff4820000 <authn_cache_hints.10695>, s=0x555555803f08, p=0x5555557d8138) at mod_socache_shmcb.c:344
344	    if (ctx->data_file == NULL) {
(gdb)

The server crashed immediately after starting, see above lines.

I hope, this helps

Notice: If I downgrade to 2.4.6 version of https, everything works fine...

Comment 3 Jan Kaluža 2014-03-11 07:41:03 UTC
Can you redo the backtrace, but execute "bt full" command in the gdb right after it prints what you pasted above and send the output? That would really help.

Comment 4 Jan Kaluža 2014-03-11 07:47:54 UTC
Do you have "AuthnCacheSOCache ..." in your httpd config file? Can you paste the part of config file where you configure mod_authn_socache?

http://httpd.apache.org/docs/2.4/mod/mod_authn_socache.html

Comment 5 Jan Kaluža 2014-03-11 08:55:49 UTC
Ok, I've been able to reproduce this issue, but still paste the AuthnCache related directives from your config file please.

The upstream commit to address the crash I was seeing: http://svn.apache.org/viewvc?view=revision&revision=1576233

Comment 6 Daniel Steiner 2014-03-11 15:45:35 UTC
Sorry for late response. 

Here, the search output for 'AuthnCacheProvideFor':

   11  		AuthName "Entry" 
   12  		AuthBasicProvider socache dbd
   13: 		AuthnCacheProvideFor dbd
   14  		AuthnCacheContext something
   15  		AuthDBDUserPWQuery "SELECT Password FROM DB WHERE UserID = %s AND sdfa = 1 AND dkas = 1" 

23 matches across 20 files


I found 23 matches AuthnCacheProvedFor entries, see pasted search output (part off).

I searched for the AuthnCachSOCache pattern in all my configs:
Searching 108 files for "AuthnCacheSOCache"

0 matches across 0 files

"AuthnCacheSOCache" was not defined in any configuration file!

I added the following directive into apache config, and this resolved the problem:
AuthnCacheSOCache dbm

Thank you very much for your help.

Comment 7 Daniel Steiner 2014-03-11 15:47:51 UTC
Do you need the output of gdb debugger? Or are the information enough I pasted in comment 6?

Let me know.

Comment 8 Jan Kaluža 2014-03-12 07:02:45 UTC
No, I have fixed the bug already in upstream repository :).

Comment 9 Marko Vukovic 2014-03-30 23:07:42 UTC
I am getting a different crash after installing mod_ssl (maybe I should have opened another defect?). I have removed everything I put on this web server, I think the configuration is now identical to what it looked like "out of the box", including the default self signed certificate from the default configuration. I tried adding AuthnCacheSOCache option but it did not make a difference, which is not surprising because I do not have any authentication configured anyway. Here is what I see:


[root@hd httpd]# gdb httpd
GNU gdb (GDB) Fedora 7.6.50.20130731-19.fc20
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
..
Reading symbols from /usr/sbin/httpd...Reading symbols from /usr/lib/debug/usr/sbin/httpd.debug...done.
done.
(gdb) run -X -d /etc/httpd
Starting program: /usr/sbin/httpd -X -d /etc/httpd
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7f953b567700 (LWP 10551)]
[Thread 0x7f953b567700 (LWP 10551) exited]

Program received signal SIGSEGV, Segmentation fault.
0x00007f953eb98e00 in ?? ()
Missing separate debuginfos, use: debuginfo-install nss-softokn-freebl-3.15.5-2.fc20.x86_64
(gdb) bt full
#0  0x00007f953eb98e00 in ?? ()
No symbol table info available.
#1  0x00007f953e58c8c6 in int_free_ex_data (class_index=<optimized out>, obj=0x7f954ce1abf0, ad=0x7f954ce1ac18)
    at ex_data.c:522
        mx = 1
        i = 0
        item = 0x7f954cee1ec0
        ptr = <optimized out>
        storage = 0x7f954ce56230
#2  0x00007f953e621551 in x509_cb (operation=operation@entry=3, pval=pval@entry=0x7fff7dbb2e98, 
    it=it@entry=0x7f953e8e99e0 <X509_it>, exarg=exarg@entry=0x0) at x_x509.c:113
        ret = 0x7f954ce1abf0
#3  0x00007f953e62555a in asn1_item_combine_free (pval=pval@entry=0x7fff7dbb2e98, 
    it=it@entry=0x7f953e8e99e0 <X509_it>, combine=combine@entry=0) at tasn_fre.c:173
        tt = <optimized out>
        seqtt = <optimized out>
        ef = <optimized out>
        cf = <optimized out>
        aux = <optimized out>
        asn1_cb = 0x7f953e621470 <x509_cb>
        i = <optimized out>
#4  0x00007f953e625735 in ASN1_item_free (val=val@entry=0x7f954ce1abf0, it=it@entry=0x7f953e8e99e0 <X509_it>)
    at tasn_fre.c:71
No locals.
#5  0x00007f953e62165c in X509_free (a=a@entry=0x7f954ce1abf0) at x_x509.c:141
No locals.
#6  0x00007f953d35d318 in ssl_pphrase_Handle (s=s@entry=0x7f954ce06340, p=p@entry=0x7f954ce08368)
    at ssl_engine_pphrase.c:275
        using_cache = 0
        mc = 0x7f954ce01b58
        sc = 0x7f954ceb4880
        pServ = 0x7f954ceae698
        cpVHostID = 0x7f954ceb1df0 "127.0.0.1:443"
        szPath = "/etc/pki/tls/certs/localhost.crt\000/\273}\377\177\000\000\351\350\352I\225\177\000\000\000\000\000\000\000\000\000\000\262\210\rJ\225\177\000\000p\362\355L\225\177\000\000X\321\335L\225\177\000\000\330/\273}\377\177\000\000*\033\352C\225\177\000\000o/quicktime\000\000ion\350\034\353L\225\177\000\000\371/\273}\377\177\000\000\000-epoc/x-sisx-app\t\t\t\tsisx\000\000ice\000\000\000\000nifest\000\000d\000\000\070\000\000a\000\000\060 conf log\000\000nature+xm"...
        pPrivateKey = <optimized out>
        asn1 = <optimized out>
        ucp = 0x7f954cee1996 "R2a\002"
        length = <optimized out>
        pX509Cert = 0x7f954ce1abf0
        bReadable = <optimized out>
---Type <return> to continue, or q <return> to quit---
        aPassPhrase = 0x7f954ceb1db0
        nPassPhrase = 0
        nPassPhraseCur = 8192
        cpPassPhraseCur = 0x7f954ce06340 "8\262\335L\225\177"
        nPassPhraseRetry = <optimized out>
        nPassPhraseDialog = 0
        nPassPhraseDialogCur = 2109419488
        bPassPhraseDialogOnce = 32767
        cpp = <optimized out>
        i = 0
        j = 0
        algoCert = 1
        algoKey = 0
        at = <optimized out>
        an = 0x7f953d36c7a2 "RSA"
        pkey_mtime = 0
        rv = <optimized out>
#7  0x00007f953d352666 in ssl_init_Module (p=0x7f954cddd158, plog=<optimized out>, ptemp=0x7f954ce08368, 
    base_server=0x7f954ce06340) at ssl_engine_init.c:197
        mc = 0x7f954ce01b58
        sc = 0x7f954ceb4880
        s = 0x0
#8  0x00007f954b447629 in ap_run_post_config (pconf=0x7f954cddd158, plog=0x7f954ce0a378, ptemp=0x7f954ce08368, 
    s=0x7f954ce06340) at config.c:103
        pHook = 0x7f954ceb7a18
        n = 25
        rv = 0
#9  0x00007f954b427547 in main (argc=4, argv=0x7fff7dbb5228) at main.c:765
        c = 100 'd'
        showcompile = 0
        showdirectives = 0
        confname = 0x7f954b46368f "conf/httpd.conf"
        def_server_root = 0x7fff7dbb581b "/etc/httpd"
        temp_error_log = 0x0
        error = <optimized out>
        process = 0x7f954cddb238
        pconf = 0x7f954cddd158
        plog = 0x7f954ce0a378
        ptemp = 0x7f954ce08368
        pcommands = 0x7f954cdff268
        opt = 0x7f954cdff358
        rv = <optimized out>
        mod = 0x7f954b681098 <ap_prelinked_modules+24>
        opt_arg = 0x7fff7dbb581b "/etc/httpd"
---Type <return> to continue, or q <return> to quit---
        signal_server = <optimized out>
(gdb) 

If there is anything else I can provide, please let me know.

Comment 10 Marko Vukovic 2014-04-03 00:04:36 UTC
Upgrading to packages httpd-2.4.9-2.fc20.x86_64.rpm and mod_ssl-2.4.9-2.fc20.x86_64.rpm resolved this problem.

Comment 11 Fedora Admin XMLRPC Client 2014-06-30 09:54:08 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 12 Jan Kaluža 2014-08-20 07:14:37 UTC
All the bugs reported as part of this bug in bugzilla should be fixed in httpd-2.4.10 in Fedora 20.