Bug 193633 - mod_ssl causes httpd to crash at startup when using genkey cert
mod_ssl causes httpd to crash at startup when using genkey cert
Status: CLOSED NOTABUG
Product: Fedora
Classification: Fedora
Component: httpd (Show other bugs)
5
All Linux
medium Severity medium
: ---
: ---
Assigned To: Joe Orton
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2006-05-31 03:37 EDT by Greg Martyn
Modified: 2007-11-30 17:11 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2006-09-11 09:29:15 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
the certificate (1.13 KB, application/x-x509-ca-cert)
2006-06-02 04:15 EDT, Greg Martyn
no flags Details
the key (887 bytes, application/octet-stream)
2006-06-02 04:16 EDT, Greg Martyn
no flags Details
GDB output including backtrace (6.60 KB, text/plain)
2006-06-26 23:17 EDT, Robin Bowes
no flags Details
Steps used to build httpd-2.2.2 from Fedora Development SRPM (2.88 KB, text/plain)
2006-06-26 23:19 EDT, Robin Bowes
no flags Details
LD_DEBUG_OUTPUT of LD_DEBUG=files httpd -X (46.87 KB, text/plain)
2006-07-18 11:47 EDT, Dmitry Butskoy
no flags Details
strace httpd -X (11.46 KB, application/octet-stream)
2006-07-18 11:55 EDT, Dmitry Butskoy
no flags Details

  None (edit)
Description Greg Martyn 2006-05-31 03:37:33 EDT
Description of problem:
[root@home conf.d]# apachectl start
/usr/sbin/apachectl: line 102: 26812 Segmentation fault      $HTTPD $OPTIONS -k
$ARGV
[root@home conf.d]#

Version-Release number of selected component (if applicable):
Name   : mod_ssl
Arch   : i386
Epoch  : 1
Version: 2.2.0
Release: 5.1.2
Size   : 172 k


How reproducible:
Always


Steps to Reproduce:
yum -y install mod_ssl
genkey --days 730 home.gregmartyn.com
[Next]
Choose 1024
[Next]
send a Certificate Request (CSR)?
[No]
Fill out info
[Next]
Don't encrypt the key
[Next]

(Done with genkey)

Edit /etc/httpd/conf.d/ssl.conf:
change only these two values:
SSLCertificateFile /etc/pki/tls/certs/home.gregmartyn.com.cert
SSLCertificateKeyFile /etc/pki/tls/private/home.gregmartyn.com.key

apachectl start
/usr/sbin/apachectl: line 102: 26812 Segmentation fault      $HTTPD $OPTIONS -k
$ARGV

Actual results:
[root@home private]# apachectl start
/usr/sbin/apachectl: line 102: 32495 Segmentation fault      $HTTPD $OPTIONS -k
$ARGV

Expected results:
[root@home private]# apachectl start
[root@home private]# 

Additional info:
Note that the number after "line 102:" always changes

If I uninstall mod_ssl, the problem disappears.

Generating the key is new territory for me. I'm expecting this to be something
that I'm doing wrong, but in that case a better error message should be given.
If I am doing something wrong, where is the best place for me to find out how to
create my own certificate?

I'll respond promptly to any requests for additional information.

thanks
Comment 1 Joe Orton 2006-06-01 05:46:06 EDT
Thanks for the report.  Can you try:

# yum install httpd-debuginfo openssl-debuginfo
# gdb --args /usr/sbin/httpd -X
...
(gdb) run
...
<segfault>
(gdb) bt full

and then attach the output of the "bt full" command to this bug report.
Comment 2 Greg Martyn 2006-06-01 14:11:26 EDT
(gdb) bt full
#0  0x00ba573e in BN_BLINDING_free () from /lib/libcrypto.so.6
No symbol table info available.
#1  0x00bb120d in RSA_free () from /lib/libcrypto.so.6
No symbol table info available.
#2  0x00bd4a8d in EVP_PKEY_type () from /lib/libcrypto.so.6
No symbol table info available.
#3  0x00bd4b07 in EVP_PKEY_free () from /lib/libcrypto.so.6
No symbol table info available.
#4  0x00be0693 in d2i_X509_VAL () from /lib/libcrypto.so.6
No symbol table info available.
#5  0x00be761e in ASN1_primitive_free () from /lib/libcrypto.so.6
No symbol table info available.
#6  0x00be784b in ASN1_template_free () from /lib/libcrypto.so.6
No symbol table info available.
#7  0x00be7751 in ASN1_primitive_free () from /lib/libcrypto.so.6
No symbol table info available.
#8  0x00be784b in ASN1_template_free () from /lib/libcrypto.so.6
No symbol table info available.
#9  0x00be7751 in ASN1_primitive_free () from /lib/libcrypto.so.6
No symbol table info available.
#10 0x00be7893 in ASN1_item_free () from /lib/libcrypto.so.6
No symbol table info available.
#11 0x00be22d7 in X509_free () from /lib/libcrypto.so.6
No symbol table info available.
#12 0x005720d7 in ?? () from /etc/httpd/modules/mod_ssl.so
No symbol table info available.
#13 0x00739f0d in apr_pool_cleanup_run () from /usr/lib/libapr-1.so.0
No symbol table info available.
#14 0x0073a867 in apr_pool_clear () from /usr/lib/libapr-1.so.0
No symbol table info available.
#15 0x008077a1 in main () from /usr/sbin/httpd
No symbol table info available.
(gdb)                                                               
Comment 3 Greg Martyn 2006-06-01 14:28:46 EDT
d'oh.. didn't have the devel repo enabled (apparently it's disabled by default)
here you go:

(gdb) run
Starting program: /usr/sbin/httpd -X
Reading symbols from shared object read from target memory...done.
Loaded system supplied DSO at 0xbfa000
[Thread debugging using libthread_db enabled]
[New Thread -1208400192 (LWP 7712)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1208400192 (LWP 7712)]
BN_BLINDING_free (r=0x11) at bn_blind.c:167
167             if (r->A  != NULL) BN_free(r->A );
(gdb) bt full
#0  BN_BLINDING_free (r=0x11) at bn_blind.c:167
No locals.
#1  0x0065720d in RSA_free (r=0x8382f08) at rsa_lib.c:236
        i = Variable "i" is not available.
(gdb)
Comment 4 Joe Orton 2006-06-02 03:49:46 EDT
Thanks.  I can't reproduce that here so it could be something special concerning
the created keypair.  If possible, could you attach both key and cert to this
bug report? (obviously then you should not use them on a public site!)

This is reproducible every time?  Can you confirm some package versions:

# rpm -q openssl crypto-utils

and whether this is i386, ppc, ...?

Can you also try to reproduce the backtrace like this:

# export MALLOC_CHECK_=3
# gdb --args /usr/sbin/httpd -X
...
(gdb) run
...
(gdb) bt
Comment 5 Greg Martyn 2006-06-02 04:15:46 EDT
Created attachment 130383 [details]
the certificate
Comment 6 Greg Martyn 2006-06-02 04:16:28 EDT
Created attachment 130384 [details]
the key
Comment 7 Greg Martyn 2006-06-02 04:19:51 EDT
It happens every time. I haven't been able to get apache up with mod_ssl.

[root@home Desktop]# rpm -q openssl crypto-utils
openssl-0.9.8a-5.2
crypto-utils-2.2-9.2.1
[root@home Desktop]# uname -a
Linux home.gregmartyn.com 2.6.16-1.2111_FC5 #1 Thu May 4 21:16:58 EDT 2006 i686
athlon i386 GNU/Linux


(gdb) run
Starting program: /usr/sbin/httpd -X
malloc: using debugging hooks
Reading symbols from shared object read from target memory...done.
Loaded system supplied DSO at 0x420000
[Thread debugging using libthread_db enabled]
[New Thread -1208195392 (LWP 21886)]
malloc: using debugging hooks

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1208195392 (LWP 21886)]
BN_BLINDING_free (r=0xc9b19a65) at bn_blind.c:167
167             if (r->A  != NULL) BN_free(r->A );
(gdb) bt
#0  BN_BLINDING_free (r=0xc9b19a65) at bn_blind.c:167
#1  0x00ffc20d in RSA_free (r=0x9857660) at rsa_lib.c:236
#2  0x0101fa8d in EVP_PKEY_free_it (x=Variable "x" is not available.
) at p_lib.c:479
#3  0x0101fb07 in EVP_PKEY_free (x=0x9857640) at p_lib.c:466
#4  0x0102b693 in pubkey_cb (operation=3, pval=0x9856510, it=0x10af808) at
x_pubkey.c:76
#5  0x0103261e in asn1_item_combine_free (pval=0x9856510, it=0x10af808,
combine=0) at tasn_fre.c:175
#6  0x0103284b in ASN1_template_free (pval=0x9856510, tt=0x10b2418) at
tasn_fre.c:202
#7  0x01032751 in asn1_item_combine_free (pval=0x9855c30, it=0x10af9bc,
combine=0) at tasn_fre.c:172
#8  0x0103284b in ASN1_template_free (pval=0x9855c30, tt=0x10b2480) at
tasn_fre.c:202
#9  0x01032751 in asn1_item_combine_free (pval=0xbf8ed3c0, it=0x10af9d8,
combine=0) at tasn_fre.c:172
#10 0x01032893 in ASN1_item_free (val=0x9855c30, it=0x10af9d8) at tasn_fre.c:71
#11 0x0102d2d7 in X509_free (a=0x9855c30) at x_x509.c:128
#12 0x004670d7 in ssl_init_ModuleKill (data=0x971c170) at
/usr/src/debug/httpd-2.2.0/modules/ssl/ssl_engine_init.c:1233
#13 0x00e52f0d in run_cleanups (cref=0x97169c0) at memory/unix/apr_pools.c:2027
#14 0x00e53867 in apr_pool_clear (pool=0x97169b0) at memory/unix/apr_pools.c:689
#15 0x008457a1 in main (argc=158419496, argv=0x97e7f70) at
/usr/src/debug/httpd-2.2.0/server/main.c:662
(gdb)                                                                          
                                 

Thanks
Comment 8 Greg Martyn 2006-06-07 23:55:11 EDT
What is the best way to reinstall everything that could be causing this without
having to reinstall the OS?
Comment 9 Greg Martyn 2006-06-08 02:29:52 EDT
I tried restarting, then moved the /etc/pki folder over from a fresh install of
FC5. I ran apachectl start, and all was well. https://localhost popped up the
certificate warning dialogs, then showed a blank page. I hit refresh, and now
I'm back to httpd crashing when I try to start it. Bizarre.
Comment 10 Joe Orton 2006-06-08 09:24:08 EDT
I can configure using your cert/key pair without problems here.  Can you attach
the ssl.conf you use to configure the server?
Comment 11 Robin Bowes 2006-06-26 23:14:48 EDT
I'm seeing exactly the same symptoms.

I'm building apache-2.2.2 from FC5 Development SRPM on CentOS 4.3.

I've attached my build procedure for reference.

I'm not even creating a key - just using the "default", and apache segfaults on
startup.

I ran it under gdb (with -e debug -X) and get this output:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1208994112 (LWP 25655)]
BN_BLINDING_free (r=0x11) at bn_blind.c:167
167             if (r->A  != NULL) BN_free(r->A );
(gdb) list
162     void BN_BLINDING_free(BN_BLINDING *r)
163             {
164             if(r == NULL)
165                 return;
166
167             if (r->A  != NULL) BN_free(r->A );
168             if (r->Ai != NULL) BN_free(r->Ai);
169             if (r->e  != NULL) BN_free(r->e );
170             OPENSSL_free(r);
171             }

A backtrace is attached also.
Comment 12 Robin Bowes 2006-06-26 23:17:20 EDT
Created attachment 131584 [details]
GDB output including backtrace
Comment 13 Robin Bowes 2006-06-26 23:19:35 EDT
Created attachment 131585 [details]
Steps used to build httpd-2.2.2 from Fedora Development SRPM
Comment 14 Robin Bowes 2006-06-27 00:35:49 EDT
Update: I've repeated my build process but this time using
openssl-0.9.7f-7.10.src.rpm from FC4 updates and I no longer see segfaults.

So this appears to be a problem with openssl-0.9.8.
Comment 15 Joe Orton 2006-06-27 03:40:58 EDT
Robin, please try:

# LD_DEBUG_OUTPUT=/tmp/foo LD_DEBUG=files httpd -X

and upload the /tmp/foo.<pid> produced.
Comment 16 Dmitry Butskoy 2006-07-18 11:45:38 EDT
The same problem with me.

Both httpd-2.2.0-5.1.2 and httpd-2.2.2-1.0 are affected.
Under both kernel-2.6.16-1.2133 and kernel-2.6.17-1.2157

I made the test of comment #15, and strace log (see below).

Comment 17 Dmitry Butskoy 2006-07-18 11:47:14 EDT
Created attachment 132606 [details]
LD_DEBUG_OUTPUT of LD_DEBUG=files httpd -X
Comment 18 Dmitry Butskoy 2006-07-18 11:55:00 EDT
Created attachment 132607 [details]
strace httpd -X
Comment 19 Joe Orton 2006-07-18 12:33:55 EDT
Dmitry, please attach the output of:

# rpm -q httpd mod_ssl openssl openldap
# rpm -V httpd mod_ssl openssl openldap

are you running this a vanilla FC5 install?  Is it an upgrade?  What third-party
software do you have intstalled?

The bugzilla data loss hides the successful diagnosis of Greg Martyn's issue as
being caused by a Zimbra installation (which included replacement and seemingly
conflictling OpenLDAP libraries).

Comment 14 indicates a similar library issue.
Comment 20 Dmitry Butskoy 2006-07-18 12:58:26 EDT
> rpm -q httpd mod_ssl openssl openldap
...and I've understood the reason :)

It was an "newest-version" openldap package, 2.3.24, taked from rawhide and
compiled under FC3.
Then FC3 was upgraded to FC5. The problem appears...

After updating of the old custom FC3 openldap to either standard FC5 openldap or
FC5-compiled 2.3.24, the problem goes away.

I.e., some "old-library" issue.
Comment 21 Joe Orton 2006-09-11 09:29:15 EDT
I'd guess this is some OpenLDAP ABI break or something.  But in general
mis-matched library versions like that will undoubtedly cause problems and is
not supported -> marking as NOTABUG.

Note You need to log in before you can comment on or make changes to this bug.