Bug 193633

Thanks for the report.  Can you try:

# yum install httpd-debuginfo openssl-debuginfo
# gdb --args /usr/sbin/httpd -X
...
(gdb) run
...
<segfault>
(gdb) bt full

and then attach the output of the "bt full" command to this bug report.

(gdb) bt full
#0  0x00ba573e in BN_BLINDING_free () from /lib/libcrypto.so.6
No symbol table info available.
#1  0x00bb120d in RSA_free () from /lib/libcrypto.so.6
No symbol table info available.
#2  0x00bd4a8d in EVP_PKEY_type () from /lib/libcrypto.so.6
No symbol table info available.
#3  0x00bd4b07 in EVP_PKEY_free () from /lib/libcrypto.so.6
No symbol table info available.
#4  0x00be0693 in d2i_X509_VAL () from /lib/libcrypto.so.6
No symbol table info available.
#5  0x00be761e in ASN1_primitive_free () from /lib/libcrypto.so.6
No symbol table info available.
#6  0x00be784b in ASN1_template_free () from /lib/libcrypto.so.6
No symbol table info available.
#7  0x00be7751 in ASN1_primitive_free () from /lib/libcrypto.so.6
No symbol table info available.
#8  0x00be784b in ASN1_template_free () from /lib/libcrypto.so.6
No symbol table info available.
#9  0x00be7751 in ASN1_primitive_free () from /lib/libcrypto.so.6
No symbol table info available.
#10 0x00be7893 in ASN1_item_free () from /lib/libcrypto.so.6
No symbol table info available.
#11 0x00be22d7 in X509_free () from /lib/libcrypto.so.6
No symbol table info available.
#12 0x005720d7 in ?? () from /etc/httpd/modules/mod_ssl.so
No symbol table info available.
#13 0x00739f0d in apr_pool_cleanup_run () from /usr/lib/libapr-1.so.0
No symbol table info available.
#14 0x0073a867 in apr_pool_clear () from /usr/lib/libapr-1.so.0
No symbol table info available.
#15 0x008077a1 in main () from /usr/sbin/httpd
No symbol table info available.
(gdb)                                                               

d'oh.. didn't have the devel repo enabled (apparently it's disabled by default)
here you go:

(gdb) run
Starting program: /usr/sbin/httpd -X
Reading symbols from shared object read from target memory...done.
Loaded system supplied DSO at 0xbfa000
[Thread debugging using libthread_db enabled]
[New Thread -1208400192 (LWP 7712)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1208400192 (LWP 7712)]
BN_BLINDING_free (r=0x11) at bn_blind.c:167
167             if (r->A  != NULL) BN_free(r->A );
(gdb) bt full
#0  BN_BLINDING_free (r=0x11) at bn_blind.c:167
No locals.
#1  0x0065720d in RSA_free (r=0x8382f08) at rsa_lib.c:236
        i = Variable "i" is not available.
(gdb)

Thanks.  I can't reproduce that here so it could be something special concerning
the created keypair.  If possible, could you attach both key and cert to this
bug report? (obviously then you should not use them on a public site!)

This is reproducible every time?  Can you confirm some package versions:

# rpm -q openssl crypto-utils

and whether this is i386, ppc, ...?

Can you also try to reproduce the backtrace like this:

# export MALLOC_CHECK_=3
# gdb --args /usr/sbin/httpd -X
...
(gdb) run
...
(gdb) bt

Created attachment 130383 [details]
the certificate

Created attachment 130384 [details]
the key

It happens every time. I haven't been able to get apache up with mod_ssl.

[root@home Desktop]# rpm -q openssl crypto-utils
openssl-0.9.8a-5.2
crypto-utils-2.2-9.2.1
[root@home Desktop]# uname -a
Linux home.gregmartyn.com 2.6.16-1.2111_FC5 #1 Thu May 4 21:16:58 EDT 2006 i686
athlon i386 GNU/Linux


(gdb) run
Starting program: /usr/sbin/httpd -X
malloc: using debugging hooks
Reading symbols from shared object read from target memory...done.
Loaded system supplied DSO at 0x420000
[Thread debugging using libthread_db enabled]
[New Thread -1208195392 (LWP 21886)]
malloc: using debugging hooks

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1208195392 (LWP 21886)]
BN_BLINDING_free (r=0xc9b19a65) at bn_blind.c:167
167             if (r->A  != NULL) BN_free(r->A );
(gdb) bt
#0  BN_BLINDING_free (r=0xc9b19a65) at bn_blind.c:167
#1  0x00ffc20d in RSA_free (r=0x9857660) at rsa_lib.c:236
#2  0x0101fa8d in EVP_PKEY_free_it (x=Variable "x" is not available.
) at p_lib.c:479
#3  0x0101fb07 in EVP_PKEY_free (x=0x9857640) at p_lib.c:466
#4  0x0102b693 in pubkey_cb (operation=3, pval=0x9856510, it=0x10af808) at
x_pubkey.c:76
#5  0x0103261e in asn1_item_combine_free (pval=0x9856510, it=0x10af808,
combine=0) at tasn_fre.c:175
#6  0x0103284b in ASN1_template_free (pval=0x9856510, tt=0x10b2418) at
tasn_fre.c:202
#7  0x01032751 in asn1_item_combine_free (pval=0x9855c30, it=0x10af9bc,
combine=0) at tasn_fre.c:172
#8  0x0103284b in ASN1_template_free (pval=0x9855c30, tt=0x10b2480) at
tasn_fre.c:202
#9  0x01032751 in asn1_item_combine_free (pval=0xbf8ed3c0, it=0x10af9d8,
combine=0) at tasn_fre.c:172
#10 0x01032893 in ASN1_item_free (val=0x9855c30, it=0x10af9d8) at tasn_fre.c:71
#11 0x0102d2d7 in X509_free (a=0x9855c30) at x_x509.c:128
#12 0x004670d7 in ssl_init_ModuleKill (data=0x971c170) at
/usr/src/debug/httpd-2.2.0/modules/ssl/ssl_engine_init.c:1233
#13 0x00e52f0d in run_cleanups (cref=0x97169c0) at memory/unix/apr_pools.c:2027
#14 0x00e53867 in apr_pool_clear (pool=0x97169b0) at memory/unix/apr_pools.c:689
#15 0x008457a1 in main (argc=158419496, argv=0x97e7f70) at
/usr/src/debug/httpd-2.2.0/server/main.c:662
(gdb)                                                                          
                                 

Thanks

What is the best way to reinstall everything that could be causing this without
having to reinstall the OS?

I tried restarting, then moved the /etc/pki folder over from a fresh install of
FC5. I ran apachectl start, and all was well. https://localhost popped up the
certificate warning dialogs, then showed a blank page. I hit refresh, and now
I'm back to httpd crashing when I try to start it. Bizarre.

I can configure using your cert/key pair without problems here.  Can you attach
the ssl.conf you use to configure the server?

I'm seeing exactly the same symptoms.

I'm building apache-2.2.2 from FC5 Development SRPM on CentOS 4.3.

I've attached my build procedure for reference.

I'm not even creating a key - just using the "default", and apache segfaults on
startup.

I ran it under gdb (with -e debug -X) and get this output:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1208994112 (LWP 25655)]
BN_BLINDING_free (r=0x11) at bn_blind.c:167
167             if (r->A  != NULL) BN_free(r->A );
(gdb) list
162     void BN_BLINDING_free(BN_BLINDING *r)
163             {
164             if(r == NULL)
165                 return;
166
167             if (r->A  != NULL) BN_free(r->A );
168             if (r->Ai != NULL) BN_free(r->Ai);
169             if (r->e  != NULL) BN_free(r->e );
170             OPENSSL_free(r);
171             }

A backtrace is attached also.

Created attachment 131584 [details]
GDB output including backtrace

Created attachment 131585 [details]
Steps used to build httpd-2.2.2 from Fedora Development SRPM

Update: I've repeated my build process but this time using
openssl-0.9.7f-7.10.src.rpm from FC4 updates and I no longer see segfaults.

So this appears to be a problem with openssl-0.9.8.

Robin, please try:

# LD_DEBUG_OUTPUT=/tmp/foo LD_DEBUG=files httpd -X

and upload the /tmp/foo.<pid> produced.

The same problem with me.

Both httpd-2.2.0-5.1.2 and httpd-2.2.2-1.0 are affected.
Under both kernel-2.6.16-1.2133 and kernel-2.6.17-1.2157

I made the test of comment #15, and strace log (see below).

Created attachment 132606 [details]
LD_DEBUG_OUTPUT of LD_DEBUG=files httpd -X

Created attachment 132607 [details]
strace httpd -X

Dmitry, please attach the output of:

# rpm -q httpd mod_ssl openssl openldap
# rpm -V httpd mod_ssl openssl openldap

are you running this a vanilla FC5 install?  Is it an upgrade?  What third-party
software do you have intstalled?

The bugzilla data loss hides the successful diagnosis of Greg Martyn's issue as
being caused by a Zimbra installation (which included replacement and seemingly
conflictling OpenLDAP libraries).

Comment 14 indicates a similar library issue.

> rpm -q httpd mod_ssl openssl openldap
...and I've understood the reason :)

It was an "newest-version" openldap package, 2.3.24, taked from rawhide and
compiled under FC3.
Then FC3 was upgraded to FC5. The problem appears...

After updating of the old custom FC3 openldap to either standard FC5 openldap or
FC5-compiled 2.3.24, the problem goes away.

I.e., some "old-library" issue.

I'd guess this is some OpenLDAP ABI break or something.  But in general
mis-matched library versions like that will undoubtedly cause problems and is
not supported -> marking as NOTABUG.