Bug 1414258

Description of problem:
This is a clone of https://bz.apache.org/bugzilla/show_bug.cgi?id=54357

Original Description:
Reason for crash: the stapling code in mod_ssl implements a free_func which it registers in OpenSSL as a function pointer. After restart, the function address registered during the original startup is still used, because the OpenSSL libs were not reloaded, but due to a different module load order, mod_ssl.so was loaded at another address. The function pointer points to memory now used for other stuff and calling the function there crashes.

It seems in my config the OpenSSL libs don't get reloaded during restart, so still hold the original reference.

Stack:

#0  0xfee71920 in ?? ()
#1  0xfec4c91c in int_free_ex_data (class_index=<optimized out>, obj=0x1cfc48, ad=0x1cfc60) at ex_data.c:522
#2  0xfec4c65c in CRYPTO_free_ex_data (class_index=class_index@entry=10, obj=obj@entry=0x1cfc48, ad=ad@entry=0x1cfc60) at ex_data.c:592
#3  0xfeced474 in x509_cb (operation=operation@entry=3, pval=pval@entry=0xffbfd68c, it=it@entry=0xfed9c358, exarg=exarg@entry=0x0) at x_x509.c:113
#4  0xfecf2a30 in asn1_item_combine_free (pval=0xffbfd68c, it=0xfed9c358, combine=0) at tasn_fre.c:173
#5  0xfecf2c38 in ASN1_item_free (val=0x1cfc48, it=0xfed9c358) at tasn_fre.c:71
#6  0xfe9ccf74 in ssl_pphrase_Handle (s=0xb8ee0, p=0xf3e08) at ssl_engine_pphrase.c:295
#7  0xfe9c13cc in ssl_init_Module (p=0x93c48, plog=<optimized out>, ptemp=0xf3e08, base_server=0xb8ee0) at ssl_engine_init.c:368
#8  0x0004ae0c in ap_run_post_config (pconf=pconf@entry=0x93c48, plog=0xeabd8, ptemp=0xf3e08, s=0xb8ee0) at config.c:105
#9  0x00065b98 in main (argc=5, argv=0xffbff90c) at main.c:765

The address 0xfee71920 after the restart lies between two memory segments used by the prefork MPM:

FEE60000      24K r-x--  /shared/build/autobuild/install/apache24-2.4.4_dev_r1422615_1.0.1csp1-X3.solaris10.sparc/modules/mod_mpm_prefork.so
FEE74000      16K rwx--  /shared/build/autobuild/install/apache24-2.4.4_dev_r1422615_1.0.1csp1-X3.solaris10.sparc/modules/mod_mpm_prefork.so


Directly after the original apache start it is in the region occupied by mod_ssl:

FEE50000     192K r-x--  /shared/build/autobuild/install/apache24-2.4.4_dev_r1422615_1.0.1csp1-X3.solaris10.sparc/modules/mod_ssl.so
FEE8E000      16K rwx--  /shared/build/autobuild/install/apache24-2.4.4_dev_r1422615_1.0.1csp1-X3.solaris10.sparc/modules/mod_ssl.so

The problematic code is in modules/ssl/ssl_util_stapling.c (trunk and 2.4):

 68 static int stapling_ex_idx = -1;
 69
 70 void ssl_stapling_ex_init(void)
 71 {
 72     if (stapling_ex_idx != -1)
 73         return;
 74     stapling_ex_idx = X509_get_ex_new_index(0, "X509 cached OCSP info", 0, 0,
 75                                             certinfo_free);
 76 }

It registers the function certinfo_free in OpenSSL via a function pointer. The function is implemented in mod_ssl (ssl_util_stapling.c) which can be loaded into another adress range after restart (when linked dynamically). Then the pointer is no longer valid:

AH00060: seg fault or similar nasty error detected in the parent process

Currently I don't know how o fix that:

- there seems to be no API to remove an index once registered.
- there seems to be no API to set the free_func to a new value for an existing index (maybe I didn't look thorougly enough)

Note that furthermore there's a slight order problem in ssl_init_Module: it first calls ssl_pphrase_Handle(), then ssl_stapling_ex_init(). After restart, the call to ssl_pphrase_Handle() already triggers the problem before the next ssl_stapling_ex_init() could fix it.

Modules and libs are build as DSOs, but it seems only the modules get reloaded but not the SSL libs.

Reproducible by running start-restart-stop in a loop. No requests need to be send. Another scenario is adding modules to the LoadModule list before restart.

Config used (a more complex config makes the crash happen more frequently):

Define base "/path/to/my/apache/installation"
Define user "myuser"
Define group "mygroup"
Define sslport "8443"
Define cert "snakeoil-rsa"

ServerRoot "${base}"
ServerName "www.snakeoil.dom"
Listen ${sslport}

LoadModule socache_shmcb_module modules/mod_socache_shmcb.so
LoadModule slotmem_shm_module modules/mod_slotmem_shm.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule env_module modules/mod_env.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule session_module modules/mod_session.so
LoadModule session_crypto_module modules/mod_session_crypto.so
LoadModule ssl_module modules/mod_ssl.so
LoadModule mpm_prefork_module modules/mod_mpm_prefork.so
LoadModule unixd_module modules/mod_unixd.so

<IfModule unixd_module>
  User ${user}
  Group ${group}
</IfModule>

<IfModule ssl_module>
  SSLPassPhraseDialog  builtin
  SSLSessionCache        "shmcb:run/ssl_scache.base(512000)"
  SSLSessionCacheTimeout  300
  SSLRandomSeed startup builtin
  SSLRandomSeed connect builtin
</IfModule>

<VirtualHost _default_:${sslport}>
  SSLEngine on
  SSLCertificateFile "conf/ssl.crt/${cert}.crt"
  SSLCertificateKeyFile "conf/ssl.key/${cert}.key"
</VirtualHost>

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info: