Bug 1414258 - Crash during restart or at startup in mod_ssl, in certinfo_free() function registered by ssl_stapling_ex_init()
Summary: Crash during restart or at startup in mod_ssl, in certinfo_free() function re...
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: httpd
Version: 7.2
Hardware: Unspecified
OS: Unspecified
Target Milestone: rc
: ---
Assignee: Luboš Uhliarik
QA Contact: Jan Houska
: 1456852 (view as bug list)
Depends On:
Blocks: 1420851 1465904
TreeView+ depends on / blocked
Reported: 2017-01-18 07:57 UTC by Hisanobu Okuda
Modified: 2020-12-14 08:01 UTC (History)
6 users (show)

Fixed In Version: httpd-2.4.6-61.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2017-08-01 21:36:44 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Apache Bugzilla 54357 0 None None None 2017-01-18 07:58:30 UTC
Red Hat Product Errata RHBA-2017:2175 0 normal SHIPPED_LIVE httpd bug fix update 2017-08-01 18:40:47 UTC

Description Hisanobu Okuda 2017-01-18 07:57:48 UTC
Description of problem:
This is a clone of https://bz.apache.org/bugzilla/show_bug.cgi?id=54357

Original Description:
Reason for crash: the stapling code in mod_ssl implements a free_func which it registers in OpenSSL as a function pointer. After restart, the function address registered during the original startup is still used, because the OpenSSL libs were not reloaded, but due to a different module load order, mod_ssl.so was loaded at another address. The function pointer points to memory now used for other stuff and calling the function there crashes.

It seems in my config the OpenSSL libs don't get reloaded during restart, so still hold the original reference.


#0  0xfee71920 in ?? ()
#1  0xfec4c91c in int_free_ex_data (class_index=<optimized out>, obj=0x1cfc48, ad=0x1cfc60) at ex_data.c:522
#2  0xfec4c65c in CRYPTO_free_ex_data (class_index=class_index@entry=10, obj=obj@entry=0x1cfc48, ad=ad@entry=0x1cfc60) at ex_data.c:592
#3  0xfeced474 in x509_cb (operation=operation@entry=3, pval=pval@entry=0xffbfd68c, it=it@entry=0xfed9c358, exarg=exarg@entry=0x0) at x_x509.c:113
#4  0xfecf2a30 in asn1_item_combine_free (pval=0xffbfd68c, it=0xfed9c358, combine=0) at tasn_fre.c:173
#5  0xfecf2c38 in ASN1_item_free (val=0x1cfc48, it=0xfed9c358) at tasn_fre.c:71
#6  0xfe9ccf74 in ssl_pphrase_Handle (s=0xb8ee0, p=0xf3e08) at ssl_engine_pphrase.c:295
#7  0xfe9c13cc in ssl_init_Module (p=0x93c48, plog=<optimized out>, ptemp=0xf3e08, base_server=0xb8ee0) at ssl_engine_init.c:368
#8  0x0004ae0c in ap_run_post_config (pconf=pconf@entry=0x93c48, plog=0xeabd8, ptemp=0xf3e08, s=0xb8ee0) at config.c:105
#9  0x00065b98 in main (argc=5, argv=0xffbff90c) at main.c:765

The address 0xfee71920 after the restart lies between two memory segments used by the prefork MPM:

FEE60000      24K r-x--  /shared/build/autobuild/install/apache24-2.4.4_dev_r1422615_1.0.1csp1-X3.solaris10.sparc/modules/mod_mpm_prefork.so
FEE74000      16K rwx--  /shared/build/autobuild/install/apache24-2.4.4_dev_r1422615_1.0.1csp1-X3.solaris10.sparc/modules/mod_mpm_prefork.so

Directly after the original apache start it is in the region occupied by mod_ssl:

FEE50000     192K r-x--  /shared/build/autobuild/install/apache24-2.4.4_dev_r1422615_1.0.1csp1-X3.solaris10.sparc/modules/mod_ssl.so
FEE8E000      16K rwx--  /shared/build/autobuild/install/apache24-2.4.4_dev_r1422615_1.0.1csp1-X3.solaris10.sparc/modules/mod_ssl.so

The problematic code is in modules/ssl/ssl_util_stapling.c (trunk and 2.4):

 68 static int stapling_ex_idx = -1;
 70 void ssl_stapling_ex_init(void)
 71 {
 72     if (stapling_ex_idx != -1)
 73         return;
 74     stapling_ex_idx = X509_get_ex_new_index(0, "X509 cached OCSP info", 0, 0,
 75                                             certinfo_free);
 76 }

It registers the function certinfo_free in OpenSSL via a function pointer. The function is implemented in mod_ssl (ssl_util_stapling.c) which can be loaded into another adress range after restart (when linked dynamically). Then the pointer is no longer valid:

AH00060: seg fault or similar nasty error detected in the parent process

Currently I don't know how o fix that:

- there seems to be no API to remove an index once registered.
- there seems to be no API to set the free_func to a new value for an existing index (maybe I didn't look thorougly enough)

Note that furthermore there's a slight order problem in ssl_init_Module: it first calls ssl_pphrase_Handle(), then ssl_stapling_ex_init(). After restart, the call to ssl_pphrase_Handle() already triggers the problem before the next ssl_stapling_ex_init() could fix it.

Modules and libs are build as DSOs, but it seems only the modules get reloaded but not the SSL libs.

Reproducible by running start-restart-stop in a loop. No requests need to be send. Another scenario is adding modules to the LoadModule list before restart.

Config used (a more complex config makes the crash happen more frequently):

Define base "/path/to/my/apache/installation"
Define user "myuser"
Define group "mygroup"
Define sslport "8443"
Define cert "snakeoil-rsa"

ServerRoot "${base}"
ServerName "www.snakeoil.dom"
Listen ${sslport}

LoadModule socache_shmcb_module modules/mod_socache_shmcb.so
LoadModule slotmem_shm_module modules/mod_slotmem_shm.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule env_module modules/mod_env.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule session_module modules/mod_session.so
LoadModule session_crypto_module modules/mod_session_crypto.so
LoadModule ssl_module modules/mod_ssl.so
LoadModule mpm_prefork_module modules/mod_mpm_prefork.so
LoadModule unixd_module modules/mod_unixd.so

<IfModule unixd_module>
  User ${user}
  Group ${group}

<IfModule ssl_module>
  SSLPassPhraseDialog  builtin
  SSLSessionCache        "shmcb:run/ssl_scache.base(512000)"
  SSLSessionCacheTimeout  300
  SSLRandomSeed startup builtin
  SSLRandomSeed connect builtin

<VirtualHost _default_:${sslport}>
  SSLEngine on
  SSLCertificateFile "conf/ssl.crt/${cert}.crt"
  SSLCertificateKeyFile "conf/ssl.key/${cert}.key"

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

Actual results:

Expected results:

Additional info:

Comment 9 Jan Houska 2017-06-16 15:39:04 UTC
Verified -- sanity only.

It was not possible to reproduce bug in our environment. Patch seems ok.

Comment 11 Joe Orton 2017-07-13 16:40:11 UTC
*** Bug 1456852 has been marked as a duplicate of this bug. ***

Comment 15 errata-xmlrpc 2017-08-01 21:36:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.