This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours
Bug 546707 - (CVE-2009-4355) CVE-2009-4355 openssl significant memory leak in certain SSLv3 requests (DoS)
CVE-2009-4355 openssl significant memory leak in certain SSLv3 requests (DoS)
Status: CLOSED ERRATA
Product: Security Response
Classification: Other
Component: vulnerability (Show other bugs)
unspecified
All Linux
low Severity high
: ---
: ---
Assigned To: Red Hat Product Security
impact=moderate,source=researcher,pub...
: Security
Depends On: 555167 555168 555170 1127896
Blocks:
  Show dependency treegraph
 
Reported: 2009-12-11 13:25 EST by Michael K Johnson
Modified: 2016-03-04 07:34 EST (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-09-08 11:41:47 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
alternate patch (to openssl rather than curl) (1.53 KB, patch)
2010-01-06 12:36 EST, Andy Grimm
no flags Details | Diff
Newer, upstream fix to openssl (2.36 KB, patch)
2010-01-11 20:26 EST, Michael K Johnson
no flags Details | Diff

  None (edit)
Description Michael K Johnson 2009-12-11 13:25:34 EST
rPath has had a report of a Denial of Service attack which we have reproduced on Red Hat Enterprise Linux 5 with the latest affected components installed.

Description of problem:
When the php module is enabled, after apache receives a "graceful" (USR1) signal, every SSLv3 request leaks a significant amount of memory (hundreds of KB).

Version-Release number of selected component (if applicable):
# rpm -q php httpd mod_ssl openssl
php-5.1.6-23.2.el5_3
httpd-2.2.3-31.el5_4.2
mod_ssl-2.2.3-31.el5_4.2
openssl-0.9.8e-12.el5

How reproducible:
Reliably

Steps to Reproduce:
1. Install mentioned components
2. service httpd start
3. note that memory utilization is stable for httpd
4. start a request loop:
   while :; do curl https://localhost/ -3 --insecure >/dev/null 2>&1; done
5. service httpd graceful
  
Actual results:
watch memory utilization spike

Expected results:
watch memory utilization remain stable

Additional info:
rPath's original report of this issue will be available pending coordinated
disclosure or decision not to require coordinated disclosure at
https://issues.rpath.com/browse/RPL-3157
I am marking this bugzilla report as security-sensitive as well pending
this decision about coordinated disclosure.
Comment 2 Joe Orton 2009-12-14 05:58:41 EST
strace-ing a child shows the memory leak is coming from repeated brk calls.
When I trap on 'brk' in a child process after it's been hammered this way for a while, the only calls I'm seeing are like the below, so I'd guess this is the culprits (though it's not conclusive).

So this looks like a re-incarnation of CVE-2008-1678 - see bug 447268.

    #1  0x00245c0d in __sbrk (increment=<value optimized out>) at sbrk.c:50
    #2  0x001e8b31 in __default_morecore (increment=Could not find the frame base
    for "__default_morecore".
    ) at morecore.c:49
    #3  0x001e5527 in _int_malloc (av=<value optimized out>, bytes=<value optimized
    out>) at malloc.c:3142
    #4  0x001e6a1e in __libc_malloc (bytes=<value optimized out>) at malloc.c:3605
    #5  0x004b421e in default_malloc_ex (num=Could not find the frame base for
    "default_malloc_ex".
    ) at mem.c:79
    #6  0x004b48cf in CRYPTO_malloc (num=<value optimized out>, file=<value
    optimized out>, line=<value optimized out>)
        at mem.c:328
    #7  0x004948eb in zlib_zalloc (opaque=Could not find the frame base for
    "zlib_zalloc".
    ) at c_zlib.c:40
    #8  0x005fcc9a in deflateInit2_ (strm=<value optimized out>, level=<value
    optimized out>, method=<value optimized out>, 
        windowBits=<value optimized out>, memLevel=<value optimized out>,
    strategy=<value optimized out>, 
        version=<value optimized out>, stream_size=<value optimized out>) at
    deflate.c:287
    #9  0x005fce42 in deflateInit_ (strm=Could not find the frame base for
    "deflateInit_".
    ) at deflate.c:210
    #10 0x0049483e in zlib_stateful_init (ctx=Could not find the frame base for
    "zlib_stateful_init".
    ) at c_zlib.c:173
    #11 0x004943ab in COMP_CTX_new (meth=<value optimized out>) at comp_lib.c:18
    #12 0x003a65db in ssl3_change_cipher_state (s=<value optimized out>,
    which=<value optimized out>) at s3_enc.c:239
    #13 0x003a6c01 in ssl3_do_change_cipher_spec (s=<value optimized out>) at
    s3_pkt.c:1239
    #14 0x003a81bc in ssl3_read_bytes (s=<value optimized out>, type=<value
    optimized out>, buf=<value optimized out>, 
        len=<value optimized out>, peek=<value optimized out>) at s3_pkt.c:1103
    #15 0x003a8c4b in ssl3_get_message (s=<value optimized out>, st1=<value
    optimized out>, stn=<value optimized out>, 
        mt=<value optimized out>, max=<value optimized out>, ok=<value optimized
    out>) at s3_both.c:394
    #16 0x0039db10 in ssl3_get_cert_verify (s=<value optimized out>) at
    s3_srvr.c:2229
    #17 0x003a015d in ssl3_accept (s=<value optimized out>) at s3_srvr.c:483
    #18 0x003b5f0a in SSL_accept (s=<value optimized out>) at ssl_lib.c:841
    #19 0x003a97db in ssl23_get_client_hello (s=<value optimized out>) at
    s23_srvr.c:577
    #20 0x003aa06b in ssl23_accept (s=<value optimized out>) at s23_srvr.c:203
    #21 0x003b5f0a in SSL_accept (s=<value optimized out>) at ssl_lib.c:841
    #22 0x0086c8e5 in ssl_io_filter_connect (filter_ctx=0x9b89ce0) at
    /usr/src/debug/httpd-2.2.3/modules/ssl/ssl_engine_io.c:1060
Comment 3 Joe Orton 2009-12-14 07:38:05 EST
If I use a LD_PRELOAD thunk to intercept the call:

  zlib_stateful_ex_idx =                                   CRYPTO_get_ex_new_index(CRYPTO_EX_INDEX_COMP,                                                0,NULL,NULL,NULL,zlib_stateful_free_ex_data);

(in OpenSSL crypto/comp/c_zlib.c)

the last time it returns after a graceful, the ex_idx is set to 4.

When using gdb to intercept the brk call as above, the ex_idx in the child process is set to zero.  This matches the root cause in bug 447268 - the zlib_stateful_free_ex_data callback won't get called on the ex_data because the index doesn't match.  We've seen this type of issue before when libcrypto.so gets reloaded from the process and static data is reinitialized.

I'm not sure whether we'd treat this as a security issue, and I'm unlikely to have time to look into this further before the new year.
Comment 4 Michael K Johnson 2009-12-14 10:37:04 EST
Thanks for sharing the current state of your research.  We'll continue our research and let you know what we find as we make significant progress.
Comment 5 Vincent Danen 2009-12-14 12:09:07 EST
I have httpd-2.2.14-jason.1 installed from the utterramblings repo on my CentOS host and did a quick test following the procedure outlined above and I don't see any significant spikes in memory usage by the httpd processes (watching using "top -u apache").  I see memory usage go up, but not significantly (keeping in mind that the curl command is making continual connections).  On RHEL5, however, I see similar behaviour (much less memory used, in fact, than on RHEL5 in vmware due to it being a production server).

Am I missing something when trying to reproduce this?
Comment 6 Michael K Johnson 2009-12-14 12:40:33 EST
In our tests (we haven't tried CentOS yet) mod_php had to be loaded and
SSLv3 had to be used for the request, and then it has reliably reproduced
after a SIGUSR1.  If mod_php isn't loaded, or SSLv2 is used for the request,
or you haven't sent a SIGUSR1 to ask for a graceful restart, it hasn't.  We
have so far looked only at the most recent package versions; we have not
reverted to earlier versions to compare.  So I'm not sure if you are
missing anything, but I've tried to summarize a bit differently in case
it makes the differences obvious.
Comment 7 Andy Grimm 2009-12-14 13:04:51 EST
A couple more notes:

* It appears that you must be running multiple threads for this to reproduce.  So, running "httpd -X" won't reproduce the problem.

* I updated my RHEL system to the packages in the utterramblings repo (httpd-2.2.14, php-5.2.11), and was still able to reproduce the issue.
Comment 8 Joe Orton 2009-12-14 15:19:13 EST
I could reproduce this on a box with php loaded (only vanilla RHEL5 packages) - haven't tried without.  Note that you must run the graceful restart to trigger it.  It did see a slow increase in the RSS figure in top without the restart or without passing -3 to curl, but no corresponding increase in the VIRT figure - I wasn't sure whether that was symptomatic of a leak.  When triggering the issue for real I saw a rapid increase in the VIRT figure.

php could make a difference if it causes libcrypto to stay resident in memory during the graceful restart (when all httpd modules are dlclosed then dlopened again), or if one of the php modules causes reinitialization/deinitialization of libcrypto to happen in a different order or anything insane like that.  There have been a couple of bugs like this over time, e.g.

https://issues.apache.org/bugzilla/show_bug.cgi?id=32529

springs to mind.
Comment 9 Andy Grimm 2009-12-17 16:21:22 EST
The problem is in libcurl, and it's identical to bug 447268 :

void Curl_ossl_cleanup(void)
{
  /* Free the SSL error strings */
  ERR_free_strings();

  /* EVP_cleanup() removes all ciphers and digests from the
     table. */
  EVP_cleanup();

#ifdef HAVE_ENGINE_cleanup
  ENGINE_cleanup();
#endif

#ifdef HAVE_CRYPTO_CLEANUP_ALL_EX_DATA
  /* this function was not present in 0.9.6b, but was added sometimes
     later */
  CRYPTO_cleanup_all_ex_data();
#endif
}


the php module calls this function.  I've verified that removing this call to CRYPTO_cleanup_all_ex_data fixes the problem.
Comment 10 Vincent Danen 2009-12-18 12:40:33 EST
This is CVE-2009-4355
Comment 11 Andy Grimm 2010-01-06 12:36:50 EST
Created attachment 382032 [details]
alternate patch (to openssl rather than curl)

I'm attaching an alternate patch that changes (hopefully fixes) the cleanup behavior in openssl, rather than altering php or curl.  This may be a better long-term solution, as it allows the zlib ex_data callbacks to be properly reinitialized the next time SSL_library_init is called after CRYPTO_cleanup_all_ex_data.
Comment 13 Michael K Johnson 2010-01-11 20:26:12 EST
Created attachment 383132 [details]
Newer, upstream fix to openssl

As noted at https://issues.rpath.com/browse/RPL-3157, Dr. Stephen Henson
of the openssl core team has provided a different implementation of the
openssl fix that rPath has successfully tested and that Dr. Henson will
commit in upstream openssl to resolve this issue.
Comment 19 Michael K Johnson 2010-01-13 10:15:53 EST
This issue is now publicly disclosed, and RPL-3157 is now publicly visible.
Please make this bugzilla report publicly visible as well.  Thank you!
Comment 24 errata-xmlrpc 2010-01-19 19:23:53 EST
This issue has been addressed in following products:

  Red Hat Enterprise Linux 5

Via RHSA-2010:0054 https://rhn.redhat.com/errata/RHSA-2010-0054.html
Comment 25 Michael Maniaci 2010-02-19 16:48:02 EST
Is it going to be addressed for Red hat Enterprise Linux 4?
Comment 26 Tomas Hoger 2010-02-22 11:36:37 EST
Red Hat Enterprise Linux 3 and 4 were not affected.
Comment 27 Fedora Update System 2010-03-23 03:27:15 EDT
openssl-0.9.8m-1.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/openssl-0.9.8m-1.fc11
Comment 28 Fedora Update System 2010-03-25 08:53:00 EDT
openssl-0.9.8n-1.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/openssl-0.9.8n-1.fc11
Comment 29 Fedora Update System 2010-03-30 06:44:01 EDT
openssl-1.0.0-1.fc13 has been submitted as an update for Fedora 13.
http://admin.fedoraproject.org/updates/openssl-1.0.0-1.fc13
Comment 30 Fedora Update System 2010-03-30 08:27:11 EDT
openssl-1.0.0-1.fc12 has been submitted as an update for Fedora 12.
http://admin.fedoraproject.org/updates/openssl-1.0.0-1.fc12
Comment 31 Fedora Update System 2010-04-08 23:42:33 EDT
openssl-1.0.0-1.fc13 has been pushed to the Fedora 13 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 32 Fedora Update System 2010-04-16 19:49:34 EDT
openssl-0.9.8n-1.fc11 has been pushed to the Fedora 11 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 33 Andy Grimm 2011-09-08 11:41:47 EDT
I believe that errata were released for all relevant distros.  It's time to close this one.

Note You need to log in before you can comment on or make changes to this bug.