rPath has had a report of a Denial of Service attack which we have reproduced on Red Hat Enterprise Linux 5 with the latest affected components installed. Description of problem: When the php module is enabled, after apache receives a "graceful" (USR1) signal, every SSLv3 request leaks a significant amount of memory (hundreds of KB). Version-Release number of selected component (if applicable): # rpm -q php httpd mod_ssl openssl php-5.1.6-23.2.el5_3 httpd-2.2.3-31.el5_4.2 mod_ssl-2.2.3-31.el5_4.2 openssl-0.9.8e-12.el5 How reproducible: Reliably Steps to Reproduce: 1. Install mentioned components 2. service httpd start 3. note that memory utilization is stable for httpd 4. start a request loop: while :; do curl https://localhost/ -3 --insecure >/dev/null 2>&1; done 5. service httpd graceful Actual results: watch memory utilization spike Expected results: watch memory utilization remain stable Additional info: rPath's original report of this issue will be available pending coordinated disclosure or decision not to require coordinated disclosure at https://issues.rpath.com/browse/RPL-3157 I am marking this bugzilla report as security-sensitive as well pending this decision about coordinated disclosure.
strace-ing a child shows the memory leak is coming from repeated brk calls. When I trap on 'brk' in a child process after it's been hammered this way for a while, the only calls I'm seeing are like the below, so I'd guess this is the culprits (though it's not conclusive). So this looks like a re-incarnation of CVE-2008-1678 - see bug 447268. #1 0x00245c0d in __sbrk (increment=<value optimized out>) at sbrk.c:50 #2 0x001e8b31 in __default_morecore (increment=Could not find the frame base for "__default_morecore". ) at morecore.c:49 #3 0x001e5527 in _int_malloc (av=<value optimized out>, bytes=<value optimized out>) at malloc.c:3142 #4 0x001e6a1e in __libc_malloc (bytes=<value optimized out>) at malloc.c:3605 #5 0x004b421e in default_malloc_ex (num=Could not find the frame base for "default_malloc_ex". ) at mem.c:79 #6 0x004b48cf in CRYPTO_malloc (num=<value optimized out>, file=<value optimized out>, line=<value optimized out>) at mem.c:328 #7 0x004948eb in zlib_zalloc (opaque=Could not find the frame base for "zlib_zalloc". ) at c_zlib.c:40 #8 0x005fcc9a in deflateInit2_ (strm=<value optimized out>, level=<value optimized out>, method=<value optimized out>, windowBits=<value optimized out>, memLevel=<value optimized out>, strategy=<value optimized out>, version=<value optimized out>, stream_size=<value optimized out>) at deflate.c:287 #9 0x005fce42 in deflateInit_ (strm=Could not find the frame base for "deflateInit_". ) at deflate.c:210 #10 0x0049483e in zlib_stateful_init (ctx=Could not find the frame base for "zlib_stateful_init". ) at c_zlib.c:173 #11 0x004943ab in COMP_CTX_new (meth=<value optimized out>) at comp_lib.c:18 #12 0x003a65db in ssl3_change_cipher_state (s=<value optimized out>, which=<value optimized out>) at s3_enc.c:239 #13 0x003a6c01 in ssl3_do_change_cipher_spec (s=<value optimized out>) at s3_pkt.c:1239 #14 0x003a81bc in ssl3_read_bytes (s=<value optimized out>, type=<value optimized out>, buf=<value optimized out>, len=<value optimized out>, peek=<value optimized out>) at s3_pkt.c:1103 #15 0x003a8c4b in ssl3_get_message (s=<value optimized out>, st1=<value optimized out>, stn=<value optimized out>, mt=<value optimized out>, max=<value optimized out>, ok=<value optimized out>) at s3_both.c:394 #16 0x0039db10 in ssl3_get_cert_verify (s=<value optimized out>) at s3_srvr.c:2229 #17 0x003a015d in ssl3_accept (s=<value optimized out>) at s3_srvr.c:483 #18 0x003b5f0a in SSL_accept (s=<value optimized out>) at ssl_lib.c:841 #19 0x003a97db in ssl23_get_client_hello (s=<value optimized out>) at s23_srvr.c:577 #20 0x003aa06b in ssl23_accept (s=<value optimized out>) at s23_srvr.c:203 #21 0x003b5f0a in SSL_accept (s=<value optimized out>) at ssl_lib.c:841 #22 0x0086c8e5 in ssl_io_filter_connect (filter_ctx=0x9b89ce0) at /usr/src/debug/httpd-2.2.3/modules/ssl/ssl_engine_io.c:1060
If I use a LD_PRELOAD thunk to intercept the call: zlib_stateful_ex_idx = CRYPTO_get_ex_new_index(CRYPTO_EX_INDEX_COMP, 0,NULL,NULL,NULL,zlib_stateful_free_ex_data); (in OpenSSL crypto/comp/c_zlib.c) the last time it returns after a graceful, the ex_idx is set to 4. When using gdb to intercept the brk call as above, the ex_idx in the child process is set to zero. This matches the root cause in bug 447268 - the zlib_stateful_free_ex_data callback won't get called on the ex_data because the index doesn't match. We've seen this type of issue before when libcrypto.so gets reloaded from the process and static data is reinitialized. I'm not sure whether we'd treat this as a security issue, and I'm unlikely to have time to look into this further before the new year.
Thanks for sharing the current state of your research. We'll continue our research and let you know what we find as we make significant progress.
I have httpd-2.2.14-jason.1 installed from the utterramblings repo on my CentOS host and did a quick test following the procedure outlined above and I don't see any significant spikes in memory usage by the httpd processes (watching using "top -u apache"). I see memory usage go up, but not significantly (keeping in mind that the curl command is making continual connections). On RHEL5, however, I see similar behaviour (much less memory used, in fact, than on RHEL5 in vmware due to it being a production server). Am I missing something when trying to reproduce this?
In our tests (we haven't tried CentOS yet) mod_php had to be loaded and SSLv3 had to be used for the request, and then it has reliably reproduced after a SIGUSR1. If mod_php isn't loaded, or SSLv2 is used for the request, or you haven't sent a SIGUSR1 to ask for a graceful restart, it hasn't. We have so far looked only at the most recent package versions; we have not reverted to earlier versions to compare. So I'm not sure if you are missing anything, but I've tried to summarize a bit differently in case it makes the differences obvious.
A couple more notes: * It appears that you must be running multiple threads for this to reproduce. So, running "httpd -X" won't reproduce the problem. * I updated my RHEL system to the packages in the utterramblings repo (httpd-2.2.14, php-5.2.11), and was still able to reproduce the issue.
I could reproduce this on a box with php loaded (only vanilla RHEL5 packages) - haven't tried without. Note that you must run the graceful restart to trigger it. It did see a slow increase in the RSS figure in top without the restart or without passing -3 to curl, but no corresponding increase in the VIRT figure - I wasn't sure whether that was symptomatic of a leak. When triggering the issue for real I saw a rapid increase in the VIRT figure. php could make a difference if it causes libcrypto to stay resident in memory during the graceful restart (when all httpd modules are dlclosed then dlopened again), or if one of the php modules causes reinitialization/deinitialization of libcrypto to happen in a different order or anything insane like that. There have been a couple of bugs like this over time, e.g. https://issues.apache.org/bugzilla/show_bug.cgi?id=32529 springs to mind.
The problem is in libcurl, and it's identical to bug 447268 : void Curl_ossl_cleanup(void) { /* Free the SSL error strings */ ERR_free_strings(); /* EVP_cleanup() removes all ciphers and digests from the table. */ EVP_cleanup(); #ifdef HAVE_ENGINE_cleanup ENGINE_cleanup(); #endif #ifdef HAVE_CRYPTO_CLEANUP_ALL_EX_DATA /* this function was not present in 0.9.6b, but was added sometimes later */ CRYPTO_cleanup_all_ex_data(); #endif } the php module calls this function. I've verified that removing this call to CRYPTO_cleanup_all_ex_data fixes the problem.
This is CVE-2009-4355
Created attachment 382032 [details] alternate patch (to openssl rather than curl) I'm attaching an alternate patch that changes (hopefully fixes) the cleanup behavior in openssl, rather than altering php or curl. This may be a better long-term solution, as it allows the zlib ex_data callbacks to be properly reinitialized the next time SSL_library_init is called after CRYPTO_cleanup_all_ex_data.
Created attachment 383132 [details] Newer, upstream fix to openssl As noted at https://issues.rpath.com/browse/RPL-3157, Dr. Stephen Henson of the openssl core team has provided a different implementation of the openssl fix that rPath has successfully tested and that Dr. Henson will commit in upstream openssl to resolve this issue.
This issue is now publicly disclosed, and RPL-3157 is now publicly visible. Please make this bugzilla report publicly visible as well. Thank you!
This issue has been addressed in following products: Red Hat Enterprise Linux 5 Via RHSA-2010:0054 https://rhn.redhat.com/errata/RHSA-2010-0054.html
Is it going to be addressed for Red hat Enterprise Linux 4?
Red Hat Enterprise Linux 3 and 4 were not affected.
openssl-0.9.8m-1.fc11 has been submitted as an update for Fedora 11. http://admin.fedoraproject.org/updates/openssl-0.9.8m-1.fc11
openssl-0.9.8n-1.fc11 has been submitted as an update for Fedora 11. http://admin.fedoraproject.org/updates/openssl-0.9.8n-1.fc11
openssl-1.0.0-1.fc13 has been submitted as an update for Fedora 13. http://admin.fedoraproject.org/updates/openssl-1.0.0-1.fc13
openssl-1.0.0-1.fc12 has been submitted as an update for Fedora 12. http://admin.fedoraproject.org/updates/openssl-1.0.0-1.fc12
openssl-1.0.0-1.fc13 has been pushed to the Fedora 13 stable repository. If problems still persist, please make note of it in this bug report.
openssl-0.9.8n-1.fc11 has been pushed to the Fedora 11 stable repository. If problems still persist, please make note of it in this bug report.
I believe that errata were released for all relevant distros. It's time to close this one.