|HTTPS+SSLVerifyClient require in <Directory>+big POST = Apache error
|Red Hat Enterprise Linux 5
|Karl Grindley <kgrindley>
|Joe Orton <jorton>
|BaseOS QE <qe-baseos-auto>
|Fixed In Version:
|2009-09-02 11:50:48 UTC
|RHEL 7.3 requirements from Atomic Host:
|Target Upstream Version:
Description Karl Grindley 2009-03-23 21:47:55 UTC
Description of problem: When a web application is configured for deep level SSLVerifyClient credentials, an SSL Renegotiation is called. Due to this, the POST data must be cached somewhere. The standard config for Apache 2.2.3 a hard coded 128K buffer. If the POST data exceeds this buffer size, then the Apache server terminates the client connection. The apache.org group has incorporated a new configuration option to allow the site to be configured on the Directory level the size of this buffer. (note the size of the buffer is a possible DoS vector as an attacker could us this to keep an Apache server busy if the buffer was large enough to exhaust memory with the server's MaxClients configuration. ie, if (MaxClients * SSLRenegBufferSize > free+swap) boxDies(); Most applications these days, especially with AJAX callbacks, can easily exceed this 128K limit. This is the case with a Drupal instance we are currently using requiring much larger HTTP POSTs to be buffered. Version-Release number of selected component (if applicable): httpd-2.2.3-22.el5 mod_ssl-2.2.3-22.el5 How reproducible: every time Steps to Reproduce: 1. setup an HTTPS site with Client certificates, and SSLVerifyClient in a <Directory> path 2. attempt to upload a POST > 128K in size 3. see ssl_error_log and white screen in browser Actual results: [Fri Mar 20 10:52:38 2009] [error] [client 188.8.131.52] request body exceeds maximum size for SSL buffer, referer: https://example.com/protected/posttest.php [Fri Mar 20 10:52:38 2009] [error] [client 184.108.40.206] could not buffer message body to allow SSL renegotiation to proceed, referer: https://example.com/protected/posttest.php Expected results: POST should be buffered, SSL client re-negotiation handshake completed, and POST variables passed to client. Additional info: origional issues.apache.org ticket discussion: https://issues.apache.org/bugzilla/show_bug.cgi?id=39243 Proposed Patch to add SSLRenegBufferSize to httpd.conf: http://svn.apache.org/viewvc?view=rev&revision=726109 documentation on the new SSLRenegBufferSize Config option: http://httpd.apache.org/docs/2.2/mod/mod_ssl.html#sslrenegbuffersize *NOTE* By simply changing SSL_MAX_IO_BUFFER to something larger in the httpd-2.2.3-22.el5.src.rpm, DOES NOT WORK well. There is some other bug that will cause large POSTs to fail during the re-negotiation process. I can verify that this is fixed in 2.2.11 from apache.org. This may be another bug ticket that would need to be filed if the issue could be tracked down. As a side note/my 2 cents: IMHO, this whole business of buffering the POST data in memory seems to be non-sense, and perhaps should be an option to buffer to /tmp file on disk? (much like a multi-part post is done with PHP. Perhaps if the POST were to exceed some config or predetermined max memory size. While i realize this just moves the DoS target, it seems less likely RAM would fill much faster than a Disk, AND slower IO would just slow down the attacker rather than kill the server completely) Seems that being able to DoS a web server by flooding large POST data is silly and a design bug in itself.
Comment 2 Joe Orton 2009-03-24 09:40:22 UTC
I can produce test packages with the SSLRenegBufferSize directive added for verification. w.r.t. to the 2 cents: it will be necessary to buffer the request body data somewhere in this case because of the way HTTP over SSL works. Buffering to disk just adds a whole world of extra complexity and different security risks. A well designed web application is able to avoid the case where buffering is required altogether, the buffering in mod_ssl is a workaround for legacy applications. Fixing the web application desing is only the way to avoid the DoS.
Comment 4 Karl Grindley 2009-03-24 11:49:04 UTC
Joe, I would be happy to test a test package for you. point taken about the complexity and security issues with disk buffering. However, the issue (i *think*) we are having is that there is no way (since http is designed to be stateless) to guarantee that the user would submit the POST before the SSL session cache timeout expires. Therefore some users that take their time submitting the POST data, would get the error. (this however is only my observation, and not sure 100% if this is a correct statement as alot under the hood of apache and mod_ssl is somewhat magic) So far i have been able to find very limited documentation that characterizes this particular issue no documentation as it applies to application design constraints. Do you know of any documentation you might be able to point me too that would shed some light on the issue? Thanks, Karl
Comment 5 Joe Orton 2009-03-24 15:14:09 UTC
Ways to avoid this issue are: 1) use a (separate) vhost as the client-cert-protected area, with SSLVerifyClient in vhost context. This ensures that the client cert request is sent in the initial SSL handshake, and a POST request with a 10GB request body will work fine without needing any buffering. 2) ensure by application layout/design that the first request to a client-cert-protected area is a GET, so the server does not have to stop and rehandshake after receiving the POST If you are suffering this issue because of session cache timeouts, why not bump the session cache size/timeout settings?
Comment 6 Karl Grindley 2009-05-18 14:12:16 UTC
Sorry for the big delay in updating this ticket. We have been characterizing all of the above proposed solutions. Some discovery on our side shows that: -not only does the SSLSessionCacheTimeout impact this issue, as does the KeepAlive and KeepAliveTimeout values -if the keepalive times out first, the SSL renegotiation is triggered, causing this bug to occur I can verify that a get before post (using some simple javascipt on the application side) does the trick, if: -the server has KeepAlive enabled -the KeepAlive timeout values is > than the longest time between the GET and then the POST (should be within seconds) -the SSLSessionCache is enabled and at least the same value of the KeepAlive timeout note that this still does not solve this issue for 100% all users. Since this is dependent on the keepalive working, the user must: -have keepalive enabled in their browser -using a proxy that allows keepalive In our market sector, some DoD agencies might prohibit keepalive either at their proxy or have configured managed desktops with keepalive to be disabled. While #2 solution looks like in the short term, it looks like #1 is the ONLY way to iron-clad solve this issue for all clients.
Comment 10 errata-xmlrpc 2009-09-02 11:50:48 UTC
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2009-1380.html