Bug 491763

Summary:	HTTPS+SSLVerifyClient require in <Directory>+big POST = Apache error
Product:	Red Hat Enterprise Linux 5	Reporter:	Karl Grindley <kgrindley>
Component:	httpd	Assignee:	Joe Orton <jorton>
Status:	CLOSED ERRATA	QA Contact:	BaseOS QE <qe-baseos-auto>
Severity:	medium	Docs Contact:
Priority:	low
Version:	5.5	CC:	ohudlick
Target Milestone:	rc
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2009-09-02 11:50:48 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Karl Grindley 2009-03-23 21:47:55 UTC

Description of problem:

When a web application is configured for deep level SSLVerifyClient credentials, an SSL Renegotiation is called. Due to this, the POST data must be cached somewhere. The standard config for Apache 2.2.3 a hard coded 128K buffer. If the POST data exceeds this buffer size, then the Apache server terminates the client connection.

The apache.org group has incorporated a new configuration option to allow the site to be configured on the Directory level the size of this buffer. (note the size of the buffer is a possible DoS vector as an attacker could us this to keep an Apache server busy if the buffer was large enough to exhaust memory with the server's MaxClients configuration.
ie, if (MaxClients * SSLRenegBufferSize > free+swap) boxDies();

Most applications these days, especially with AJAX callbacks, can easily exceed this 128K limit. This is the case with a Drupal instance we are currently using requiring much larger HTTP POSTs to be buffered.

Version-Release number of selected component (if applicable):
httpd-2.2.3-22.el5
mod_ssl-2.2.3-22.el5

How reproducible:
every time

Steps to Reproduce:
1. setup an HTTPS site with Client certificates, and SSLVerifyClient in a <Directory> path
2. attempt to upload a POST > 128K in size
3. see ssl_error_log and white screen in browser

Actual results:

[Fri Mar 20 10:52:38 2009] [error] [client 1.2.3.4] request body exceeds maximum size for SSL buffer, referer: https://example.com/protected/posttest.php
[Fri Mar 20 10:52:38 2009] [error] [client 1.2.3.4] could not buffer message body to allow SSL renegotiation to proceed, referer: https://example.com/protected/posttest.php

Expected results:
POST should be buffered, SSL client re-negotiation handshake completed, and POST variables passed to client.

Additional info:

origional issues.apache.org ticket discussion:
https://issues.apache.org/bugzilla/show_bug.cgi?id=39243

Proposed Patch to add SSLRenegBufferSize to httpd.conf:
http://svn.apache.org/viewvc?view=rev&revision=726109

documentation on the new SSLRenegBufferSize Config option:
http://httpd.apache.org/docs/2.2/mod/mod_ssl.html#sslrenegbuffersize

*NOTE* By simply changing SSL_MAX_IO_BUFFER to something larger in the httpd-2.2.3-22.el5.src.rpm, DOES NOT WORK well. There is some other bug that will cause large POSTs to fail during the re-negotiation process. I can verify that this is fixed in 2.2.11 from apache.org. This may be another bug ticket that would need to be filed if the issue could be tracked down.

As a side note/my 2 cents: IMHO, this whole business of buffering the POST data in memory seems to be non-sense, and perhaps should be an option to buffer to /tmp file on disk? (much like a multi-part post is done with PHP. Perhaps if the POST were to exceed some config or predetermined max memory size. While i realize this just moves the DoS target, it seems less likely RAM would fill much faster than a Disk, AND slower IO would just slow down the attacker rather than kill the server completely) Seems that being able to DoS a web server by flooding large POST data is silly and a design bug in itself.

Comment 2 Joe Orton 2009-03-24 09:40:22 UTC

I can produce test packages with the SSLRenegBufferSize directive added for verification.

w.r.t. to the 2 cents: it will be necessary to buffer the request body data somewhere in this case because of the way HTTP over SSL works.  Buffering to disk just adds a whole world of extra complexity and different security risks.

A well designed web application is able to avoid the case where buffering is required altogether, the buffering in mod_ssl is a workaround for legacy applications.  Fixing the web application desing is only the way to avoid the DoS.

Comment 4 Karl Grindley 2009-03-24 11:49:04 UTC

Joe,

I would be happy to test a test package for you.

point taken about the complexity and security issues with disk buffering.

However, the issue (i *think*) we are having is that there is no way (since
http is designed to be stateless) to guarantee that the user would submit the
POST before the SSL session cache timeout expires.  Therefore some users that
take their time submitting the POST data, would get the error.  (this however
is only my observation, and not sure 100% if this is a correct statement as
alot under the hood of apache and mod_ssl is somewhat magic)  So far i have
been able to find very limited documentation that characterizes this particular
issue no documentation as it applies to application design constraints.  Do you
know of any documentation you might be able to point me too that would shed
some light on the issue?

Thanks,
Karl

Comment 5 Joe Orton 2009-03-24 15:14:09 UTC

Ways to avoid this issue are:

1) use a (separate) vhost as the client-cert-protected area, with SSLVerifyClient in vhost context.  This ensures that the client cert request is sent in the initial SSL handshake, and a POST request with a 10GB request body will work fine without needing any buffering.

2) ensure by application layout/design that the first request to a client-cert-protected area is a GET, so the server does not have to stop and rehandshake after receiving the POST

If you are suffering this issue because of session cache timeouts, why not bump the session cache size/timeout settings?

Comment 6 Karl Grindley 2009-05-18 14:12:16 UTC

Sorry for the big delay in updating this ticket.  We have been characterizing all of the above proposed solutions.  Some discovery on our side shows that:

-not only does the SSLSessionCacheTimeout impact this issue, as does the KeepAlive and KeepAliveTimeout values
-if the keepalive times out first, the SSL renegotiation is triggered, causing this bug to occur

I can verify that a get before post (using some simple javascipt on the application side) does the trick, if:
-the server has KeepAlive enabled
-the KeepAlive timeout values is > than the longest time between the GET and then the POST (should be within seconds)
-the SSLSessionCache is enabled and at least the same value of the KeepAlive timeout

note that this still does not solve this issue for 100% all users.  Since this is dependent on the keepalive working, the user must:
-have keepalive enabled in their browser
-using a proxy that allows keepalive

In our market sector, some DoD agencies might prohibit keepalive either at their proxy or have configured managed desktops with keepalive to be disabled.

While #2 solution looks like in the short term, it looks like #1 is the ONLY way to iron-clad solve this issue for all clients.

Comment 10 errata-xmlrpc 2009-09-02 11:50:48 UTC

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-1380.html