It was reported  that a flaw exists in how openSSL handles SSL renegotiation. Because of the processing power required to handle an SSL/TLS handshake, with renegotiation enabled, a user can send multiple handshakes per second due to the renegotiation request being permitted. This could allow a malicious user to send multiple renegotiation requests and exhaust server resources.
Note that this is not the only way to cause a denial of service on an SSL-enabled service; there are many other ways to accomplish the same thing, this just makes it easier.
What makes this bug even more confusing is that this report is recent, with a 2011 CVE, however the recommended fix in the report is to upgrade to OpenSSL 0.9.8l, which is what disabled renegotiation, however it was subsequently re-enabled when OpenSSL added support for RFC5746 in OpenSSL 0.9.8m (which the reporter seems to imply isn't sufficient). I'm not sure where the CVE assignment came from; on MITRE's site is is reserved yet and I have not seen this discussed anywhere else but in a Nessus scan report .
As an aside, and to reference something we may need to look at, there seems to be some concern with Apache and how it works with SSL renegotiation disabled , so we will need to look into cases where disabling SSL renegotiation may impact expected behaviour of currently released products.
(In reply to comment #0)
> It was reported  that a flaw exists in how openSSL handles SSL
The referenced write-up of the issue describes this as a SSL/TLS protocol issue that may affect various implementations that permit client-initiated renegotiation without arbitrary limits to restrict renegotiation rate. This is more likely to affect applications using SSL libraries that handle renegotiations without requiring applications support (OpenSSL, NSS, Java JRE).
> Note that this is not the only way to cause a denial of service on an
> SSL-enabled service; there are many other ways to accomplish the same thing,
> this just makes it easier.
This issue was raised on the IETF TLS working group mailing list:
Several replies in the discussion indicate that this attack offers limited advantages over an attack were only one (initial) negotiation is performed over one connection. Using renegotiation allows making this attack more efficient by using single connection for several handshakes. However, the use of SSL implies that server needs to perform certain time-expensive computations for its normal operation.
> What makes this bug even more confusing is that this report is recent, with a
> 2011 CVE, however the recommended fix in the report is to upgrade to OpenSSL
> 0.9.8l, which is what disabled renegotiation, however it was subsequently
> re-enabled when OpenSSL added support for RFC5746 in OpenSSL 0.9.8m (which the
> reporter seems to imply isn't sufficient).
Renegotiation was disabled in 0.9.8l as a temporary fix for CVE-2009-3555 issue, rather than to address this DoS attack. It was re-enabled in the next version that provided a better fix for CVE-2009-3555 - an implementation of the RFC5746.
> As an aside, and to reference something we may need to look at, there seems to
> be some concern with Apache and how it works with SSL renegotiation disabled
> , so we will need to look into cases where disabling SSL renegotiation may
> impact expected behaviour of currently released products.
This problem would not affect recent httpd packages at all. As a mitigation for CVE-2009-3555, httpd's mod_ssl was modified to reject all client-initiated renegotiations. This change was backported to Red Hat Enterprise Linux httpd packages:
Adding NSS folks too. It should be noted that NSS already provides a mechanism that can be used by applications to easily instruct the library to reject session renegotiations by changing the value of the SSL_ENABLE_RENEGOTIATION option. It's default can be controlled by the NSS_SSL_ENABLE_RENEGOTIATION environment variable. Additional details can be found in the kbase article related to the CVE-2009-3555:
Similar change to that added to httpd's mod_ssl was added to tomcat-native as well to mitigate CVE-2009-3555.
TLS implementation in IBM JDK allows disabling all renegotiations using the com.ibm.jsse2.renegotiate system property.
As previous comments, and the linked IETF TLS WG mailing list discussion indicate, there currently isn't a conclusion whether this issue should be addressed in the TLS protocol and the libraries implementing it, as it's inherent property of the protocol that client can "DoS" the server by making it perform CPU expensive cryptographic operations. Therefore, we do not plan to release update addressing this problem at this time.
There's new input: The hacker group THC released a tutorial how to DoS SSL application, again accelerated by renegotiation:
RH/Centos 5.7 still provides OpenSSL 0.9.8e, which does not allow to disable renegotiation system wide as a work around. I guess the only choice is then to disable it on a per application basis, which may not be for some applications.
Few decent write-ups on the topic:
MITRE has assigned CVE-2011-5094 to nss, and this CVE is specific to openssl. Bug #833118 has been filed for the nss issue.
As of 6.2, this 'bug' still exists when using RHEL packaged vsftpd with FTPeS enabled. This is (it appears) a trivially exploited threat to availability of services. Why is the status still 'new', over a year after an exploit to the vulnerability has been published?