Due to a bug in handling return code from openssl native calls, the SslHandler that uses OpenSslEngine can end up in an infinite loop and eat up all CPU cycles, which may lead to DoS of the system.
This can only happen if the server has renegotiation enabled (which is set as default). Only systems using SslProvider.OpenSsl are affected, that will be true if netty-tcnative is in the classpath and openssl is installed.
If netty-tcnative is linked against boringssl, the problem does not exist, since renegotiation is not supported in boringssl.
Versions affected: Netty 4.0.0.Final - 4.0.36.Final and 4.1.0.Final
Users can use -Djdk.tls.rejectClientInitiatedRenegotiation=true to disable renegotiation and avoid this issue.
Created netty tracking bugs for this issue:
Affects: fedora-all [bug 1347497]
This was fixed in netty 4.0.37 and 4.1.1:
Upstream bug / pull request:
There are 3 netty versions in Red Hat Software Collections and Red Hat Developer Toolset:
- rh-java-common-netty and devtoolset-4-netty - both used on upstream version 4.0.28. However, OpenSslEngine is removed from those packages, similar to netty packages in Fedora:
- thermostat1-netty - based on upstream version 3.6.3, which does not include OpenSslEngine, which was only added upstream in version 3.9.2:
Additionally, netty-tcnative required by OpenSslEngine is not shipped as part of RHSCL or DTS.
Fedora also includes netty3 packages, using upstream version 3.9.3, and hence including OpenSslEngine. The information above suggests only 4.0.0 and later were affected by this issue, but, based on a quick look at the code and patch, it's not obvious to me why 3.9.x are not affected. CCing netty3 packages owner so that they can be double checked.
ODL upstream announcement and patch: https://git.opendaylight.org/gerrit/#/c/40098/
This issue has been addressed in the following products:
Via RHSA-2017:1097 https://access.redhat.com/errata/RHSA-2017:1097