Bug 98712 - RSA blinding fix is not thread-safe
RSA blinding fix is not thread-safe
Status: CLOSED CURRENTRELEASE
Product: Red Hat Linux
Classification: Retired
Component: openssl (Show other bugs)
9
i686 Linux
medium Severity high
: ---
: ---
Assigned To: Tomas Mraz
Brian Brock
https://vxn.datawire.net
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2003-07-07 17:10 EDT by Ken Snider
Modified: 2007-04-18 12:55 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2005-03-30 04:56:23 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
httpd conf file (35.44 KB, text/plain)
2003-07-08 13:30 EDT, Ken Snider
no flags Details
ssl conf file (11.13 KB, text/plain)
2003-07-08 13:31 EDT, Ken Snider
no flags Details

  None (edit)
Description Ken Snider 2003-07-07 17:10:14 EDT
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4) Gecko/20030701

Description of problem:
(note, we're running with the worker mpm, though this shouldn't differ
significantly from the RH9 distributed httpd).

We recently upgraded to 2.0.40-11.5 from the earlier errata. Upon doing this,
the following errors began to pop up in our error log:

[Mon Jul 07 17:56:18 2003] [error] Spurious SSL handshake interrupt [Hint:
Usually just one of those OpenSSL confusions!?]
[Mon Jul 07 17:56:18 2003] [error] SSL handshake failed (server
vxn.datawire.net:443, client 207.16.131.100)
[Mon Jul 07 17:56:18 2003] [error] SSL Library Error: 336131157
error:1408F455:lib(20):func(143):reason(1109)
[Mon Jul 07 17:56:18 2003] [error] SSL handshake failed (server
vxn.datawire.net:443, client 216.222.99.69)
[Mon Jul 07 17:56:18 2003] [error] SSL Library Error: 336131157
error:1408F455:lib(20):func(143):reason(1109)
[Mon Jul 07 17:56:18 2003] [error] SSL handshake failed (server
vxn.datawire.net:443, client 66.180.107.137)
[Mon Jul 07 17:56:18 2003] [error] SSL Library Error: 336131157
error:1408F455:lib(20):func(143):reason(1109)
[Mon Jul 07 17:56:18 2003] [error] SSL handshake failed (server
vxn.datawire.net:443, client 195.172.225.190)
[Mon Jul 07 17:56:18 2003] [error] SSL Library Error: 336131157
error:1408F455:lib(20):func(143):reason(1109)
[Mon Jul 07 17:56:18 2003] [error] SSL handshake failed (server
vxn.datawire.net:443, client 68.99.115.36)
[Mon Jul 07 17:56:18 2003] [error] SSL Library Error: 336131157
error:1408F455:lib(20):func(143):reason(1109)

These continue with increasing frequency and regularity, until finally the
server fails all SSL handshakes with a MAC decode error.

I will attach our httpd and ssl conf files.

We haven't been able to determine the exact cause in-house, and it looks like
*any* ssl request may lead to this problem. We stave off the issue somewhat by
running a great many threads to prevent the problem for a short time.

Version-Release number of selected component (if applicable):
httpd-2.0.40-11.5

How reproducible:
Always

Steps to Reproduce:
1. Start httpd.
2. Send a reasonable amount of traffic to the server over a period of time
    

Actual Results:  these errors will begin to fill the logs with increasing
frequency until the server is unable to respond to SSL handshakes.

Expected Results:  No Issue with SSL Handshakes.

Additional info:

openSSL was also upgraded around this time, so I can't place the blame squarely
on apache.

I'm more than prepared to run any tests, or provide access to a system for
testing if required.
Comment 1 Joe Orton 2003-07-08 02:58:19 EDT
When you say:

> (note, we're running with the worker mpm, though this shouldn't differ
> significantly from the RH9 distributed httpd).

can you confirm that you are using the /usr/sbin/httpd.worker binary included in
our httpd package, and you haven't recompiled httpd from source?  If you switch
to prefork is this problem still reproduceable?
Comment 2 Joe Orton 2003-07-08 09:42:27 EDT
The SSL session cache used can also be important; please do attach your config
files httpd.conf and ssl.conf (from /etc/httpd/conf.d).
Comment 3 Ken Snider 2003-07-08 13:29:32 EDT
We're definately running worker - we have a custom apache module that requires
as few copies as possible to be running, so we run worker with the attached
configs, which allow us to have 5 parent threads, with 40 subthreads each.

We're also using the standard session cache, which I believe is file-based.
Comment 4 Ken Snider 2003-07-08 13:30:25 EDT
Created attachment 92802 [details]
httpd conf file
Comment 5 Ken Snider 2003-07-08 13:31:34 EDT
Created attachment 92803 [details]
ssl conf file
Comment 6 Joe Orton 2003-07-08 17:02:52 EDT
Can you try using the shmcb session cache?
Comment 7 Ken Snider 2003-07-08 19:38:26 EDT
More information.

Detailed testing today has shown that the problem lies somewhere between
openssl-0.9.6b-28 and -32. We have confirmed that this problem goes away
*completely* if we downgrade to -28.

As such, -> openssl.

I'm now in the process of working backwards to -28, patch-by-patch, until I find
the source of the issue. I'll then let you know the specific patch causing the
issue.

More details:

We're running 2.0.40-11.5, recompiled (via rpm --rebuild) on a redhat-7.3
platform (this has been stable for literally *millions* of hits prior to our
openssl upgrade for nearly six months now).

given this fact, it's safe to assume this would affect *any* apache-2.0 SSL
installation on RH 7.3.
Comment 8 Ken Snider 2003-07-09 00:56:00 EDT
joe, I've cc'd you on this as well, as you may be able to confirm my findings.

The problem has been localized to the RSA Blinding patch.

It appears that the version of the patch being included by Red Hat does not
properly support multithreaded environments, and as such is breaking in this
case because the worker hybrid threading model is being used within apache. This
is apparently affecting both 0.9.6 and 0.9.7, so it affects clients all the way
to Red Hat 9, *not* just 7.3. I will verify this against a RH9 box tomorrow, but
based on several usenet discussions it would appear that the issue appears on
*any* client using multithreading.

The fixed ghecked into 0.9.7b apparently includes an updated patch that *does*
handle multithreading appropriately. I would suggest that the current Red Hat
distributed patch be replaced by the 0.9.7b-based one to address this issue.

Until then, simply backing out the RSA blinding patch completely eliminates this
problem.

References:

http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&safe=off&frame=right&th=272632459dc5ef1&seekm=b8716r%242csk%241%40FreeBSD.csie.NCTU.edu.tw#link1

http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&safe=off&frame=right&th=1ce07ab31fece53c&seekm=b60t7m%242k11%241%40FreeBSD.csie.NCTU.edu.tw#link1
Comment 9 Joe Orton 2003-07-09 02:02:33 EDT
Spot on, that's a known bug in the OpenSSL erratum, I should have thought of that.
Comment 10 Ken Snider 2003-07-09 12:08:41 EDT
Thanks for the verification, Joe.

I've now verified that this affects openssl-0.9.7a-5 and httpd-2.0.40-21.3 on a
stock (patched) RH9 system if you use httpd.worker with SSL.

as such, -> rh9.

My question now is, what is the resolution? Is the answer simply that
httpd.worker on an SSL site is completely unusable against the RH-supplied openssl?

I'd also have to believe that virtually any multithreaded SSL app, including
stunnel in daemon mode, would be bit by this bug as well..
Comment 11 Joe Orton 2003-07-09 15:49:30 EDT
Yes, unfortunately there is no workaround at the current time other than
downgrading to an old version of OpenSSL.  An erratum is in progress with the
thread-safety fix.
Comment 12 Mark J. Cox (Product Security) 2003-07-17 06:47:54 EDT
RHBA-2003:237 is in progress with a fix for this issue.

Note You need to log in before you can comment on or make changes to this bug.