From Bugzilla Helper: User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4) Gecko/20030701 Description of problem: (note, we're running with the worker mpm, though this shouldn't differ significantly from the RH9 distributed httpd). We recently upgraded to 2.0.40-11.5 from the earlier errata. Upon doing this, the following errors began to pop up in our error log: [Mon Jul 07 17:56:18 2003] [error] Spurious SSL handshake interrupt [Hint: Usually just one of those OpenSSL confusions!?] [Mon Jul 07 17:56:18 2003] [error] SSL handshake failed (server vxn.datawire.net:443, client 207.16.131.100) [Mon Jul 07 17:56:18 2003] [error] SSL Library Error: 336131157 error:1408F455:lib(20):func(143):reason(1109) [Mon Jul 07 17:56:18 2003] [error] SSL handshake failed (server vxn.datawire.net:443, client 216.222.99.69) [Mon Jul 07 17:56:18 2003] [error] SSL Library Error: 336131157 error:1408F455:lib(20):func(143):reason(1109) [Mon Jul 07 17:56:18 2003] [error] SSL handshake failed (server vxn.datawire.net:443, client 66.180.107.137) [Mon Jul 07 17:56:18 2003] [error] SSL Library Error: 336131157 error:1408F455:lib(20):func(143):reason(1109) [Mon Jul 07 17:56:18 2003] [error] SSL handshake failed (server vxn.datawire.net:443, client 195.172.225.190) [Mon Jul 07 17:56:18 2003] [error] SSL Library Error: 336131157 error:1408F455:lib(20):func(143):reason(1109) [Mon Jul 07 17:56:18 2003] [error] SSL handshake failed (server vxn.datawire.net:443, client 68.99.115.36) [Mon Jul 07 17:56:18 2003] [error] SSL Library Error: 336131157 error:1408F455:lib(20):func(143):reason(1109) These continue with increasing frequency and regularity, until finally the server fails all SSL handshakes with a MAC decode error. I will attach our httpd and ssl conf files. We haven't been able to determine the exact cause in-house, and it looks like *any* ssl request may lead to this problem. We stave off the issue somewhat by running a great many threads to prevent the problem for a short time. Version-Release number of selected component (if applicable): httpd-2.0.40-11.5 How reproducible: Always Steps to Reproduce: 1. Start httpd. 2. Send a reasonable amount of traffic to the server over a period of time Actual Results: these errors will begin to fill the logs with increasing frequency until the server is unable to respond to SSL handshakes. Expected Results: No Issue with SSL Handshakes. Additional info: openSSL was also upgraded around this time, so I can't place the blame squarely on apache. I'm more than prepared to run any tests, or provide access to a system for testing if required.
When you say: > (note, we're running with the worker mpm, though this shouldn't differ > significantly from the RH9 distributed httpd). can you confirm that you are using the /usr/sbin/httpd.worker binary included in our httpd package, and you haven't recompiled httpd from source? If you switch to prefork is this problem still reproduceable?
The SSL session cache used can also be important; please do attach your config files httpd.conf and ssl.conf (from /etc/httpd/conf.d).
We're definately running worker - we have a custom apache module that requires as few copies as possible to be running, so we run worker with the attached configs, which allow us to have 5 parent threads, with 40 subthreads each. We're also using the standard session cache, which I believe is file-based.
Created attachment 92802 [details] httpd conf file
Created attachment 92803 [details] ssl conf file
Can you try using the shmcb session cache?
More information. Detailed testing today has shown that the problem lies somewhere between openssl-0.9.6b-28 and -32. We have confirmed that this problem goes away *completely* if we downgrade to -28. As such, -> openssl. I'm now in the process of working backwards to -28, patch-by-patch, until I find the source of the issue. I'll then let you know the specific patch causing the issue. More details: We're running 2.0.40-11.5, recompiled (via rpm --rebuild) on a redhat-7.3 platform (this has been stable for literally *millions* of hits prior to our openssl upgrade for nearly six months now). given this fact, it's safe to assume this would affect *any* apache-2.0 SSL installation on RH 7.3.
joe, I've cc'd you on this as well, as you may be able to confirm my findings. The problem has been localized to the RSA Blinding patch. It appears that the version of the patch being included by Red Hat does not properly support multithreaded environments, and as such is breaking in this case because the worker hybrid threading model is being used within apache. This is apparently affecting both 0.9.6 and 0.9.7, so it affects clients all the way to Red Hat 9, *not* just 7.3. I will verify this against a RH9 box tomorrow, but based on several usenet discussions it would appear that the issue appears on *any* client using multithreading. The fixed ghecked into 0.9.7b apparently includes an updated patch that *does* handle multithreading appropriately. I would suggest that the current Red Hat distributed patch be replaced by the 0.9.7b-based one to address this issue. Until then, simply backing out the RSA blinding patch completely eliminates this problem. References: http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&safe=off&frame=right&th=272632459dc5ef1&seekm=b8716r%242csk%241%40FreeBSD.csie.NCTU.edu.tw#link1 http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&safe=off&frame=right&th=1ce07ab31fece53c&seekm=b60t7m%242k11%241%40FreeBSD.csie.NCTU.edu.tw#link1
Spot on, that's a known bug in the OpenSSL erratum, I should have thought of that.
Thanks for the verification, Joe. I've now verified that this affects openssl-0.9.7a-5 and httpd-2.0.40-21.3 on a stock (patched) RH9 system if you use httpd.worker with SSL. as such, -> rh9. My question now is, what is the resolution? Is the answer simply that httpd.worker on an SSL site is completely unusable against the RH-supplied openssl? I'd also have to believe that virtually any multithreaded SSL app, including stunnel in daemon mode, would be bit by this bug as well..
Yes, unfortunately there is no workaround at the current time other than downgrading to an old version of OpenSSL. An erratum is in progress with the thread-safety fix.
RHBA-2003:237 is in progress with a fix for this issue.