Bug 688901

Summary:	occasional 502 errors on httpd load balancer
Product:	Red Hat Enterprise Linux 5	Reporter:	Michael Young <m.a.young>
Component:	openssl	Assignee:	Tomas Mraz <tmraz>
Status:	CLOSED ERRATA	QA Contact:	BaseOS QE Security Team <qe-baseos-security>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	5.6	CC:	cward, mvadkert, pvrabec
Target Milestone:	rc	Keywords:	OtherQA
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openssl-0.9.8e-19.el5	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2011-07-21 07:41:41 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Michael Young 2011-03-18 12:48:35 UTC

We have an RHEL5 load balancer which balances apache httpd traffic between 3 backend servers (via the <Proxy balancer://balancername> and ProxyPass / balancer://balancername/ directives). This has package versions httpd-2.2.3-43.el5_5.3 and openssl-0.9.8e-12.el5_5.7

This works well for most of the time, but it occasionally reports a lot of errors like the following

[Thu Mar 17 08:31:47 2011] [error] (502)Unknown error 502: proxy: pass request body failed to 1.2.3.4:443 (1.2.3.4)
[Thu Mar 17 08:31:47 2011] [error] proxy: pass request body failed to 1.2.3.4:443 (1.2.3.4) from 5.6.7.8 ()

sometimes it gets better by itself, in other cases the service continues to return such errors until we restart it. With more logging turned on on a test server this looks like

[Thu Mar 17 19:36:39 2011] [info] [client 1.2.3.4] SSL Proxy connect failed
[Thu Mar 17 19:36:39 2011] [info] SSL Library Error: 336142597 error:14092105:SSL routines:SSL3_GET_SERVER_HELLO:wrong cipher returned
[Thu Mar 17 19:36:39 2011] [info] [client 1.2.3.4] Connection closed to child 0 with abortive shutdown (server 5.6.7.8:443)
[Thu Mar 17 19:36:39 2011] [error] (502)Unknown error 502: proxy: pass request body failed to 1.2.3.4:443 (1.2.3.4)
[Thu Mar 17 19:36:39 2011] [error] proxy: pass request body failed to 1.2.3.4:443 (1.2.3.4) from 5.6.7.8 ()
[Thu Mar 17 19:36:39 2011] [info] [client 5.6.7.8] Connection closed to child 14 with standard shutdown (server 5.6.7.8:443)

I think I am seeing openssl bug 1795 (eg. see http://marc.info/?t=122788276300003&r=1&w=2 ). This is fixed in later versions of openssl (the cvs commit is http://cvs.openssl.org/chngview?cn=17992 ). Could this fix be backported to the RHEL5 package please? The bug may also be in the RHEL6 openssl098e package.

Comment 1 Eduard Benes 2011-03-31 08:48:36 UTC

Hello guys, is there a way how to reproduce this bug other than trying to simulate the reported environment where it occured? Could you please provide some steps how to reproduce this bug and verify the potential patch other than code review? Thanks

Comment 2 Tomas Mraz 2011-03-31 09:11:15 UTC

Unfortunately I do not have any reproducer - it would have to be a multithreaded SSL client application that tries to connect to the server simultaneously with multiple threads.

Comment 3 Michael Young 2011-03-31 09:25:44 UTC

My test system was apache running as an https load balancer in front of two apache https backends. I loaded the system by running 20-30 jobs which were repeatedly doing a wget (set to discard the page after retrieving it). The backends were running Blackboard software behind apache but as I was only ever fetching the front page that probably doesn't matter.
When I was testing I got it to show these 502 outbursts 3 times that day. I then ran it over the weekend with the patch applied (but without the Blackboard software as I had broken it by filling up the database with log entries) without any further failures.
We have also been running the patched openssl on our live system for a week now and not seen any repeat of these 502 outbursts.

Comment 5 Michael Young 2011-03-31 13:53:03 UTC

It looks like I forgot to mention that the httpd load balancers were using the worker MPM, though I don't know if that is significant.

Comment 9 errata-xmlrpc 2011-07-21 07:41:41 UTC

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-1010.html