Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 1379701 - Customer cert for route caused fatal error in haproxy router
Customer cert for route caused fatal error in haproxy router
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Routing (Show other bugs)
3.2.0
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Ram Ranganathan
zhaozhanqi
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-09-27 09:24 EDT by Steven Walter
Modified: 2017-03-08 13 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: By default extended certificate validation was not enabled so bad certs could crash the router. Consequence: Bad certificates in routes could crash the router. Fix: We changed the default in 'oadm router' to turn on extended validation when a router is created. Result: Bad certificates are caught and the route they are associated with is not used (and an appropriate status is set on it)
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-01-18 07:41:47 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Origin (Github) 11218 None None None 2016-11-30 08:13 EST
Red Hat Product Errata RHBA-2017:0066 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.4 RPM Release Advisory 2017-01-18 12:23:26 EST

  None (edit)
Description Steven Walter 2016-09-27 09:24:01 EDT
Description of problem:

Using an improperly formatted cert for a route, using a key that had a passphrase on it, caused an outage in the router for all customers.


It appears the offending cert causes issues when re-encrypting the route on the F5. When the certificate was created they had a passphrase on it, which needs to be removed for it to work. Apparently this caused a fatal error which required deleting the route to recover from. One customer using an improperly formatted certificate caused a router outage for the whole environment.



Version-Release number of selected component (if applicable):
# openshift version
openshift v3.2.1.4-1-g1864c8f
kubernetes v1.2.0-36-g4a3f9c5
etcd 2.2.5


How reproducible:
Unverified


Actual results:

The application which had the bad cert showed these logs:

[ALERT] 255/130843 (51214) : parsing [/var/lib/haproxy/conf/haproxy.config:112] : 'bind 127.0.0.1:10444' :
  unable to load SSL private key from PEM file '/var/lib/containers/router/certs/example.pem'.
  unable to load SSL private key from PEM file '/var/lib/containers/router/certs/example.pem'.
  unable to load SSL private key from PEM file '/var/lib/containers/router/certs/example.pem'.
[ALERT] 255/130843 (51214) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
[ALERT] 255/130843 (51214) : Fatal errors found in configuration.

The fatal error from the logs in the router was:

[ALERT] 255/130843 (51214) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
[ALERT] 255/130843 (51214) : Fatal errors found in configuration.

Expected results:

Either throw a warning and refuse to serve route or else work.

Additional info:
Is there any way to automatically sanity check certificates before they are used in routes? The problem can be rectified by deleting the route.
Comment 2 Ben Bennett 2016-09-27 10:35:40 EDT
Is the cert in the route broken?  If so, can they include the route yaml.

Obviously, if it contains sensitive keys, they shouldn't give it to us.
Comment 18 Ram Ranganathan 2016-10-26 13:44:12 EDT
Defaults set to true with PR: https://github.com/openshift/origin/pull/11218
Comment 19 Troy Dawson 2016-10-27 12:14:07 EDT
This has been merged into ose and is in OSE v3.4.0.16 or newer.
Comment 21 zhaozhanqi 2016-10-27 23:03:18 EDT
Verified this bug on haproxy images (id: 227ebcf6c7d8). the default EXTENDED_VALIDATION is true. and the invalid route will be skip.
Comment 24 errata-xmlrpc 2017-01-18 07:41:47 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0066

Note You need to log in before you can comment on or make changes to this bug.