Bug 1379701 - Customer cert for route caused fatal error in haproxy router
Summary: Customer cert for route caused fatal error in haproxy router
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 3.2.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Ram Ranganathan
QA Contact: zhaozhanqi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-09-27 13:24 UTC by Steven Walter
Modified: 2022-08-04 22:20 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: By default extended certificate validation was not enabled so bad certs could crash the router. Consequence: Bad certificates in routes could crash the router. Fix: We changed the default in 'oadm router' to turn on extended validation when a router is created. Result: Bad certificates are caught and the route they are associated with is not used (and an appropriate status is set on it)
Clone Of:
Environment:
Last Closed: 2017-01-18 12:41:47 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Origin (Github) 11218 0 None None None 2016-11-30 13:13:11 UTC
Red Hat Product Errata RHBA-2017:0066 0 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.4 RPM Release Advisory 2017-01-18 17:23:26 UTC

Description Steven Walter 2016-09-27 13:24:01 UTC
Description of problem:

Using an improperly formatted cert for a route, using a key that had a passphrase on it, caused an outage in the router for all customers.


It appears the offending cert causes issues when re-encrypting the route on the F5. When the certificate was created they had a passphrase on it, which needs to be removed for it to work. Apparently this caused a fatal error which required deleting the route to recover from. One customer using an improperly formatted certificate caused a router outage for the whole environment.



Version-Release number of selected component (if applicable):
# openshift version
openshift v3.2.1.4-1-g1864c8f
kubernetes v1.2.0-36-g4a3f9c5
etcd 2.2.5


How reproducible:
Unverified


Actual results:

The application which had the bad cert showed these logs:

[ALERT] 255/130843 (51214) : parsing [/var/lib/haproxy/conf/haproxy.config:112] : 'bind 127.0.0.1:10444' :
  unable to load SSL private key from PEM file '/var/lib/containers/router/certs/example.pem'.
  unable to load SSL private key from PEM file '/var/lib/containers/router/certs/example.pem'.
  unable to load SSL private key from PEM file '/var/lib/containers/router/certs/example.pem'.
[ALERT] 255/130843 (51214) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
[ALERT] 255/130843 (51214) : Fatal errors found in configuration.

The fatal error from the logs in the router was:

[ALERT] 255/130843 (51214) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
[ALERT] 255/130843 (51214) : Fatal errors found in configuration.

Expected results:

Either throw a warning and refuse to serve route or else work.

Additional info:
Is there any way to automatically sanity check certificates before they are used in routes? The problem can be rectified by deleting the route.

Comment 2 Ben Bennett 2016-09-27 14:35:40 UTC
Is the cert in the route broken?  If so, can they include the route yaml.

Obviously, if it contains sensitive keys, they shouldn't give it to us.

Comment 18 Ram Ranganathan 2016-10-26 17:44:12 UTC
Defaults set to true with PR: https://github.com/openshift/origin/pull/11218

Comment 19 Troy Dawson 2016-10-27 16:14:07 UTC
This has been merged into ose and is in OSE v3.4.0.16 or newer.

Comment 21 zhaozhanqi 2016-10-28 03:03:18 UTC
Verified this bug on haproxy images (id: 227ebcf6c7d8). the default EXTENDED_VALIDATION is true. and the invalid route will be skip.

Comment 24 errata-xmlrpc 2017-01-18 12:41:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0066


Note You need to log in before you can comment on or make changes to this bug.