Bug 1312292 - broken SSL definition causes router to stop adding routes
broken SSL definition causes router to stop adding routes
Product: OpenShift Container Platform
Classification: Red Hat
Component: Routing (Show other bugs)
Unspecified Unspecified
urgent Severity high
: ---
: ---
Assigned To: Ram Ranganathan
: 1328266 (view as bug list)
Depends On:
Blocks: 1267746 1328266 1339502 1339909
  Show dependency treegraph
Reported: 2016-02-26 05:47 EST by Evgheni Dereveanchin
Modified: 2017-03-08 13 EST (History)
10 users (show)

See Also:
Fixed In Version: atomic-openshift-
Doc Type: Bug Fix
Doc Text:
When a bad certificate was configured for a route, HAProxy was unable to reload the configuration. This caused any routes configured after the bad route to never be applied. This bug fix updates the route validation to fail for bad configurations, and as a result this issue no longer occurs.
Story Points: ---
Clone Of:
: 1328266 1339909 (view as bug list)
Last Closed: 2016-06-09 17:10:30 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Evgheni Dereveanchin 2016-02-26 05:47:24 EST
Description of problem:
When a bogus certificate is entered in a random route, this makes haproxy unable to reload the config. So any routes configured after the bad route are never added an probably no other changes are applied.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. create a new app "test1" in a project
2. edit its route and put the following in the "spec" section
    termination: edge
    caCertificate: BADVALUE
    certificate: ALSOBAD
    key: NOTAKEY
3. create new app "test2"

Actual results:
"test2" route never appears on the router due to errors related to "test1"

router logs have multiple messages like this (for each reload attempt):

E0226 05:36:28.352895       1 controller.go:84] error reloading router: exit status 1
+ config_file=/var/lib/haproxy/conf/haproxy.config
+ pid_file=/var/lib/haproxy/run/haproxy.pid
+ old_pid=
+ haproxy_conf_dir=/var/lib/haproxy/conf
+ for mapfile in '"$haproxy_conf_dir"/*.map'
+ sort -r /var/lib/haproxy/conf/os_edge_http_be.map -o /var/lib/haproxy/conf/os_edge_http_be.map
+ for mapfile in '"$haproxy_conf_dir"/*.map'
+ sort -r /var/lib/haproxy/conf/os_edge_http_expose.map -o /var/lib/haproxy/conf/os_edge_http_expose.map
+ for mapfile in '"$haproxy_conf_dir"/*.map'
+ sort -r /var/lib/haproxy/conf/os_edge_http_redirect.map -o /var/lib/haproxy/conf/os_edge_http_redirect.map
+ for mapfile in '"$haproxy_conf_dir"/*.map'
+ sort -r /var/lib/haproxy/conf/os_http_be.map -o /var/lib/haproxy/conf/os_http_be.map
+ for mapfile in '"$haproxy_conf_dir"/*.map'
+ sort -r /var/lib/haproxy/conf/os_reencrypt.map -o /var/lib/haproxy/conf/os_reencrypt.map
+ for mapfile in '"$haproxy_conf_dir"/*.map'
+ sort -r /var/lib/haproxy/conf/os_sni_passthrough.map -o /var/lib/haproxy/conf/os_sni_passthrough.map
+ for mapfile in '"$haproxy_conf_dir"/*.map'
+ sort -r /var/lib/haproxy/conf/os_tcp_be.map -o /var/lib/haproxy/conf/os_tcp_be.map
+ '[' -f /var/lib/haproxy/run/haproxy.pid ']'
+ old_pid=20412
+ '[' -n 20412 ']'
+ /usr/sbin/haproxy -f /var/lib/haproxy/conf/haproxy.config -p /var/lib/haproxy/run/haproxy.pid -sf 20412
[ALERT] 056/053628 (20502) : parsing [/var/lib/haproxy/conf/haproxy.config:111] : 'bind' : unable to load SSL private key from PEM file '/var/lib/containers/router/certs/10_test1.pem'.
[ALERT] 056/053628 (20502) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
[ALERT] 056/053628 (20502) : Fatal errors found in configuration.

Expected results:
Bad route is ignored, other routes are not affected

Additional info:
As soon as route "test1" is removed, "test2" is added

It looks like the router will take the values, construct a PEM file out of them and blindly try to load it. We need to check that each individual PEM is actually valid and only add the good ones to the config.
Comment 1 Eric Paris 2016-02-26 10:39:51 EST
Comment 5 Ram Ranganathan 2016-04-07 14:55:49 EDT
Associated PRs - 2 options:
  1. https://github.com/openshift/origin/pull/8353  - validates generated haproxy
  2. https://github.com/openshift/origin/pull/8366 - validates the certificate.
Comment 9 Ben Bennett 2016-04-20 09:36:02 EDT
*** Bug 1328266 has been marked as a duplicate of this bug. ***
Comment 10 Ben Bennett 2016-04-21 11:34:31 EDT
Online Beta (for other reasons) is disallowing custom DNS names.  Because of that they will not need custom SSL certs.  This will be needed for the Online GA.
Comment 20 zhaozhanqi 2016-05-26 02:54:52 EDT

When creating edge route without 'CACertificate'. this should be pass since it is a option keyword. but now failed

oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/routing/edge/route_edge.json

# oc get route secured-edge-route
NAME                 HOST/PORT                  PATH      SERVICE            TERMINATION   LABELS
secured-edge-route   ExtendedValidationFailed             service-unsecure   edge 

# oc logs router-xxx

E0526 02:33:26.614271       1 extended_validator.go:62] Skipping route zzhao/secured-edge-route due to invalid configuration: 
  - spec.tls.certificate: Invalid value: "-----BEGIN CERTIFICATE-----\nMIIDEzCCAfugAwIBAgIBDjANBgkqhkiG9w0BAQUFADCBoTELMAkGA1UEBhMCVVMx\nCzAJBgNVBAgMAlNDMRUwEwYDVQQHDAxEZWZhdWx0IENpdHkxHDAaBgNVBAoME0Rl\nZmF1bHQgQ29tcGFueSBMdGQxEDAOBgNVBAsMB1Rlc3QgQ0ExGjAYBgNVBAMMEXd3\ndy5leGFtcGxlY2EuY29tMSIwIAYJKoZIhvcNAQkBFhNleGFtcGxlQGV4YW1wbGUu\nY29tMB4XDTE2MDUyNTA4MTI1OFoXDTI2MDUyMzA4MTI1OFowbTEWMBQGA1UEAwwN\nKi5leGFtcGxlLmNvbTELMAkGA1UECAwCQkoxCzAJBgNVBAYTAkNOMR8wHQYJKoZI\nhvcNAQkBFhBibWVuZ0ByZWRoYXQuY29tMQswCQYDVQQKDAJSSDELMAkGA1UECwwC\nT1MwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAKARzcO20lcg+qDfe9xAEde2\ndbUij9LWclX/VfGp0Xhydzf/ODmL5c/Iy/cxgKvoo7DZTuPYsXrS7z9uuLI4S4st\nqj/n21KrYIwDdIXvaOc6CTTQxqUE20LZ08LkR8BLra4Lbn7lhlRgayOMClfdUL47\n4Cv0S4OlDS07idbD1kXzAgMBAAGjDTALMAkGA1UdEwQCMAAwDQYJKoZIhvcNAQEF\nBQADggEBAMIRge8dXWyZJsve1aycniBxdyWUMoM9tPBDvfZAlLLDWubuoaEXLojy\n3wGHGzDGOWrvYHwmPfWDNf+IlrxetiIOiXxKfGtTsOuqdJCcbz3y70WiICziX5m7\ndqeoGfnGhf6Ys6/L0/hecHLxw86RlhlJnH7W0eB3qeT7vc7ytDxcRFlvhFxgAD3O\nF1H8XKJWuaghzus0rDPlQviEPYkYfmUBMNLl/dbWEVNV3wCakaaMoYg12y4p1Rd4\npgW3DwXWYbnAX5K1TbtuALWvmiOIcGbtLTwKqI6pdPJx4bo+zbwOuo/Q9lbjRcZG\nAErbDKA4OfpTCrpu/qADXfnJVGCuWUo=\n-----END CERTIFICATE-----\n": error verifying certificate: x509: certificate signed by unknown authority
E0526 02:33:26.614317       1 controller.go:85] invalid route configuration
Comment 21 Ram Ranganathan 2016-05-26 17:39:09 EDT
@zhaozhanqi, if the certificate is not signed by one of the system trusted CAs, you will need to pass the CACert value as well or add it to the system trusted CAs.
That is the extended validation we do now.

Since you are probably generating the cert (self-signed or signed via a locally generated CA), add that CA to the route if its not a well-known CA. 
Or you can add it to the system trusted - copy the cert to I think /etc/pki/ca-trust/source/anchors and run update-ca-trust enable && update-ca-trust extract
(Caveat: haven't tried that above set but see: man 8 update-ca-trust )
Comment 22 zhaozhanqi 2016-05-26 23:29:19 EDT

thanks your confirm..

then this bug has been fixed with following version.

oc v3.2.0.45
kubernetes v1.2.0-36-g4a3f9c5

haproxy images:
openshift3/ose-haproxy-router          v3.2.0.45           3a931abad1e2
Comment 24 errata-xmlrpc 2016-06-09 17:10:30 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.