Bug 1312292 - broken SSL definition causes router to stop adding routes
Summary: broken SSL definition causes router to stop adding routes
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 3.1.0
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ---
: ---
Assignee: Ram Ranganathan
QA Contact: zhaozhanqi
URL:
Whiteboard:
: 1328266 (view as bug list)
Depends On:
Blocks: 1267746 1328266 1339502 1339909
TreeView+ depends on / blocked
 
Reported: 2016-02-26 10:47 UTC by Evgheni Dereveanchin
Modified: 2022-08-04 22:20 UTC (History)
10 users (show)

Fixed In Version: atomic-openshift-3.2.0.45-1.git.0.a2ee9db.el7
Doc Type: Bug Fix
Doc Text:
When a bad certificate was configured for a route, HAProxy was unable to reload the configuration. This caused any routes configured after the bad route to never be applied. This bug fix updates the route validation to fail for bad configurations, and as a result this issue no longer occurs.
Clone Of:
: 1328266 1339909 (view as bug list)
Environment:
Last Closed: 2016-06-09 21:10:30 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1221 0 normal SHIPPED_LIVE Red Hat OpenShift Enterprise 3.2 bug fix update 2016-06-10 01:10:20 UTC

Description Evgheni Dereveanchin 2016-02-26 10:47:24 UTC
Description of problem:
When a bogus certificate is entered in a random route, this makes haproxy unable to reload the config. So any routes configured after the bad route are never added an probably no other changes are applied.

Version-Release number of selected component (if applicable):
3.1.1.6

How reproducible:
Always

Steps to Reproduce:
1. create a new app "test1" in a project
2. edit its route and put the following in the "spec" section
  tls:
    termination: edge
    caCertificate: BADVALUE
    certificate: ALSOBAD
    key: NOTAKEY
3. create new app "test2"

Actual results:
"test2" route never appears on the router due to errors related to "test1"

router logs have multiple messages like this (for each reload attempt):

E0226 05:36:28.352895       1 controller.go:84] error reloading router: exit status 1
---
+ config_file=/var/lib/haproxy/conf/haproxy.config
+ pid_file=/var/lib/haproxy/run/haproxy.pid
+ old_pid=
+ haproxy_conf_dir=/var/lib/haproxy/conf
+ for mapfile in '"$haproxy_conf_dir"/*.map'
+ sort -r /var/lib/haproxy/conf/os_edge_http_be.map -o /var/lib/haproxy/conf/os_edge_http_be.map
+ for mapfile in '"$haproxy_conf_dir"/*.map'
+ sort -r /var/lib/haproxy/conf/os_edge_http_expose.map -o /var/lib/haproxy/conf/os_edge_http_expose.map
+ for mapfile in '"$haproxy_conf_dir"/*.map'
+ sort -r /var/lib/haproxy/conf/os_edge_http_redirect.map -o /var/lib/haproxy/conf/os_edge_http_redirect.map
+ for mapfile in '"$haproxy_conf_dir"/*.map'
+ sort -r /var/lib/haproxy/conf/os_http_be.map -o /var/lib/haproxy/conf/os_http_be.map
+ for mapfile in '"$haproxy_conf_dir"/*.map'
+ sort -r /var/lib/haproxy/conf/os_reencrypt.map -o /var/lib/haproxy/conf/os_reencrypt.map
+ for mapfile in '"$haproxy_conf_dir"/*.map'
+ sort -r /var/lib/haproxy/conf/os_sni_passthrough.map -o /var/lib/haproxy/conf/os_sni_passthrough.map
+ for mapfile in '"$haproxy_conf_dir"/*.map'
+ sort -r /var/lib/haproxy/conf/os_tcp_be.map -o /var/lib/haproxy/conf/os_tcp_be.map
+ '[' -f /var/lib/haproxy/run/haproxy.pid ']'
+ old_pid=20412
+ '[' -n 20412 ']'
+ /usr/sbin/haproxy -f /var/lib/haproxy/conf/haproxy.config -p /var/lib/haproxy/run/haproxy.pid -sf 20412
[ALERT] 056/053628 (20502) : parsing [/var/lib/haproxy/conf/haproxy.config:111] : 'bind 127.0.0.1:10444' : unable to load SSL private key from PEM file '/var/lib/containers/router/certs/10_test1.pem'.
[ALERT] 056/053628 (20502) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
[ALERT] 056/053628 (20502) : Fatal errors found in configuration.

Expected results:
Bad route is ignored, other routes are not affected

Additional info:
As soon as route "test1" is removed, "test2" is added

It looks like the router will take the values, construct a PEM file out of them and blindly try to load it. We need to check that each individual PEM is actually valid and only add the good ones to the config.

Comment 1 Eric Paris 2016-02-26 15:39:51 UTC
https://github.com/openshift/origin/issues/7403

Comment 5 Ram Ranganathan 2016-04-07 18:55:49 UTC
Associated PRs - 2 options:
  1. https://github.com/openshift/origin/pull/8353  - validates generated haproxy
     config.
  2. https://github.com/openshift/origin/pull/8366 - validates the certificate.

Comment 9 Ben Bennett 2016-04-20 13:36:02 UTC
*** Bug 1328266 has been marked as a duplicate of this bug. ***

Comment 10 Ben Bennett 2016-04-21 15:34:31 UTC
Online Beta (for other reasons) is disallowing custom DNS names.  Because of that they will not need custom SSL certs.  This will be needed for the Online GA.

Comment 20 zhaozhanqi 2016-05-26 06:54:52 UTC
hi,Ram

When creating edge route without 'CACertificate'. this should be pass since it is a option keyword. but now failed


eg:
oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/routing/edge/route_edge.json

# oc get route secured-edge-route
NAME                 HOST/PORT                  PATH      SERVICE            TERMINATION   LABELS
secured-edge-route   ExtendedValidationFailed             service-unsecure   edge 



# oc logs router-xxx

E0526 02:33:26.614271       1 extended_validator.go:62] Skipping route zzhao/secured-edge-route due to invalid configuration: 
  - spec.tls.certificate: Invalid value: "-----BEGIN CERTIFICATE-----\nMIIDEzCCAfugAwIBAgIBDjANBgkqhkiG9w0BAQUFADCBoTELMAkGA1UEBhMCVVMx\nCzAJBgNVBAgMAlNDMRUwEwYDVQQHDAxEZWZhdWx0IENpdHkxHDAaBgNVBAoME0Rl\nZmF1bHQgQ29tcGFueSBMdGQxEDAOBgNVBAsMB1Rlc3QgQ0ExGjAYBgNVBAMMEXd3\ndy5leGFtcGxlY2EuY29tMSIwIAYJKoZIhvcNAQkBFhNleGFtcGxlQGV4YW1wbGUu\nY29tMB4XDTE2MDUyNTA4MTI1OFoXDTI2MDUyMzA4MTI1OFowbTEWMBQGA1UEAwwN\nKi5leGFtcGxlLmNvbTELMAkGA1UECAwCQkoxCzAJBgNVBAYTAkNOMR8wHQYJKoZI\nhvcNAQkBFhBibWVuZ0ByZWRoYXQuY29tMQswCQYDVQQKDAJSSDELMAkGA1UECwwC\nT1MwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAKARzcO20lcg+qDfe9xAEde2\ndbUij9LWclX/VfGp0Xhydzf/ODmL5c/Iy/cxgKvoo7DZTuPYsXrS7z9uuLI4S4st\nqj/n21KrYIwDdIXvaOc6CTTQxqUE20LZ08LkR8BLra4Lbn7lhlRgayOMClfdUL47\n4Cv0S4OlDS07idbD1kXzAgMBAAGjDTALMAkGA1UdEwQCMAAwDQYJKoZIhvcNAQEF\nBQADggEBAMIRge8dXWyZJsve1aycniBxdyWUMoM9tPBDvfZAlLLDWubuoaEXLojy\n3wGHGzDGOWrvYHwmPfWDNf+IlrxetiIOiXxKfGtTsOuqdJCcbz3y70WiICziX5m7\ndqeoGfnGhf6Ys6/L0/hecHLxw86RlhlJnH7W0eB3qeT7vc7ytDxcRFlvhFxgAD3O\nF1H8XKJWuaghzus0rDPlQviEPYkYfmUBMNLl/dbWEVNV3wCakaaMoYg12y4p1Rd4\npgW3DwXWYbnAX5K1TbtuALWvmiOIcGbtLTwKqI6pdPJx4bo+zbwOuo/Q9lbjRcZG\nAErbDKA4OfpTCrpu/qADXfnJVGCuWUo=\n-----END CERTIFICATE-----\n": error verifying certificate: x509: certificate signed by unknown authority
E0526 02:33:26.614317       1 controller.go:85] invalid route configuration

Comment 21 Ram Ranganathan 2016-05-26 21:39:09 UTC
@zhaozhanqi, if the certificate is not signed by one of the system trusted CAs, you will need to pass the CACert value as well or add it to the system trusted CAs.
That is the extended validation we do now.

Since you are probably generating the cert (self-signed or signed via a locally generated CA), add that CA to the route if its not a well-known CA. 
Or you can add it to the system trusted - copy the cert to I think /etc/pki/ca-trust/source/anchors and run update-ca-trust enable && update-ca-trust extract
(Caveat: haven't tried that above set but see: man 8 update-ca-trust )

Comment 22 zhaozhanqi 2016-05-27 03:29:19 UTC
@Ram

thanks your confirm..

then this bug has been fixed with following version.

oc v3.2.0.45
kubernetes v1.2.0-36-g4a3f9c5

haproxy images:
openshift3/ose-haproxy-router          v3.2.0.45           3a931abad1e2

Comment 24 errata-xmlrpc 2016-06-09 21:10:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1221


Note You need to log in before you can comment on or make changes to this bug.