Bug 1511732 - [3.5] extended route validation does not catch newlines in certificate
Summary: [3.5] extended route validation does not catch newlines in certificate
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Routing
Version: 3.5.1
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.5.z
Assignee: Jacob Tanenbaum
QA Contact: zhaozhanqi
Depends On:
TreeView+ depends on / blocked
Reported: 2017-11-10 00:30 UTC by Steven Walter
Modified: 2021-06-10 13:32 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Invalid PEM data is left in the route config file during extended validation. Consequence: The router crashes Fix: Sanitize PEM data from route configs Result: Properly catch malformed certificates in extended validation.
Clone Of:
Last Closed: 2017-12-14 21:02:32 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift ose pull 926 0 None None None 2017-11-29 15:41:32 UTC
Red Hat Product Errata RHBA-2017:3438 0 normal SHIPPED_LIVE OpenShift Container Platform 3.6 and 3.5 bug fix and enhancement update 2017-12-15 01:58:11 UTC

Description Steven Walter 2017-11-10 00:30:51 UTC
Description of problem:

In the route definition we saw empty lines:
    certificate: |
      -----BEGIN CERTIFICATE-----
    key: |
      -----BEGIN PRIVATE KEY-----

It is hardly noticeable in YAML format. On the OpenShift master console GUI it can be seen better.
Normally it should be |+ instead of just a | character.
Then the haproxy concatenates the certificates into a PEM file where the empty lines appear, and we think it causes those error messages.

Version-Release number of selected component (if applicable):
image: ose-haproxy-router:v3.5.5.31
OpenShift version: v3.

How reproducible:

Steps to Reproduce:
1. Created route in web console

Actual results:
2017-11-09T09:20:32.957490000Z [ALERT] 312/092032 (127789) : parsing [/var/lib/haproxy/conf/haproxy.config:142] : 'bind' : 'crt-list' : error processing line 150 in file '/var/lib/haproxy/conf/cert_config.map' : unable to load SSL certificate from PEM file '/var/lib/haproxy/router/certs/example.pem'.
2017-11-09T09:20:32.957682000Z [ALERT] 312/092032 (127789) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
2017-11-09T09:20:32.957873000Z [ALERT] 312/092032 (127789) : Fatal errors found in configuration.

Expected results:
Non-fatal warning, not accepting the route

Additional info:
Verified oc set env dc/router EXTENDED_VALIDATION=true
Deleting route causes error to go away

Comment 3 Steven Walter 2017-11-10 00:43:09 UTC
From Clayton:

Extended validation in 3.5 had gaps.  Most were fixed in 3.6.

You can run the 3.6 diagnostic command against a 3.5 cluster to check whether any routes would fail EV - if they can recreate in a test cluster please have them rerun.

Most of the things fixed were invalid PEM blocks and incomplete content.

Comment 7 Ryan Howe 2017-11-10 18:43:06 UTC
To reproduce this I removed one "-" from the -----END CERTIFICATE----

Was only able to reproduce in 3.5 removing a dash from -----END CERTIFICATE----  and only END CERTIFICATE section hit this issue. 

|+  |- and |  all work fine with 3.5 routes. Fatal errors only happened after editing -----END CERTIFICATE----

In 3.6 the router corrected the pem file adding the - back to the route when the pem was added to /var/lib/haproxy/router/certs/, the route object itself still showed the missing -. This did not cause 3.6 router to fail.

Comment 13 Weibin Liang 2017-11-16 20:44:16 UTC
PR/926 fixed the problem. Testing passed

Comment 15 zhaozhanqi 2017-11-29 07:54:31 UTC
QE did the testing on

1.  As comment 7 said: |+  |- and |  all work fine
2. if I removed one "-" from the -----END CERTIFICATE----, the route will be marked as 'ExtendedValidationFailed'

Comment 16 Dana Safford 2017-11-30 16:24:41 UTC
This situation is now very critical.
I just set the Customer Escalation flag and the am requesting a hotfix (so I flipped that flag too).

Hello Jacob/Rashid

I wrote yesterday in ref Bug 1511732  which is still in Status (Modified) and the following question below from the Action Plan:

1. Are we able to meet customers expectations on being able to deliver an errata fixing this by the 15th of December? If not
2. When is the cut off date on knowing if this expectation cant be achieved?
3. How long would the hotifix take? and when or who can deliver this.

For more info, contact Aaron Ship (Critsit Manager)  aship in irc

Comment 18 zhaozhanqi 2017-12-01 02:20:58 UTC
verified this bug according to comment 15

Comment 21 errata-xmlrpc 2017-12-14 21:02:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.