Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 1511732 - [3.5] extended route validation does not catch newlines in certificate
[3.5] extended route validation does not catch newlines in certificate
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Routing (Show other bugs)
3.5.1
Unspecified Unspecified
urgent Severity urgent
: ---
: 3.5.z
Assigned To: jtanenba
zhaozhanqi
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-11-09 19:30 EST by Steven Walter
Modified: 2018-07-10 22:46 EDT (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Invalid PEM data is left in the route config file during extended validation. Consequence: The router crashes Fix: Sanitize PEM data from route configs Result: Properly catch malformed certificates in extended validation.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-12-14 16:02:32 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Github openshift/ose/pull/926 None None None 2017-11-29 10:41 EST
Red Hat Product Errata RHBA-2017:3438 normal SHIPPED_LIVE OpenShift Container Platform 3.6 and 3.5 bug fix and enhancement update 2017-12-14 20:58:11 EST

  None (edit)
Description Steven Walter 2017-11-09 19:30:51 EST
Description of problem:

In the route definition we saw empty lines:
...
    certificate: |
      -----BEGIN CERTIFICATE-----
...
    key: |
      -----BEGIN PRIVATE KEY-----
...

It is hardly noticeable in YAML format. On the OpenShift master console GUI it can be seen better.
Normally it should be |+ instead of just a | character.
Then the haproxy concatenates the certificates into a PEM file where the empty lines appear, and we think it causes those error messages.

Version-Release number of selected component (if applicable):
image: ose-haproxy-router:v3.5.5.31
OpenShift version: v3.5.5.31.24

How reproducible:
Unconfirmed

Steps to Reproduce:
1. Created route in web console

Actual results:
2017-11-09T09:20:32.957490000Z [ALERT] 312/092032 (127789) : parsing [/var/lib/haproxy/conf/haproxy.config:142] : 'bind 127.0.0.1:10444' : 'crt-list' : error processing line 150 in file '/var/lib/haproxy/conf/cert_config.map' : unable to load SSL certificate from PEM file '/var/lib/haproxy/router/certs/example.pem'.
2017-11-09T09:20:32.957682000Z [ALERT] 312/092032 (127789) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
2017-11-09T09:20:32.957873000Z [ALERT] 312/092032 (127789) : Fatal errors found in configuration.

Expected results:
Non-fatal warning, not accepting the route

Additional info:
Verified oc set env dc/router EXTENDED_VALIDATION=true
Deleting route causes error to go away
Comment 3 Steven Walter 2017-11-09 19:43:09 EST
From Clayton:

Extended validation in 3.5 had gaps.  Most were fixed in 3.6.

You can run the 3.6 diagnostic command against a 3.5 cluster to check whether any routes would fail EV - if they can recreate in a test cluster please have them rerun.

Most of the things fixed were invalid PEM blocks and incomplete content.
Comment 7 Ryan Howe 2017-11-10 13:43:06 EST
To reproduce this I removed one "-" from the -----END CERTIFICATE----

Was only able to reproduce in 3.5 removing a dash from -----END CERTIFICATE----  and only END CERTIFICATE section hit this issue. 


|+  |- and |  all work fine with 3.5 routes. Fatal errors only happened after editing -----END CERTIFICATE----


In 3.6 the router corrected the pem file adding the - back to the route when the pem was added to /var/lib/haproxy/router/certs/, the route object itself still showed the missing -. This did not cause 3.6 router to fail.
Comment 13 Weibin Liang 2017-11-16 15:44:16 EST
PR/926 fixed the problem. Testing passed
Comment 15 zhaozhanqi 2017-11-29 02:54:31 EST
QE did the testing on 3.5.5.31.47

1.  As comment 7 said: |+  |- and |  all work fine
2. if I removed one "-" from the -----END CERTIFICATE----, the route will be marked as 'ExtendedValidationFailed'
Comment 16 Dana Safford 2017-11-30 11:24:41 EST
This situation is now very critical.
I just set the Customer Escalation flag and the am requesting a hotfix (so I flipped that flag too).

Hello Jacob/Rashid

I wrote yesterday in ref Bug 1511732  which is still in Status (Modified) and the following question below from the Action Plan:

1. Are we able to meet customers expectations on being able to deliver an errata fixing this by the 15th of December? If not
2. When is the cut off date on knowing if this expectation cant be achieved?
3. How long would the hotifix take? and when or who can deliver this.

For more info, contact Aaron Ship (Critsit Manager)  aship in irc
Comment 18 zhaozhanqi 2017-11-30 21:20:58 EST
verified this bug according to comment 15
Comment 21 errata-xmlrpc 2017-12-14 16:02:32 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3438

Note You need to log in before you can comment on or make changes to this bug.