Bug 1501133 - [H1FhCI1I]HSTS for the route is not working well due to the format is not correct
Summary: [H1FhCI1I]HSTS for the route is not working well due to the format is not cor...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 3.7.0
Hardware: All
OS: All
high
high
Target Milestone: ---
: 3.7.z
Assignee: Phil Cameron
QA Contact: zhaozhanqi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-10-12 06:56 UTC by zhaozhanqi
Modified: 2022-08-04 22:20 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: no doc changes Consequence: Fix: Result:
Clone Of:
Environment:
Last Closed: 2017-11-28 22:16:57 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:3188 0 normal SHIPPED_LIVE Moderate: Red Hat OpenShift Container Platform 3.7 security, bug, and enhancement update 2017-11-29 02:34:54 UTC

Description zhaozhanqi 2017-10-12 06:56:45 UTC
Description of problem:
see https://www.haproxy.com/blog/haproxy-and-http-strict-transport-security-hsts-header-in-http-redirects/

for haproxy 1.5. 
the format is 'http-response set-header Strict-Transport-Security max-age=16000000;\ includeSubDomains;\ preload;'
for haproxy 1.6:
the format is 'http-response set-header Strict-Transport-Security max-age=16000000; includeSubDomains; preload;'

The current version of haproxy is still using 1.5
$ haproxy -v
HA-Proxy version 1.5.18 2016/05/10

but the format of in this PR is for haproxy 1.6
https://github.com/openshift/origin/pull/16544

Version-Release number of selected component (if applicable):
v3.7.0-0.143.2
registry.ops.openshift.com/openshift3/ose-haproxy-router               v3.7.0-0.143.2      362e0d69cf26
How reproducible:
always

Steps to Reproduce:
1. Create app and route
2. Set the HSTS annotate for the route
   oc annotate route edge1 haproxy.router.openshift.io/hsts_header='max-age=31536000;\includeSubDomains;\preload' 
3. curl the route
   curl --head $route
4. 

Actual results:

step3 cannot see the 'Strict-Transport-Security' in the response header

Expected results:


Additional info:

Comment 1 Ben Bennett 2017-10-13 18:10:33 UTC
According to the HSTS RFC (and the referenced HTTP spec) the spaces are optional.

We choose not to allow spaces so that we could be compatible with both formats.  In order to get the HSTS header to show up, you need to say:
  oc annotate route edge1 haproxy.router.openshift.io/hsts_header='max-age=31536000;includeSubDomains;preload'

The \s in your annotate are going through and appearing as \i and \p... so the behavior is odd.

Comment 2 Ben Bennett 2017-10-13 20:10:04 UTC
Sorry, my last sentence was unclear.  I mean that it isn't working because \i and \p are going through.

We need to make it clear in the docs what precisely is supported, but I think the behavior we have without the spaces is what we want to allow.

Comment 3 zhaozhanqi 2017-10-16 01:23:14 UTC
I did the testing using the following steps. 

1. Create app and edge route(allow http)
2. Set the HSTS annotate for the route
   oc annotate route edge1 haproxy.router.openshift.io/hsts_header='max-age=31536000;includeSubDomains;preload'
3. Access the route in the chrome browser with 'https://$route'
4. Access the route in the chrome browser with 'http://$route'

step 4 did NOT refer to 'https'

Comment 4 Ben Bennett 2017-10-17 15:41:39 UTC
@zhaozhanqi: Can you clarify what is wrong in comment 3?

We are getting back the HSTS header when using http (which is a bug).  But if that is the only problem, let's open a new bug for that since it is a different symptom than the one this bug is about.

PR https://github.com/openshift/origin/pull/16888 addresses the http case.

Comment 5 zhaozhanqi 2017-10-18 01:39:00 UTC
@Ben Bennett

in Comment 3:

step 2 is using 'preload', So when accessing the route using the chrome browser, the route will be added record to the browser, and if accessing the route whatever 'http' or 'https' next time, it will be force using 'https', but step 4 is still using 'http'.

Comment 6 zhaozhanqi 2017-10-18 01:50:08 UTC
@Ben Bennett

I'm not sure if the description is clear,Please correct me if I'm wrong.  thanks

Comment 7 openshift-github-bot 2017-10-18 06:17:21 UTC
Commits pushed to master at https://github.com/openshift/origin

https://github.com/openshift/origin/commit/c8a9af0583f612f0739bc737b5096ea93c792001
Router - hsts for "edge" or "reencrypt" only

Suppress Strict-Transport-Security header for http requests
It is only emitted for https.

bug 1501133
https://bugzilla.redhat.com/show_bug.cgi?id=1501133
see comment #3

https://github.com/openshift/origin/commit/91ca5a49ead95b4b7a7701b4412c193abdd19f6e
Merge pull request #16888 from pecameron/bz1501133

Automatic merge from submit-queue (batch tested with PRs 16888, 16911, 16913, 16904).

Router - hsts for "edge" or "reencrypt" only

Suppress Strict-Transport-Security header for http requests
It is only emitted for https.

bug 1501133
https://bugzilla.redhat.com/show_bug.cgi?id=1501133
see comment #3

Comment 8 zhaozhanqi 2017-10-19 01:07:50 UTC
@Phil Cameron

Since I saw you update the bug to 'MODIFIED', Could you help check the Comment 5 again. I don't think the #PR 16888 is fixing this issue.

Comment 9 Phil Cameron 2017-10-19 14:31:00 UTC
zzhao

Check the haproxy-config.template file in the router pod for the fix in 16888. It is a simple code change. The merge happened around 10/18/2017 02:17 EDT. Make sure your router pod was created after that.

My local testing verified the fix. It should also be working for you.

Comment 11 zhaozhanqi 2017-10-23 08:05:11 UTC
@Phil Cameron

I checked the PR 16888 and did testing on v3.7.0-0.173.0, it works on edge and reencrype route

There is another issue:

When I set the HSTS annotate for the route
   oc annotate route edge1 haproxy.router.openshift.io/hsts_header='max-age=31536000;includeSubDomains;preload'

the 'preload' cannot load the route into browser(chrome) hsts list. 
As my understanding, the 'preload' feature is : the route will be added the browser (chrome) hsts list [1] when accessing the route, just like 'www.google.com'.

[1] chrome://net-internals/#hsts

Comment 12 Phil Cameron 2017-10-23 13:57:23 UTC
The requirement here is to return the valid header in a response.
What a user that receives the header does with it is beyond the scope of this feature.

Comment 13 zhaozhanqi 2017-10-24 01:44:03 UTC
@Phil Cameron

from Comment 12, I'm confusing about this feature. Could you help explain more about 'preload',How to use this function for normal user. I do not think only return valid header in response is the correct test result for me. here I still use 'www.google.com' as the example:  whatever I input 'http://www.google.com' or 'https://www.google.com'. both of them will refer to 'https://www.google.com'. even if I deleted the domain in `chrome://net-internals/#hsts`.  When I accessing 'http://www.google.com' again. it will be added to preload list in `chrome://net-internals/#hsts`. 

And I think our route will also take same purpose. Please correct me if I'm wrong. Thanks.

Comment 14 Ben Bennett 2017-10-24 15:01:13 UTC
@zhaozhanqi: Preload is for telling google that you want to be included in the preload list.  https://scotthelme.co.uk/hsts-preloading/

I believe we are doing the right thing and it will not show on that page until the site is added to the browser lists.

Comment 15 Ben Bennett 2017-10-24 15:04:39 UTC
@pcameron: We should check that the browsers are honoring the format we have.  Per the spec it is legit, but we need to be sure Chrome and Mozilla use it.

Comment 16 Phil Cameron 2017-10-24 15:17:41 UTC
zzhao, Where do we go from here? 

Chrome has a permanent preload list:
https://hstspreload.org/
https://www.chromium.org/hsts
Other browsers and applications may maintain their own permanent lists as well.

In general we can't control what any of the software that receives a Strict Transport Security header with preload does with it. The best that we can do is include preload into the header when directed by the annotation.

HSTS is just a hint that reduces network traffic by directing the application to always use https with the site (thus eliminating the redirect that would otherwise occur) and reduces a security vulnerability by limiting what an eavesdropper can discover.

These sites have some additional informaton:
https://www.owasp.org/index.php/HTTP_Strict_Transport_Security_Cheat_Sheet
https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security
https://tools.ietf.org/html/rfc6797

Comment 17 Phil Cameron 2017-10-24 15:25:56 UTC
zzhao, To test this, set an annotation on the route with a max-age, browse to the route and verify the HSTS header. Future http requests to the route from the browser should be converted to https. Check both Chrome and firefox.

Comment 18 zhaozhanqi 2017-10-25 06:24:11 UTC
(In reply to Phil Cameron from comment #17)
> zzhao, To test this, set an annotation on the route with a
> max-age, browse to the route and verify the HSTS header. Future http
> requests to the route from the browser should be converted to https. Check
> both Chrome and firefox.

yeah, the steps same to me, but the issue is when I'm using 'http' requests the route from the browser after setting the annotation on the route with 'max-age=31536000', it did NOT convert to 'https'. will return 503 

the following the haproxy.config info of my route

backend be_edge_http:z1:edge1
    
  mode http
  option redispatch
  option forwardfor
  balance leastconn 

  timeout check 5000ms
  http-request set-header X-Forwarded-Host %[req.hdr(host)]
  http-request set-header X-Forwarded-Port %[dst_port]
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header Forwarded for=%[src];host=%[req.hdr(host)];proto=%[req.hdr(X-Forwarded-Proto)]
  cookie 1c5965c4941d255f5b5a0174036c8510 insert indirect nocache httponly secure
  http-response set-header Strict-Transport-Security max-age=31536000
  server pod:header-test-1-86xql:header-test-insecure:10.2.4.28:8080 10.2.4.28:8080 cookie 9ec3a5646fa597b6c44479db89048f91 weight 256

Comment 19 Phil Cameron 2017-10-25 13:41:22 UTC
zzhao

To get the Strict-Transport-Security header into the browser you need to make a https request using the route. After that http requests to the same route should be changed by the browser into https requests. If the browser doesn't do this it is a browser problem, not an openshift router problem. Not every version of every browser supports hsts. 

https://my.route.com
returns Strict-Transport-Security header max-age=31536000
http://my.route.com
should be changed by the browser into 
https://my.route.com

Comment 20 Rajat Chopra 2017-10-25 22:51:44 UTC
Unfortunately the browser will not respect hsts headers if the certificate authority is not trusted. So, to test that it does work on real certificates:
1. Create a self signed chain
2. Import the rootCA in the browser's cert authority
3. Use the new key/cert/caCert in the route
4. Annotate the route
5. Reach https://route.host/ from browser (this should work without browser complaining about untrusted CA)
6. Try to reach http://route.host/ again from the same browser, it should get redirected to https://route.host/

Confirmed with Chrome Version 44.0.2403.130 (64-bit) on fc22 and OpenShift router running with version: v3.7.0-alpha.1+f65d7c8-937-dirty (one with the fix commit mentioned in comment#10)

Comment 21 zhaozhanqi 2017-10-26 08:35:54 UTC
@Rajat Chopra

Thanks for your comment which really resolve my issue. Indeed it's the browser did not import the CA.

I did the testing according to your steps on Firefox 53.  it works well

but still not working on chrome(version 60). it's always 'Not secure' even if the CA was imported. I guess it's the chrome issue and did some research. but still cannot work. 

Anyway. many thanks for your comment.

Comment 24 errata-xmlrpc 2017-11-28 22:16:57 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188


Note You need to log in before you can comment on or make changes to this bug.