Bug 1486297 - [RFE] Allow custom configuration for HSTS settings
Summary: [RFE] Allow custom configuration for HSTS settings
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Settings
Version: 6.2.11
Hardware: Unspecified
OS: Unspecified
urgent
high vote
Target Milestone: 6.4.0
Assignee: Tomer Brisker
QA Contact: Radovan Drazny
URL:
Whiteboard:
: 1302442 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-08-29 12:13 UTC by Evgeni Golov
Modified: 2019-11-05 23:07 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-10-16 15:28:38 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:2927 None None None 2018-10-16 15:30:14 UTC
Foreman Issue Tracker 22165 'High' 'Closed' 'Allow custom configuration for HSTS settings' 2019-11-26 15:12:51 UTC
Red Hat Bugzilla 1302442 'urgent' 'CLOSED' '[RFE] Allow viewing of custom products over http with the standard HSTS setup.' 2019-11-26 15:12:55 UTC

Internal Links: 1302442

Description Evgeni Golov 2017-08-29 12:13:19 UTC
Description of problem:
Foreman/Satellite currently unconditionally sets the HSTS header to "Strict-Transport-Security: max-age=631152000; includeSubdomains", see https://github.com/theforeman/foreman/blob/develop/config/initializers/secure_headers.rb

While generally a good idea, HSTS has the issue that browsers will refuse to talk unencrypted to the Satellite at all after seeing the HSTS header once.

However, we want /pub/ and /pulp/repos/…/custom/ to be available via HTTP too.
This is not a problem for yum/dnf/wget/curl, as those neither implement nor ever see the HSTS header from Foreman, but regular users quite often want to browse the repos with their browsers and that is not possible with HSTS on (as /pulp/repos/ requires a client cert from the Katello CA when accessed via HTTPS).

As HSTS works based on domains/hostnames, you cannot exclude /pub/ and /pulp/repos/ from it.

The best solution that comes into my mind would be allowing to disable and/or change the HSTS settings, so that the users at least can decide if they want HSTS or not (but still leave it on by default).

Version-Release number of selected component (if applicable):
6.2.11

How reproducible:
100%

Steps to Reproduce:
1. curl -I https://sat.example.com

Actual results:
Strict-Transport-Security: max-age=631152000; includeSubdomains


Expected results:
Depending on the settings

Additional info:
The error you see in Firefox is:
An error occurred during a connection to sat.example.com. SSL peer was unable to negotiate an acceptable set of security parameters. Error code: SSL_ERROR_HANDSHAKE_FAILURE_ALERT

The error you see in Chrome is:
This site can’t provide a secure connection sat.example.com didn’t accept your login certificate, or one may not have been provided.
Try contacting the system admin.
ERR_BAD_SSL_CLIENT_AUTH_CERT

Comment 2 Craig Donnelly 2017-08-29 20:14:19 UTC
Evgeni, I have a couple of questions about this.

First, I wanted to note that on my Satellite 6.2.11 instance, I can already access /pub over http (and im sure custom products marked as 'insecure' can as well)- which I don't believe was your point, due to the HSTS issues.

I am affected by HSTS on one of my Satellites that I have deployed with expired certs, and I have to manually remove it from the blacklist of hosts my browser refuses to communicate with.

Basically, I wanted to confirm what we are actually doing by disabling this item, and ensure that it doesn't actually open up anything content-wise in a fashion that we wouldn't expect.

I don't have a lot of understanding about what HSTS is other than some security measure, and I was not aware that we were able to control it server side prior to reading this BZ, so sorry if any of this doesn't make sense.

So long as this doesn't affect public accessibility of RH content from Satellite, I am behind the optional disabling.

Comment 3 Evgeni Golov 2017-08-30 06:57:44 UTC
Hi Craig,

(In reply to Craig Donnelly from comment #2)

> First, I wanted to note that on my Satellite 6.2.11 instance, I can already
> access /pub over http (and im sure custom products marked as 'insecure' can
> as well)- which I don't believe was your point, due to the HSTS issues.

Does that Satellite have valid certs and are you running browser that supports HSTS? For Firefox it only started with v55.

> I am affected by HSTS on one of my Satellites that I have deployed with
> expired certs, and I have to manually remove it from the blacklist of hosts
> my browser refuses to communicate with.

Wouldn't fixing the cert be easier? :)

> Basically, I wanted to confirm what we are actually doing by disabling this
> item, and ensure that it doesn't actually open up anything content-wise in a
> fashion that we wouldn't expect.
> 
> I don't have a lot of understanding about what HSTS is other than some
> security measure, and I was not aware that we were able to control it server
> side prior to reading this BZ, so sorry if any of this doesn't make sense.
> 
> So long as this doesn't affect public accessibility of RH content from
> Satellite, I am behind the optional disabling.

Let me try to explain the various bits involved:

On the Satellite side:

We serve /pub via HTTP and HTTPS and it's up for the user (and their browser) to decide which version to take. HTTP is required so that you can do smth like 'yum install http://sat6/pub/katello-ca-consumer-latest.rpm' which would fail over HTTPS as yum does not yet know the certificate (we have an interesting workaround for that in bootstrap: https://github.com/Katello/katello-client-bootstrap/blob/master/bootstrap.py#L234-L245, but that's nothing you want to do by hand).

We serve /pulp/repos with ALL repos in there via HTTPS, which requires you to authenticate with a client-certificate, which ensures you have the right to access the repos. = The content is secured on the server side.

We serve /pulp/repos with only "serve via HTTP" repos via HTTP, which allows you to access custom products via HTTP if you want to. Red Hat repos cannot be enabled to be "serve via HTTP".

On the Browser side:

A browser that sees a HSTS header from a server that also presents a valid certificate, remembers that this server has to be accessed via HTTPS (and the various HSTS settings the server sends). Any further tries to reach that server via HTTP will be automatically redirected to HTTPS. Any further tries to reach that server via HTTPS  while the server presents an invalid certificate (expired, unsigned, whatever) will be refused. This is a browser-side security feature.

This feature also means that the browser will automatically redirect HTTP requests for /pub and /pulp/repos.
As /pub is freely visible via HTTPS, you don't notice any problem unless you copy&paste a link to a client that is about to be registered and thus does not trust the cert yet.
As /pulp/repos is secured with a client-certificate-check on HTTPS, you can't browse the repo-contents anymore from your browser.

With my proposal we are only allowing toggling the HSTS settings, so how the browser shall behave. The content-security for RH repos is still in place and untouched.

Hope this makes sense and helps understanding the issue and the idea.
Evgeni

Comment 4 Tomer Brisker 2018-01-07 08:48:13 UTC
Created redmine issue http://projects.theforeman.org/issues/22165 from this bug

Comment 5 pm-sat@redhat.com 2018-01-07 11:01:49 UTC
Upstream bug assigned to tbrisker@redhat.com

Comment 6 pm-sat@redhat.com 2018-01-07 11:01:53 UTC
Upstream bug assigned to tbrisker@redhat.com

Comment 7 Bryan Kearney 2018-01-08 14:18:42 UTC
Verification steps from https://bugzilla.redhat.com/show_bug.cgi?id=1302442


1) Using a current chrome or firefox, I would like to create custom product which is exposed over http. I would then like to be able to navigate to the url from the publish page, and see the contents in my browser.

2) Using curl, I would like to be able to pull down any content which is exposed over http.

3) Using wget, I would like to be able to pull down any content which is exposed over http.

Comment 8 Tomer Brisker 2018-01-08 14:31:56 UTC
*** Bug 1302442 has been marked as a duplicate of this bug. ***

Comment 9 Erinn Looney-Triggs 2018-01-08 21:39:10 UTC
Alternatively, you could just serve all the content over https so the entire site is actually in compliance like HSTS is stating that it should be.

-Erinn

Comment 11 Tomer Brisker 2018-01-09 08:20:49 UTC
Erinn - I agree publishing everything over https is best practice, and HSTS will continue being enabled by default. 
This setting is intended to allow users who need to have browser access to http repos (in cases explained in comment #3) to disable it when needed.

Comment 12 pm-sat@redhat.com 2018-01-16 11:01:51 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/22165 has been resolved.

Comment 13 Radovan Drazny 2018-08-30 18:43:47 UTC
Tested on Satellite 6.4 Snap 19

Reproducer: 
Mozilla Firefox used for accessing the Satellite server.

$ grep hsts /etc/foreman/settings.yaml 
:hsts_enabled: true

Create and install a custom SSL certificate on the Satellite server. A private CA is needed for this, and the CA's certificate must be imported into the browser and trusted for site verification. You MUST have secure site icon present for the Satellite server (pure green padlock in the browser's address bar, connection must be flagged as "Secure Connection"), HSTS headers are ignored otherwise (Message in the FF console: "Strict-Transport-Security: The connection to the site is untrustworthy, so the specified header was ignored.").

The debug cert (Available from the "Administer-> Organizations -> <used_organization> -> Edit")for TLS authentication is not installed.

Create a custom product on the Satellite with a custom repo, publish it via HTTP.

Check the HTTPS headers for the server:
$ curl -s -I -k https://satellite.example.com | grep Strict-Transport-Security
Strict-Transport-Security: max-age=631139040; includeSubdomains

HSTS is active.

As we are by default acessing the Satellite via HTTPS, so the HSTS should be activated by now. Trying to access the Pulp HTTP URL for the custom product will make browser redirect to HTTPS, and as we don't have the debug certificate installed, the connection will fail with the following error: "Secure Connection Failed: SSL_ERROR_HANDSHAKE_FAILURE_ALERT." The redirect is caused by the browser conforming to the HSTS requirements. This is expected.

Steps to make sure this is caused by the HSTS:
Quit the FF completely, go to ~/.mozilla/firefox/<your_FF_profile>/ and open the file SiteSecurityServiceState.txt. Find the line with the Satellite server hostname, and remove it (sed -i "/satellite.example.com/d" SiteSecurityServiceState.txt). Start the FF, and go DIRECTLY to the Pulp HTTP URL for the custom product. The connection should work now. 

You MUST NOT access the Satellite server WebUI before that, because any HTTPS connection will cause the HSTS to kick back in. That means you can'ŧ find the Pulp HTTP URL via the WebUI, but you have to enter it directly into the browser. 

One final step - access the WebUI via HTTPS, and then try to access the custom product Pulp HTTP URL again. This should fail now - the HSTS is back on.

Verification:
Disable the HSTS on the Satellite server:

$ sed -i "s/:hsts_enabled: true/:hsts_enabled: false/" /etc/foreman/settings.yaml
$ foreman-maintain service restart

Check the HTTPS headers for the server:

$ curl -s -k -I https://satellite.example.com | grep Strict-Transport-Security
Strict-Transport-Security: max-age=0; includeSubdomains

The HSTS header is still present, but max-age is set to zero, so it should not be active.

Go to the WebUI, navigate to the custom product Repository Details page, and try to open the Pulp HTTP URL. The URL opens, there is no redirect to HTTPS, and it is possible to browse the repository and download rpms using wget or curl (comment #7):

$ wget http://satellite.example.com/pulp/repos/Default_Organization/Library/custom/TestProduct/Centos_Extra/Packages/p/perf-3.10.0-862.11.6.el7.centos.plus.1.x86_64.rpm
--2018-08-30 20:01:40--  http://satellite.example.com/pulp/repos/Default_Organization/Library/custom/TestProduct/Centos_Extra/Packages/p/perf-3.10.0-862.11.6.el7.centos.plus.1.x86_64.rpm
Resolving satellite.example.com (satellite.example.com)... 10.16.64.216, 2620:52:0:1040:5ef9:ddff:fe9b:2c35
Connecting to satellite.example.com (satellite.example.com)|10.16.64.216|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8002116 (7.6M) [application/x-rpm]
Saving to: ‘perf-3.10.0-862.11.6.el7.centos.plus.1.x86_64.rpm’

perf-3.10.0-862.11.6.el7.centos.plus.1.x86_64.rpm          100%[======================================================================================>]   7.63M  3.34MB/s    in 2.3s    

2018-08-30 20:01:43 (3.34 MB/s) - ‘perf-3.10.0-862.11.6.el7.centos.plus.1.x86_64.rpm’ saved [8002116/8002116]

$ rm perf-3.10.0-862.11.6.el7.centos.plus.1.x86_64.rpm 

$ curl -O http://satellite.example.com/pulp/repos/Default_Organization/Library/custom/TestProduct/Centos_Extra/Packages/p/perf-3.10.0-862.11.6.el7.centos.plus.1.x86_64.rpm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 7814k  100 7814k    0     0  2587k      0  0:00:03  0:00:03 --:--:-- 2587k

Comment 15 errata-xmlrpc 2018-10-16 15:28:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2927


Note You need to log in before you can comment on or make changes to this bug.