Bug 1990826

Summary:	New non-secure and secure routes without hsts annotation fail to get created in globally enforced hsts domain resources
Product:	OpenShift Container Platform	Reporter:	Arvind iyengar <aiyengar>
Component:	Networking	Assignee:	Candace Holman <cholman>
Networking sub component:	router	QA Contact:	Arvind iyengar <aiyengar>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	high	CC:	aos-bugs, cholman, mmasters
Version:	4.9
Target Milestone:	---
Target Release:	4.9.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-10-18 17:45:05 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Arvind iyengar 2021-08-06 11:00:40 UTC

Description of problem:
New non-secure and secure routes without hsts annotation fail to get created in globally enforced hsts domain resources. 

OpenShift release version:
4.9.0-0.nightly-2021-08-04-131508

How reproducible:
- Frequently

Steps to Reproduce (in detail):
1.Enable the global HSTS validation plugin by adding the below configuration in the cluster ingresses resource:
-----
oc edit ingresses.config.openshift.io/cluster

  requiredHSTSPolicies:
  - domainPatterns:
    - '*.internalapps.aiyengar4906.qe.devcluster.openshift.com'
    maxAge:
      smallestMaxAge: 1
      largestMaxAge: 31536000
    preloadPolicy: "RequirePreload"
    includeSubDomainsPolicy: "RequireIncludeSubDomains"
-----

2.Try exposing an insecure route through the domain:
------
oc get all                                        
NAME                      READY   STATUS    RESTARTS   AGE
pod/web-server-rc-ckxp5   1/1     Running   0          25m

NAME                                  DESIRED   CURRENT   READY   AGE
replicationcontroller/web-server-rc   1         1         1       25m

NAME                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)     AGE
service/service-secure     ClusterIP   172.30.141.23   <none>        27443/TCP   25m
service/service-unsecure   ClusterIP   172.30.80.247   <none>        27017/TCP   25m

oc expose svc service-unsecure --hostname=service-unsecure-test1.internalapps.aiyengar4906.qe.devcluster.openshift.com 
Error from server (Forbidden): routes.route.openshift.io "service-unsecure" is forbidden: max-age must be set in HSTS annotation <-----

sample test to expose route from non hsts enforce domain or default domain in this case:
oc expose svc service-unsecure                   
route.route.openshift.io/service-unsecure exposed

oc get route                                        
NAME               HOST/PORT                                                                           PATH   SERVICES           PORT    TERMINATION   WILDCARD
service-unsecure   service-unsecure-test1.apps.aiyengar4906.qe.devcluster.openshift.com ... 1 more            service-unsecure   http                  None

curl http://service-unsecure-test1.apps.aiyengar4906.qe.devcluster.openshift.com            
Hello-OpenShift web-server-rc-ckxp5 http-8080
------

3. Test exposing a secure route without hsts annotation:
------
test file to add a secure route:
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: edge-hsts4

spec:
    host: edge-hsts4-test1.internalapps.aiyengar4906.qe.devcluster.openshift.com
    to:
        kind: Service
        name: service-unsecure
    tls:
        termination: edge
        key: "-----BEGIN PRIVATE KEY-----\nMIICdwIBADANBgkqhkiG9w0BAQEFAASCAmEwggJdAgEAAoGBAKARzcO20lcg+qDf\ne9xAEde2dbUij9LWclX/VfGp0Xhydzf/ODmL5c/Iy/cxgKvoo7DZTuPYsXrS7z9u\nuLI4S4stqj/n21KrYIwDdIXvaOc6CTTQxqUE20LZ08LkR8BLra4Lbn7lhlRgayOM\nClfdUL474Cv0S4OlDS07idbD1kXzAgMBAAECgYA9vMAtFU1pV4nzFF9UYs2+8lvR\n4iOwwQ9WReYjEEl/eD6tNV29LE0V6C9rBwfGxjKkWhxIWuKRKdwnDhBkhLv1tP18\noCFadzkw1eNg6GOw/uvSB/z+JWSAdjXuGTtsXtU3tvR2nzYnMuJEk4f5vipCkDtf\nZ4AXTTPCqg8DFdLJyQJBANLlgl2IOwzaW+f5JZGesbNDdPx9L3ntNovAWv8tQD2i\n//EIRbYujVizrkp7FC78cpSxLO665a5f+VoGcrIHAL8CQQDCTYpOyJa6Kpd16SdP\nDEIzQrdvbJIhKKtN0ZJ12spUVSytrP2Q4sby4Qm915q3vOsUZGInXc7WWRpXKS5n\nc5PNAkEAv+hA/MOemE+LGkfJO/1gTnOv3KI9tYF6BSmApHuU3YGZzMduSB2MWY8H\nppbhAvCNg2jGLma74jVLPfRoIj/lGQJARwDo8uNQWVWpJZh/Gd7j7jGKMPie6ekf\nuH9GIzVBzNGXUxwtSR3mD+l2kt5QFqa9zSTlzXb1V9UV0BYnc/yDXQJBAIwdeFA2\nAFEtVY5LFke4eFD7OPpvSO6zzH5OcXTSmE2rzNizjPWweWQD5jkd/28U9iil/KwD\nnH3BZ44F1sWbT58=\n-----END PRIVATE KEY-----\n"
        certificate: "-----BEGIN CERTIFICATE-----\nMIIDEzCCAfugAwIBAgIBDjANBgkqhkiG9w0BAQUFADCBoTELMAkGA1UEBhMCVVMx\nCzAJBgNVBAgMAlNDMRUwEwYDVQQHDAxEZWZhdWx0IENpdHkxHDAaBgNVBAoME0Rl\nZmF1bHQgQ29tcGFueSBMdGQxEDAOBgNVBAsMB1Rlc3QgQ0ExGjAYBgNVBAMMEXd3\ndy5leGFtcGxlY2EuY29tMSIwIAYJKoZIhvcNAQkBFhNleGFtcGxlQGV4YW1wbGUu\nY29tMB4XDTE2MDUyNTA4MTI1OFoXDTI2MDUyMzA4MTI1OFowbTEWMBQGA1UEAwwN\nKi5leGFtcGxlLmNvbTELMAkGA1UECAwCQkoxCzAJBgNVBAYTAkNOMR8wHQYJKoZI\nhvcNAQkBFhBibWVuZ0ByZWRoYXQuY29tMQswCQYDVQQKDAJSSDELMAkGA1UECwwC\nT1MwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAKARzcO20lcg+qDfe9xAEde2\ndbUij9LWclX/VfGp0Xhydzf/ODmL5c/Iy/cxgKvoo7DZTuPYsXrS7z9uuLI4S4st\nqj/n21KrYIwDdIXvaOc6CTTQxqUE20LZ08LkR8BLra4Lbn7lhlRgayOMClfdUL47\n4Cv0S4OlDS07idbD1kXzAgMBAAGjDTALMAkGA1UdEwQCMAAwDQYJKoZIhvcNAQEF\nBQADggEBAMIRge8dXWyZJsve1aycniBxdyWUMoM9tPBDvfZAlLLDWubuoaEXLojy\n3wGHGzDGOWrvYHwmPfWDNf+IlrxetiIOiXxKfGtTsOuqdJCcbz3y70WiICziX5m7\ndqeoGfnGhf6Ys6/L0/hecHLxw86RlhlJnH7W0eB3qeT7vc7ytDxcRFlvhFxgAD3O\nF1H8XKJWuaghzus0rDPlQviEPYkYfmUBMNLl/dbWEVNV3wCakaaMoYg12y4p1Rd4\npgW3DwXWYbnAX5K1TbtuALWvmiOIcGbtLTwKqI6pdPJx4bo+zbwOuo/Q9lbjRcZG\nAErbDKA4OfpTCrpu/qADXfnJVGCuWUo=\n-----END CERTIFICATE-----\n"
        caCertificate: "-----BEGIN CERTIFICATE-----\nMIIEFzCCAv+gAwIBAgIJALK1iUpF2VQLMA0GCSqGSIb3DQEBBQUAMIGhMQswCQYD\nVQQGEwJVUzELMAkGA1UECAwCU0MxFTATBgNVBAcMDERlZmF1bHQgQ2l0eTEcMBoG\nA1UECgwTRGVmYXVsdCBDb21wYW55IEx0ZDEQMA4GA1UECwwHVGVzdCBDQTEaMBgG\nA1UEAwwRd3d3LmV4YW1wbGVjYS5jb20xIjAgBgkqhkiG9w0BCQEWE2V4YW1wbGVA\nZXhhbXBsZS5jb20wHhcNMTUwMTEyMTQxNTAxWhcNMjUwMTA5MTQxNTAxWjCBoTEL\nMAkGA1UEBhMCVVMxCzAJBgNVBAgMAlNDMRUwEwYDVQQHDAxEZWZhdWx0IENpdHkx\nHDAaBgNVBAoME0RlZmF1bHQgQ29tcGFueSBMdGQxEDAOBgNVBAsMB1Rlc3QgQ0Ex\nGjAYBgNVBAMMEXd3dy5leGFtcGxlY2EuY29tMSIwIAYJKoZIhvcNAQkBFhNleGFt\ncGxlQGV4YW1wbGUuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA\nw2rK1J2NMtQj0KDug7g7HRKl5jbf0QMkMKyTU1fBtZ0cCzvsF4CqV11LK4BSVWaK\nrzkaXe99IVJnH8KdOlDl5Dh/+cJ3xdkClSyeUT4zgb6CCBqg78ePp+nN11JKuJlV\nIG1qdJpB1J5O/kCLsGcTf7RS74MtqMFo96446Zvt7YaBhWPz6gDaO/TUzfrNcGLA\nEfHVXkvVWqb3gqXUztZyVex/gtP9FXQ7gxTvJml7UkmT0VAFjtZnCqmFxpLZFZ15\n+qP9O7Q2MpsGUO/4vDAuYrKBeg1ZdPSi8gwqUP2qWsGd9MIWRv3thI2903BczDc7\nr8WaIbm37vYZAS9G56E4+wIDAQABo1AwTjAdBgNVHQ4EFgQUugLrSJshOBk5TSsU\nANs4+SmJUGwwHwYDVR0jBBgwFoAUugLrSJshOBk5TSsUANs4+SmJUGwwDAYDVR0T\nBAUwAwEB/zANBgkqhkiG9w0BAQUFAAOCAQEAaMJ33zAMV4korHo5aPfayV3uHoYZ\n1ChzP3eSsF+FjoscpoNSKs91ZXZF6LquzoNezbfiihK4PYqgwVD2+O0/Ty7UjN4S\nqzFKVR4OS/6lCJ8YncxoFpTntbvjgojf1DEataKFUN196PAANc3yz8cWHF4uvjPv\nWkgFqbIjb+7D1YgglNyovXkRDlRZl0LD1OQ0ZWhd4Ge1qx8mmmanoBeYZ9+DgpFC\nj9tQAbS867yeOryNe7sEOIpXAAqK/DTu0hB6+ySsDfMo4piXCc2aA/eI2DCuw08e\nw17Dz9WnupZjVdwTKzDhFgJZMLDqn37HQnT6EemLFqbcR0VPEnfyhDtZIQ==\n-----END CERTIFICATE-----"


oc create -f edge-route-test.yaml
Error from server (Forbidden): error when creating "../essential-docs/test-files/edge-route-test.yaml": routes.route.openshift.io "edge-hsts4" is forbidden: max-age must be set in HSTS annotation
------

Actual results:
Only secured routes with hsts annotation gets admitted and appears to work:
-----
Test file with annotation:
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: edge-hsts3
  annotations:
    haproxy.router.openshift.io/hsts_header: max-age=20000;includeSubDomains;preload

oc create -f ../essential-docs/test-files/edge-route-test.yaml
route.route.openshift.io/edge-hsts3 created

oc get route                                        
NAME               HOST/PORT                                                                           PATH   SERVICES           PORT    TERMINATION   WILDCARD
edge-hsts3         edge-hsts3-test1.internalapps.aiyengar4906.qe.devcluster.openshift.com ... 1 more          service-unsecure   <all>   edge          None <----
-----

Expected results:
The routes without the hsts annotation should get exposed from the hsts enforced domain.


Impact of the problem:
In the current state, new routes without hsts annotation cannot be added to hsts enforced domain.


Additional info:

Setting the "MaxAge" tuning values to "0" or removing the "smallestMaxAge" and "largestMaxAge" timer options, the result seem to be same:
-----
Ingresses settings with the additional timers removed:
  requiredHSTSPolicies:
  - domainPatterns:
    - '*.internalapps.aiyengar4906.qe.devcluster.openshift.com'
    maxAge:
      smallestMaxAge: 1
      largestMaxAge: 31536000
    preloadPolicy: "RequirePreload"
    includeSubDomainsPolicy: "RequireIncludeSubDomains"

Add a new unsecure route with expose command:
oc expose svc service-unsecure --hostname=service-unsecure-test1.internalapps.aiyengar4906.qe.devcluster.openshift.com 
Error from server (Forbidden): routes.route.openshift.io "service-unsecure" is forbidden: max-age must be set in HSTS annotation


Adding a secure route without the annotation:
oc create -f edge-route-test.yaml
Error from server (Forbidden): error when creating "../essential-docs/test-files/edge-route-test.yaml": routes.route.openshift.io "edge-hsts4" is forbidden: max-age must be set in HSTS annotation
-----

Comment 4 Arvind iyengar 2021-08-17 10:45:09 UTC

Verified in "4.9.0-0.ci.test-2021-08-17-062351-ci-ln-5bhh0h2-latest" latest Ci image. With this fix in place, the following are the set of observations made with HSTS enforced domain:
- new non-TLS routes can be added with "oc expose" command.
- new non-TLS routes can be added via templates with "oc create -f" command.
- new TLS route cannot be added via "oc expose" or "oc create route" command due to no hsts annotation options with the command.
- new TLS route cannot be added via "oc create -f" without the HSTS annotations defined in the template [behavior as per the proposal]  

Test excerpts:
------
oc get clusterversion       
NAME      VERSION                                                  AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.ci.test-2021-08-17-062351-ci-ln-5bhh0h2-latest   True        False         158m    Cluster version is 4.9.0-0.ci.test-2021-08-17-062351-ci-ln-5bhh0h2-latest

Hsts configurations set:
spec:
  domain: apps.ci-ln-5bhh0h2-f76d1.origin-ci-int-gce.dev.openshift.com
  requiredHSTSPolicies:
  - domainPatterns:
    - '*.internalapps.ci-ln-5bhh0h2-f76d1.origin-ci-int-gce.dev.openshift.com'
    includeSubDomainsPolicy: RequireIncludeSubDomains
    maxAge:
      largestMaxAge: 31536000
      smallestMaxAge: 1
    preloadPolicy: RequirePreload

Creating a non-tls route in the same domain:
oc expose svc service-unsecure --hostname=service-unsecure-hsts-test.internalapps.ci-ln-5bhh0h2-f76d1.origin-ci-int-gce.dev.openshift.com --name=unsecure-route2
route.route.openshift.io/unsecure-route2 exposed


oc get route                 
NAME              HOST/PORT                                                                                                    PATH   SERVICES           PORT   TERMINATION   WILDCARD
edge-route        edge-route-hsts-test.internalapps.ci-ln-5bhh0h2-f76d1.origin-ci-int-gce.dev.openshift.com ... 1 more                service-unsecure   http   edge          None
unsecure-route2   service-unsecure-hsts-test.internalapps.ci-ln-5bhh0h2-f76d1.origin-ci-int-gce.dev.openshift.com ... 1 more          service-unsecure   http                 None

curl http://service-unsecure-hsts-test.internalapps.ci-ln-5bhh0h2-f76d1.origin-ci-int-gce.dev.openshift.com -I         
HTTP/1.1 200 OK
server: nginx/1.18.0
date: Tue, 17 Aug 2021 09:39:02 GMT
content-type: text/html
content-length: 46
last-modified: Tue, 17 Aug 2021 07:14:56 GMT
etag: "611b61f0-2e"
accept-ranges: bytes
set-cookie: c60d84bf5237065c1fc86b6a6baf745b=72e230a16cbc2cfe687dcfe93baa9f5d; path=/; HttpOnly
cache-control: private

Creating a non-tls route via template:
at unsecure-route-PR240.yaml                                                                                      
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: unsecure-route3
  namespace: hsts-test
spec:
  host: service-unsecure3-hsts-test.internalapps.ci-ln-5bhh0h2-f76d1.origin-ci-int-gce.dev.openshift.com
  port:
    targetPort: http
  to:
    kind: Service
    name: service-unsecure
    weight: 100
  wildcardPolicy: None
status:

oc create -f unsecure-route-PR240.yaml
route.route.openshift.io/unsecure-route3 created


oc get route               
NAME              HOST/PORT                                                                                                     PATH   SERVICES           PORT   TERMINATION   WILDCARD
edge-route        edge-route-hsts-test.internalapps.ci-ln-5bhh0h2-f76d1.origin-ci-int-gce.dev.openshift.com ... 1 more                 service-unsecure   http   edge          None
unsecure-route2   service-unsecure-hsts-test.internalapps.ci-ln-5bhh0h2-f76d1.origin-ci-int-gce.dev.openshift.com ... 1 more           service-unsecure   http                 None
unsecure-route3   service-unsecure3-hsts-test.internalapps.ci-ln-5bhh0h2-f76d1.origin-ci-int-gce.dev.openshift.com ... 1 more          service-unsecure   http                 None


curl -I http://service-unsecure3-hsts-test.internalapps.ci-ln-5bhh0h2-f76d1.origin-ci-int-gce.dev.openshift.com       
HTTP/1.1 200 OK
server: nginx/1.18.0
date: Tue, 17 Aug 2021 09:42:29 GMT
content-type: text/html
content-length: 46
last-modified: Tue, 17 Aug 2021 07:14:56 GMT
etag: "611b61f0-2e"
accept-ranges: bytes
set-cookie: b228a2f71026e1b0c795276730fbdd2c=72e230a16cbc2cfe687dcfe93baa9f5d; path=/; HttpOnly
cache-control: private
------

Comment 8 errata-xmlrpc 2021-10-18 17:45:05 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759