Bug 1990826
| Summary: | New non-secure and secure routes without hsts annotation fail to get created in globally enforced hsts domain resources | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Arvind iyengar <aiyengar> |
| Component: | Networking | Assignee: | Candace Holman <cholman> |
| Networking sub component: | router | QA Contact: | Arvind iyengar <aiyengar> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | high | ||
| Priority: | high | CC: | aos-bugs, cholman, mmasters |
| Version: | 4.9 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.9.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-10-18 17:45:05 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Verified in "4.9.0-0.ci.test-2021-08-17-062351-ci-ln-5bhh0h2-latest" latest Ci image. With this fix in place, the following are the set of observations made with HSTS enforced domain:
- new non-TLS routes can be added with "oc expose" command.
- new non-TLS routes can be added via templates with "oc create -f" command.
- new TLS route cannot be added via "oc expose" or "oc create route" command due to no hsts annotation options with the command.
- new TLS route cannot be added via "oc create -f" without the HSTS annotations defined in the template [behavior as per the proposal]
Test excerpts:
------
oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.9.0-0.ci.test-2021-08-17-062351-ci-ln-5bhh0h2-latest True False 158m Cluster version is 4.9.0-0.ci.test-2021-08-17-062351-ci-ln-5bhh0h2-latest
Hsts configurations set:
spec:
domain: apps.ci-ln-5bhh0h2-f76d1.origin-ci-int-gce.dev.openshift.com
requiredHSTSPolicies:
- domainPatterns:
- '*.internalapps.ci-ln-5bhh0h2-f76d1.origin-ci-int-gce.dev.openshift.com'
includeSubDomainsPolicy: RequireIncludeSubDomains
maxAge:
largestMaxAge: 31536000
smallestMaxAge: 1
preloadPolicy: RequirePreload
Creating a non-tls route in the same domain:
oc expose svc service-unsecure --hostname=service-unsecure-hsts-test.internalapps.ci-ln-5bhh0h2-f76d1.origin-ci-int-gce.dev.openshift.com --name=unsecure-route2
route.route.openshift.io/unsecure-route2 exposed
oc get route
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
edge-route edge-route-hsts-test.internalapps.ci-ln-5bhh0h2-f76d1.origin-ci-int-gce.dev.openshift.com ... 1 more service-unsecure http edge None
unsecure-route2 service-unsecure-hsts-test.internalapps.ci-ln-5bhh0h2-f76d1.origin-ci-int-gce.dev.openshift.com ... 1 more service-unsecure http None
curl http://service-unsecure-hsts-test.internalapps.ci-ln-5bhh0h2-f76d1.origin-ci-int-gce.dev.openshift.com -I
HTTP/1.1 200 OK
server: nginx/1.18.0
date: Tue, 17 Aug 2021 09:39:02 GMT
content-type: text/html
content-length: 46
last-modified: Tue, 17 Aug 2021 07:14:56 GMT
etag: "611b61f0-2e"
accept-ranges: bytes
set-cookie: c60d84bf5237065c1fc86b6a6baf745b=72e230a16cbc2cfe687dcfe93baa9f5d; path=/; HttpOnly
cache-control: private
Creating a non-tls route via template:
at unsecure-route-PR240.yaml
apiVersion: route.openshift.io/v1
kind: Route
metadata:
name: unsecure-route3
namespace: hsts-test
spec:
host: service-unsecure3-hsts-test.internalapps.ci-ln-5bhh0h2-f76d1.origin-ci-int-gce.dev.openshift.com
port:
targetPort: http
to:
kind: Service
name: service-unsecure
weight: 100
wildcardPolicy: None
status:
oc create -f unsecure-route-PR240.yaml
route.route.openshift.io/unsecure-route3 created
oc get route
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
edge-route edge-route-hsts-test.internalapps.ci-ln-5bhh0h2-f76d1.origin-ci-int-gce.dev.openshift.com ... 1 more service-unsecure http edge None
unsecure-route2 service-unsecure-hsts-test.internalapps.ci-ln-5bhh0h2-f76d1.origin-ci-int-gce.dev.openshift.com ... 1 more service-unsecure http None
unsecure-route3 service-unsecure3-hsts-test.internalapps.ci-ln-5bhh0h2-f76d1.origin-ci-int-gce.dev.openshift.com ... 1 more service-unsecure http None
curl -I http://service-unsecure3-hsts-test.internalapps.ci-ln-5bhh0h2-f76d1.origin-ci-int-gce.dev.openshift.com
HTTP/1.1 200 OK
server: nginx/1.18.0
date: Tue, 17 Aug 2021 09:42:29 GMT
content-type: text/html
content-length: 46
last-modified: Tue, 17 Aug 2021 07:14:56 GMT
etag: "611b61f0-2e"
accept-ranges: bytes
set-cookie: b228a2f71026e1b0c795276730fbdd2c=72e230a16cbc2cfe687dcfe93baa9f5d; path=/; HttpOnly
cache-control: private
------
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759 |
Description of problem: New non-secure and secure routes without hsts annotation fail to get created in globally enforced hsts domain resources. OpenShift release version: 4.9.0-0.nightly-2021-08-04-131508 How reproducible: - Frequently Steps to Reproduce (in detail): 1.Enable the global HSTS validation plugin by adding the below configuration in the cluster ingresses resource: ----- oc edit ingresses.config.openshift.io/cluster requiredHSTSPolicies: - domainPatterns: - '*.internalapps.aiyengar4906.qe.devcluster.openshift.com' maxAge: smallestMaxAge: 1 largestMaxAge: 31536000 preloadPolicy: "RequirePreload" includeSubDomainsPolicy: "RequireIncludeSubDomains" ----- 2.Try exposing an insecure route through the domain: ------ oc get all NAME READY STATUS RESTARTS AGE pod/web-server-rc-ckxp5 1/1 Running 0 25m NAME DESIRED CURRENT READY AGE replicationcontroller/web-server-rc 1 1 1 25m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/service-secure ClusterIP 172.30.141.23 <none> 27443/TCP 25m service/service-unsecure ClusterIP 172.30.80.247 <none> 27017/TCP 25m oc expose svc service-unsecure --hostname=service-unsecure-test1.internalapps.aiyengar4906.qe.devcluster.openshift.com Error from server (Forbidden): routes.route.openshift.io "service-unsecure" is forbidden: max-age must be set in HSTS annotation <----- sample test to expose route from non hsts enforce domain or default domain in this case: oc expose svc service-unsecure route.route.openshift.io/service-unsecure exposed oc get route NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD service-unsecure service-unsecure-test1.apps.aiyengar4906.qe.devcluster.openshift.com ... 1 more service-unsecure http None curl http://service-unsecure-test1.apps.aiyengar4906.qe.devcluster.openshift.com Hello-OpenShift web-server-rc-ckxp5 http-8080 ------ 3. Test exposing a secure route without hsts annotation: ------ test file to add a secure route: apiVersion: route.openshift.io/v1 kind: Route metadata: name: edge-hsts4 spec: host: edge-hsts4-test1.internalapps.aiyengar4906.qe.devcluster.openshift.com to: kind: Service name: service-unsecure tls: termination: edge key: "-----BEGIN PRIVATE KEY-----\nMIICdwIBADANBgkqhkiG9w0BAQEFAASCAmEwggJdAgEAAoGBAKARzcO20lcg+qDf\ne9xAEde2dbUij9LWclX/VfGp0Xhydzf/ODmL5c/Iy/cxgKvoo7DZTuPYsXrS7z9u\nuLI4S4stqj/n21KrYIwDdIXvaOc6CTTQxqUE20LZ08LkR8BLra4Lbn7lhlRgayOM\nClfdUL474Cv0S4OlDS07idbD1kXzAgMBAAECgYA9vMAtFU1pV4nzFF9UYs2+8lvR\n4iOwwQ9WReYjEEl/eD6tNV29LE0V6C9rBwfGxjKkWhxIWuKRKdwnDhBkhLv1tP18\noCFadzkw1eNg6GOw/uvSB/z+JWSAdjXuGTtsXtU3tvR2nzYnMuJEk4f5vipCkDtf\nZ4AXTTPCqg8DFdLJyQJBANLlgl2IOwzaW+f5JZGesbNDdPx9L3ntNovAWv8tQD2i\n//EIRbYujVizrkp7FC78cpSxLO665a5f+VoGcrIHAL8CQQDCTYpOyJa6Kpd16SdP\nDEIzQrdvbJIhKKtN0ZJ12spUVSytrP2Q4sby4Qm915q3vOsUZGInXc7WWRpXKS5n\nc5PNAkEAv+hA/MOemE+LGkfJO/1gTnOv3KI9tYF6BSmApHuU3YGZzMduSB2MWY8H\nppbhAvCNg2jGLma74jVLPfRoIj/lGQJARwDo8uNQWVWpJZh/Gd7j7jGKMPie6ekf\nuH9GIzVBzNGXUxwtSR3mD+l2kt5QFqa9zSTlzXb1V9UV0BYnc/yDXQJBAIwdeFA2\nAFEtVY5LFke4eFD7OPpvSO6zzH5OcXTSmE2rzNizjPWweWQD5jkd/28U9iil/KwD\nnH3BZ44F1sWbT58=\n-----END PRIVATE KEY-----\n" certificate: "-----BEGIN CERTIFICATE-----\nMIIDEzCCAfugAwIBAgIBDjANBgkqhkiG9w0BAQUFADCBoTELMAkGA1UEBhMCVVMx\nCzAJBgNVBAgMAlNDMRUwEwYDVQQHDAxEZWZhdWx0IENpdHkxHDAaBgNVBAoME0Rl\nZmF1bHQgQ29tcGFueSBMdGQxEDAOBgNVBAsMB1Rlc3QgQ0ExGjAYBgNVBAMMEXd3\ndy5leGFtcGxlY2EuY29tMSIwIAYJKoZIhvcNAQkBFhNleGFtcGxlQGV4YW1wbGUu\nY29tMB4XDTE2MDUyNTA4MTI1OFoXDTI2MDUyMzA4MTI1OFowbTEWMBQGA1UEAwwN\nKi5leGFtcGxlLmNvbTELMAkGA1UECAwCQkoxCzAJBgNVBAYTAkNOMR8wHQYJKoZI\nhvcNAQkBFhBibWVuZ0ByZWRoYXQuY29tMQswCQYDVQQKDAJSSDELMAkGA1UECwwC\nT1MwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAKARzcO20lcg+qDfe9xAEde2\ndbUij9LWclX/VfGp0Xhydzf/ODmL5c/Iy/cxgKvoo7DZTuPYsXrS7z9uuLI4S4st\nqj/n21KrYIwDdIXvaOc6CTTQxqUE20LZ08LkR8BLra4Lbn7lhlRgayOMClfdUL47\n4Cv0S4OlDS07idbD1kXzAgMBAAGjDTALMAkGA1UdEwQCMAAwDQYJKoZIhvcNAQEF\nBQADggEBAMIRge8dXWyZJsve1aycniBxdyWUMoM9tPBDvfZAlLLDWubuoaEXLojy\n3wGHGzDGOWrvYHwmPfWDNf+IlrxetiIOiXxKfGtTsOuqdJCcbz3y70WiICziX5m7\ndqeoGfnGhf6Ys6/L0/hecHLxw86RlhlJnH7W0eB3qeT7vc7ytDxcRFlvhFxgAD3O\nF1H8XKJWuaghzus0rDPlQviEPYkYfmUBMNLl/dbWEVNV3wCakaaMoYg12y4p1Rd4\npgW3DwXWYbnAX5K1TbtuALWvmiOIcGbtLTwKqI6pdPJx4bo+zbwOuo/Q9lbjRcZG\nAErbDKA4OfpTCrpu/qADXfnJVGCuWUo=\n-----END CERTIFICATE-----\n" caCertificate: "-----BEGIN CERTIFICATE-----\nMIIEFzCCAv+gAwIBAgIJALK1iUpF2VQLMA0GCSqGSIb3DQEBBQUAMIGhMQswCQYD\nVQQGEwJVUzELMAkGA1UECAwCU0MxFTATBgNVBAcMDERlZmF1bHQgQ2l0eTEcMBoG\nA1UECgwTRGVmYXVsdCBDb21wYW55IEx0ZDEQMA4GA1UECwwHVGVzdCBDQTEaMBgG\nA1UEAwwRd3d3LmV4YW1wbGVjYS5jb20xIjAgBgkqhkiG9w0BCQEWE2V4YW1wbGVA\nZXhhbXBsZS5jb20wHhcNMTUwMTEyMTQxNTAxWhcNMjUwMTA5MTQxNTAxWjCBoTEL\nMAkGA1UEBhMCVVMxCzAJBgNVBAgMAlNDMRUwEwYDVQQHDAxEZWZhdWx0IENpdHkx\nHDAaBgNVBAoME0RlZmF1bHQgQ29tcGFueSBMdGQxEDAOBgNVBAsMB1Rlc3QgQ0Ex\nGjAYBgNVBAMMEXd3dy5leGFtcGxlY2EuY29tMSIwIAYJKoZIhvcNAQkBFhNleGFt\ncGxlQGV4YW1wbGUuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA\nw2rK1J2NMtQj0KDug7g7HRKl5jbf0QMkMKyTU1fBtZ0cCzvsF4CqV11LK4BSVWaK\nrzkaXe99IVJnH8KdOlDl5Dh/+cJ3xdkClSyeUT4zgb6CCBqg78ePp+nN11JKuJlV\nIG1qdJpB1J5O/kCLsGcTf7RS74MtqMFo96446Zvt7YaBhWPz6gDaO/TUzfrNcGLA\nEfHVXkvVWqb3gqXUztZyVex/gtP9FXQ7gxTvJml7UkmT0VAFjtZnCqmFxpLZFZ15\n+qP9O7Q2MpsGUO/4vDAuYrKBeg1ZdPSi8gwqUP2qWsGd9MIWRv3thI2903BczDc7\nr8WaIbm37vYZAS9G56E4+wIDAQABo1AwTjAdBgNVHQ4EFgQUugLrSJshOBk5TSsU\nANs4+SmJUGwwHwYDVR0jBBgwFoAUugLrSJshOBk5TSsUANs4+SmJUGwwDAYDVR0T\nBAUwAwEB/zANBgkqhkiG9w0BAQUFAAOCAQEAaMJ33zAMV4korHo5aPfayV3uHoYZ\n1ChzP3eSsF+FjoscpoNSKs91ZXZF6LquzoNezbfiihK4PYqgwVD2+O0/Ty7UjN4S\nqzFKVR4OS/6lCJ8YncxoFpTntbvjgojf1DEataKFUN196PAANc3yz8cWHF4uvjPv\nWkgFqbIjb+7D1YgglNyovXkRDlRZl0LD1OQ0ZWhd4Ge1qx8mmmanoBeYZ9+DgpFC\nj9tQAbS867yeOryNe7sEOIpXAAqK/DTu0hB6+ySsDfMo4piXCc2aA/eI2DCuw08e\nw17Dz9WnupZjVdwTKzDhFgJZMLDqn37HQnT6EemLFqbcR0VPEnfyhDtZIQ==\n-----END CERTIFICATE-----" oc create -f edge-route-test.yaml Error from server (Forbidden): error when creating "../essential-docs/test-files/edge-route-test.yaml": routes.route.openshift.io "edge-hsts4" is forbidden: max-age must be set in HSTS annotation ------ Actual results: Only secured routes with hsts annotation gets admitted and appears to work: ----- Test file with annotation: apiVersion: route.openshift.io/v1 kind: Route metadata: name: edge-hsts3 annotations: haproxy.router.openshift.io/hsts_header: max-age=20000;includeSubDomains;preload oc create -f ../essential-docs/test-files/edge-route-test.yaml route.route.openshift.io/edge-hsts3 created oc get route NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD edge-hsts3 edge-hsts3-test1.internalapps.aiyengar4906.qe.devcluster.openshift.com ... 1 more service-unsecure <all> edge None <---- ----- Expected results: The routes without the hsts annotation should get exposed from the hsts enforced domain. Impact of the problem: In the current state, new routes without hsts annotation cannot be added to hsts enforced domain. Additional info: Setting the "MaxAge" tuning values to "0" or removing the "smallestMaxAge" and "largestMaxAge" timer options, the result seem to be same: ----- Ingresses settings with the additional timers removed: requiredHSTSPolicies: - domainPatterns: - '*.internalapps.aiyengar4906.qe.devcluster.openshift.com' maxAge: smallestMaxAge: 1 largestMaxAge: 31536000 preloadPolicy: "RequirePreload" includeSubDomainsPolicy: "RequireIncludeSubDomains" Add a new unsecure route with expose command: oc expose svc service-unsecure --hostname=service-unsecure-test1.internalapps.aiyengar4906.qe.devcluster.openshift.com Error from server (Forbidden): routes.route.openshift.io "service-unsecure" is forbidden: max-age must be set in HSTS annotation Adding a secure route without the annotation: oc create -f edge-route-test.yaml Error from server (Forbidden): error when creating "../essential-docs/test-files/edge-route-test.yaml": routes.route.openshift.io "edge-hsts4" is forbidden: max-age must be set in HSTS annotation -----