Bug 1903519
| Summary: | Wrong Ingress to Route conversion for wildcard hostnames | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Sergio G. <sgarciam> |
| Component: | Networking | Assignee: | Miheer Salunke <misalunk> |
| Networking sub component: | router | QA Contact: | Arvind iyengar <aiyengar> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | medium | ||
| Priority: | medium | CC: | aconstan, aiyengar, amcdermo, aos-bugs, dana.l.reale, hongli, max.maurer, mfojtik, misalunk, mmasters |
| Version: | 4.6.z | ||
| Target Milestone: | --- | ||
| Target Release: | 4.9.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
Cause: Setting *. entry in spec.rules.host of ingress is not supported by Route.
Consequence: controller fails with Route.route.openshift.io "route-wildcard-2-zkcdj" is invalid: spec.host: Invalid value:
Fix: If user sets *. in the ingress object having the intention for ingress to route conversion then we set wildcard. instead if *. for the route host as route hostname does not suppport *.
Backport warning-
* A user has an ingress with a wildcard host name.
* The ingress is exposed using a third-party ingress controller.
* The user does *not* want the ingress exposed by OpenShift router (for example, maybe OpenShift router is exposed on the Internet, and the third-party ingress controller is internal only; or maybe having a status entry from OpenShift router on the ingress would confuse the third-party ingress controller, as in bug 1935808).
* The router is configured to allow wildcard routes (as described in comment 8).
* The user upgrades to a version of OpenShift with this backported fix.
In this sceneario, the ingress is now exposed by the router when the user only wants it exposed by the third-party ingress controller. This is an admittedly extremely contrived situation, but sometimes users do unexpected things or have different expectations, and exposing an ingress when the user doesn't expect it could be problematic.
Therefore we need to document this new behavior very explicitly in a release note to reduce the risk that users will be caught unawares.
Result: With this fix the ingress controller does not throw the error described in the Consequence.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-10-18 17:28:52 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 2004273 | ||
Working on this now https://github.com/openshift/openshift-controller-manager/pull/169 It is still Work in Progress. @misalunk We are having this same issue in 4.6.12 and want to make sure that it is backported to 4.6.x.
The fix is working fine. Unit test have passed. Passing to QE.
Manual testing ->
oc new-project wildcard
oc new-app django-psql-example
[miheer@miheer openshift-controller-manager]$ cat ingress-wildcard.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress1
annotations:
haproxy.router.openshift.io/timeout: 5m
namespace: wildcard
spec:
rules:
- host: "*.testing.ci-ln-03c9nrb-d5d6b.origin-ci-int-aws.dev.rhcloud.com"
http:
paths:
- path: /
backend:
serviceName: django-psql-example
servicePort: 8080
[miheer@miheer openshift-controller-manager]$
[miheer@miheer openshift-controller-manager]$ oc create -f ingress-wildcard.yaml
W0217 14:05:31.500559 50971 warnings.go:67] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
ingress.networking.k8s.io/ingress1 created
[miheer@miheer openshift-controller-manager]$ oc get routes --all-namespaces
NAMESPACE NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
openshift-authentication oauth-openshift oauth-openshift.apps.ci-ln-03c9nrb-d5d6b.origin-ci-int-aws.dev.rhcloud.com oauth-openshift 6443 passthrough/Redirect None
openshift-console console console-openshift-console.apps.ci-ln-03c9nrb-d5d6b.origin-ci-int-aws.dev.rhcloud.com console https reencrypt/Redirect None
openshift-console downloads downloads-openshift-console.apps.ci-ln-03c9nrb-d5d6b.origin-ci-int-aws.dev.rhcloud.com downloads http edge/Redirect None
openshift-ingress-canary canary canary-openshift-ingress-canary.apps.ci-ln-03c9nrb-d5d6b.origin-ci-int-aws.dev.rhcloud.com ingress-canary 8080 None
openshift-monitoring alertmanager-main alertmanager-main-openshift-monitoring.apps.ci-ln-03c9nrb-d5d6b.origin-ci-int-aws.dev.rhcloud.com alertmanager-main web reencrypt/Redirect None
openshift-monitoring grafana grafana-openshift-monitoring.apps.ci-ln-03c9nrb-d5d6b.origin-ci-int-aws.dev.rhcloud.com grafana https reencrypt/Redirect None
openshift-monitoring prometheus-k8s prometheus-k8s-openshift-monitoring.apps.ci-ln-03c9nrb-d5d6b.origin-ci-int-aws.dev.rhcloud.com prometheus-k8s web reencrypt/Redirect None
openshift-monitoring thanos-querier thanos-querier-openshift-monitoring.apps.ci-ln-03c9nrb-d5d6b.origin-ci-int-aws.dev.rhcloud.com thanos-querier web reencrypt/Redirect None
wildcard django-psql-example django-psql-example-wildcard.apps.ci-ln-03c9nrb-d5d6b.origin-ci-int-aws.dev.rhcloud.com django-psql-example <all> None
wildcard ingress1-xmsqh wildcard.testing.ci-ln-03c9nrb-d5d6b.origin-ci-int-aws.dev.rhcloud.com / django-psql-example web Subdomain
[miheer@miheer openshift-controller-manager]$
[miheer@miheer openshift-controller-manager]$
[miheer@miheer openshift-controller-manager]$
[miheer@miheer openshift-controller-manager]$
Modify ingress. I just changed "*.testing" in spec.rules.host to "*.testing1"
[miheer@miheer openshift-controller-manager]$
[miheer@miheer openshift-controller-manager]$ oc edit ing ingress1
ingress.networking.k8s.io/ingress1 edited
[miheer@miheer openshift-controller-manager]$ oc get routes --all-namespaces
NAMESPACE NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
openshift-authentication oauth-openshift oauth-openshift.apps.ci-ln-03c9nrb-d5d6b.origin-ci-int-aws.dev.rhcloud.com oauth-openshift 6443 passthrough/Redirect None
openshift-console console console-openshift-console.apps.ci-ln-03c9nrb-d5d6b.origin-ci-int-aws.dev.rhcloud.com console https reencrypt/Redirect None
openshift-console downloads downloads-openshift-console.apps.ci-ln-03c9nrb-d5d6b.origin-ci-int-aws.dev.rhcloud.com downloads http edge/Redirect None
openshift-ingress-canary canary canary-openshift-ingress-canary.apps.ci-ln-03c9nrb-d5d6b.origin-ci-int-aws.dev.rhcloud.com ingress-canary 8080 None
openshift-monitoring alertmanager-main alertmanager-main-openshift-monitoring.apps.ci-ln-03c9nrb-d5d6b.origin-ci-int-aws.dev.rhcloud.com alertmanager-main web reencrypt/Redirect None
openshift-monitoring grafana grafana-openshift-monitoring.apps.ci-ln-03c9nrb-d5d6b.origin-ci-int-aws.dev.rhcloud.com grafana https reencrypt/Redirect None
openshift-monitoring prometheus-k8s prometheus-k8s-openshift-monitoring.apps.ci-ln-03c9nrb-d5d6b.origin-ci-int-aws.dev.rhcloud.com prometheus-k8s web reencrypt/Redirect None
openshift-monitoring thanos-querier thanos-querier-openshift-monitoring.apps.ci-ln-03c9nrb-d5d6b.origin-ci-int-aws.dev.rhcloud.com thanos-querier web reencrypt/Redirect None
wildcard django-psql-example django-psql-example-wildcard.apps.ci-ln-03c9nrb-d5d6b.origin-ci-int-aws.dev.rhcloud.com django-psql-example <all> None
wildcard ingress1-gt9hw wildcard.testing1.ci-ln-03c9nrb-d5d6b.origin-ci-int-aws.dev.rhcloud.com / django-psql-example web Subdomain
[miheer@miheer openshift-controller-manager]$
(In reply to Dana from comment #4) > @misalunk We are having this same issue in 4.6.12 and want to > make sure that it is backported to 4.6.x. OK we will see what can be done as I will need to discuss with our team on that. You need run this as pre-requisite for allowing wildcard routes.
[miheer@miheer openshift-controller-manager]$ oc patch -n openshift-ingress-operator ingresscontroller/default --patch '{"spec":{"routeAdmission":{"wildcardPolicy": "WildcardsAllowed"}}}' --type=merge
ingresscontroller.operator.openshift.io/default patched
Verified in "4.8.0-0.ci.test-2021-02-17-044628-ci-ln-1ldbxqb" release version. With this payload the ingress resource could be seen getting translated correctly to the route resource with the required wildcard mapping:
---------
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.8.0-0.ci.test-2021-02-17-044628-ci-ln-1ldbxqb True False 51m Cluster version is 4.8.0-0.ci.test-2021-02-17-044628-ci-ln-1ldbxqb
$ oc patch -n openshift-ingress-operator ingresscontroller/internalapps --patch '{"spec":{"routeAdmission":{"wildcardPolicy": "WildcardsAllowed"}}}' --type=merge
ingresscontroller.operator.openshift.io/default patched
$ cat test-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: unsecure-route
spec:
rules:
- host: '*.test1.internalapps.ci-ln-1ldbxqb-d5d6b.origin-ci-int-aws.dev.rhcloud.com'
http:
paths:
- path: "/"
pathType: "Prefix"
backend:
service:
name: service-unsecure
port:
number: 27017
$ oc get route
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
unsecure-route-2f4w4 wildcard.test1.internalapps.ci-ln-1ldbxqb-d5d6b.origin-ci-int-aws.dev.rhcloud.com ... 1 more / service-unsecure http Subdomain
$ oc get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
unsecure-route <none> *.test1.internalapps.ci-ln-1ldbxqb-d5d6b.origin-ci-int-aws.dev.rhcloud.com apps.ci-ln-1ldbxqb-d5d6b.origin-ci-int-aws.dev.rhcloud.com,internalapps.ci-ln-1ldbxqb-d5d6b.origin-ci-int-aws.dev.rhcloud.com 80 13m
---------
Any news on if this will be backported to 4.6 and 4.7? We discussed this BZ and decided that we will backport the fix to 4.8, 4.7, and 4.6. However, there is a small risk in backporting this change. Suppose the following: * A user has an ingress with a wildcard host name. * The ingress is exposed using a third-party ingress controller. * The user does *not* want the ingress exposed by OpenShift router (for example, maybe OpenShift router is exposed on the Internet, and the third-party ingress controller is internal only; or maybe having a status entry from OpenShift router on the ingress would confuse the third-party ingress controller, as in bug 1935808). * The router is configured to allow wildcard routes (as described in comment 8). * The user upgrades to a version of OpenShift with this backported fix. In this sceneario, the ingress is now exposed by the router when the user only wants it exposed by the third-party ingress controller. This is an admittedly extremely contrived situation, but sometimes users do unexpected things or have different expectations, and exposing an ingress when the user doesn't expect it could be problematic. Therefore we need to document this new behavior very explicitly in a release note to reduce the risk that users will be caught unawares. Miheer, could you write a release note and proceed with the backport? Verified in "4.9.0-0.nightly-2021-08-14-065522" release version, this time. With this payload the ingress resource could be seen getting translated correctly to the route resource with the required wildcard mapping:
---------
oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.9.0-0.nightly-2021-08-14-065522 True False 173m Cluster version is 4.9.0-0.nightly-2021-08-14-065522
$ oc patch -n openshift-ingress-operator ingresscontroller/internalapps --patch '{"spec":{"routeAdmission":{"wildcardPolicy": "WildcardsAllowed"}}}' --type=merge
ingresscontroller.operator.openshift.io/default patched
$ cat test-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: unsecure-route
spec:
rules:
- host: '*.test1.internalapps.aiyengar4916.qe.devcluster.openshift.com'
http:
paths:
- path: "/"
pathType: "Prefix"
backend:
service:
name: service-unsecure
port:
number: 27017
oc get route
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
unsecure-route-ghq74 wildcard.test1.internalapps.aiyengar4916.qe.devcluster.openshift.com ... 1 more / service-unsecure http Subdomain
oc get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
unsecure-route <none> *.test1.internalapps.aiyengar4916.qe.devcluster.openshift.com 80 38s
curl -I http://wildcard.test1.internalapps.aiyengar4916.qe.devcluster.openshift.comserver: nginx/1.18.0
date: Mon, 16 Aug 2021 06:25:03 GMT
content-type: text/html
content-length: 46
last-modified: Mon, 16 Aug 2021 06:14:23 GMT
etag: "611a023f-2e"
accept-ranges: bytes
set-cookie: 491bb4f494b480a2be42674d291d426f=a37c3b8df559f554ce53bfe738097aab; path=/; HttpOnly
cache-control: private
---------
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759 |
Description of problem: - When creating a wildcard ingress the route is never created due to a mismatch in the host format in both resources. - In k8s ingress the hostname is configured as '*.domain.tld' (see [1]) but that's not a valid hostname for a route, and the conversion code copies it literally as seen in [2], failing in the creation. Version-Release number of selected component (if applicable): - 4.6.4 How reproducible: - Always Additional info: - The error can be seen in the controller after setting Debug logLevel: I1202 08:42:11.429894 1 ingress.go:342] Error syncing {dev route-wildcard-2}: Route.route.openshift.io "route-wildcard-2-zkcdj" is invalid: spec.host: Invalid value: "*.dev.apps.sgarciam-02807641.emeashift.support": host must conform to DNS 952 subdomain conventions - Valid k8s ingress: kind: Ingress apiVersion: networking.k8s.io/v1beta1 metadata: name: ingress-1 spec: rules: - host: '*.ingress-1.domain.tld' http: paths: - path: / pathType: Prefix backend: serviceName: ingress-1-service servicePort: 80 - Expected route: kind: Route apiVersion: route.openshift.io/v1 metadata: name: route-1 spec: host: ingress-1.domain.tld to: kind: Service name: ingress-1-service port: targetPort: 80-tcp wildcardPolicy: Subdomain References: [1] https://kubernetes.io/docs/concepts/services-networking/ingress/#hostname-wildcards [2] https://github.com/openshift/openshift-controller-manager/blob/master/pkg/route/ingress/ingress.go#L555