Hide Forgot
Description of problem: The following combinations of Routes with the same host route traffic to the wrong endpoint: [sig-network][Conformance][Feature:Route] Path based routing should work for "edge+Allow" route and "edge+Redirect" route [Suite:openshift/conformance/parallel/minimal] [sig-network][Conformance][Feature:Route] Path based routing should work for "edge+Allow" route and "passthrough" route [Suite:openshift/conformance/parallel/minimal] [sig-network][Conformance][Feature:Route] Path based routing should work for "http-only" route and "edge+Redirect" route [Suite:openshift/conformance/parallel/minimal] One instance where path based routing is especially important is exposing token for http-01 ACME validation but our users has likely suffered too at some point. We have done various hacks in github.com/tnozicka/openshift-acme to make it work but it turned out the bug just moved to a different combination of TLS setting. I've written e2e coverage to prove it's broken and also to keep this working after these bugs gets fixed. Hopefully it will help the team to deliver the fix and be confident about the change. Version-Release number of selected component (if applicable): 4.7.0-0.ci-2020-11-10-061858 How reproducible: Always Steps to Reproduce: 1. https://github.com/openshift/origin/pull/25676 2. 3. Actual results: Expected results: Additional info:
1. The insecure policy allow allows requests to be sent on an insecure scheme HTTP. The insecure policy redirect gets requests sent on an insecure scheme HTTP to a secure scheme HTTPS. So, [sig-network][Conformance][Feature:Route] Path based routing should work for "edge+Allow" route and "edge+Redirect" route [Suite:openshift/conformance/parallel/minimal] is conflicting and an invalid combination 2. [sig-network][Conformance][Feature:Route] Path based routing should work for "http-only" route and "edge+Redirect" route [Suite:openshift/conformance/parallel/minimal] This is also invalid as http-only means terminationPolicy is disabled so http traffic is not allowed. "edge+Redirect" implies that the edge route sets up a redirect for HTTP requests, but that conflicts with having an http-only route handle HTTP requests. The insecure policy redirect requests sent on an insecure scheme HTTP to a secure scheme HTTPS. So it is conflicting and an invalid combination. 3.[sig-network][Conformance][Feature:Route] Path based routing should work for "edge+Allow" route and "passthrough" route [Suite:openshift/conformance/parallel/minimal] Path-based routing is not available when using passthrough TLS, as the router does not terminate TLS in that case and cannot read the contents of the request. So this combination won't work.
> 1. [sig-network][Conformance][Feature:Route] Path based routing should work for "edge+Allow" route and "edge+Redirect" route [Suite:openshift/conformance/parallel/minimal] is conflicting and an invalid combination You are right, the https is conflicting there. I guess the http could still be routed path based before the redirect but that's the same issue as in the other case. I suppose this combination should fail to be admitted. > 2. [sig-network][Conformance][Feature:Route] Path based routing should work for "http-only" route and "edge+Redirect" route [Suite:openshift/conformance/parallel/minimal] I don't think this is conflicting though. The path applies for http scheme so one Route accepts the traffic for "/specific_path" which should be routed to the service, the other accepts traffic for path "/" and redirects it to https. It's just that the router should first look at the path before the redirect to https. I recall an older issue where someone just fixed the order in haproxy config. Also this already works for the other combinations like edge+allow, reencrypt, passthough, ... > 3.[sig-network][Conformance][Feature:Route] Path based routing should work for "edge+Allow" route and "passthrough" route [Suite:openshift/conformance/parallel/minimal] yep, same as 1., the http part could still work, while the https part conflicts. I care primarily about 2. which is important for ACME requests.
(In reply to Tomáš Nožička from comment #5) > > 1. [sig-network][Conformance][Feature:Route] Path based routing should work for "edge+Allow" route and "edge+Redirect" route [Suite:openshift/conformance/parallel/minimal] is conflicting and an invalid combination > > You are right, the https is conflicting there. I guess the http could still > be routed path based before the redirect but that's the same issue as in the > other case. I suppose this combination should fail to be admitted. > > > > 2. [sig-network][Conformance][Feature:Route] Path based routing should work for "http-only" route and "edge+Redirect" route [Suite:openshift/conformance/parallel/minimal] > > I don't think this is conflicting though. The path applies for http scheme > so one Route accepts the traffic for "/specific_path" which should be routed > to the service, the other accepts traffic for path "/" and redirects it to > https. It's just that the router should first look at the path before the > redirect to https. I recall an older issue where someone just fixed the > order in haproxy config. > > Also this already works for the other combinations like edge+allow, > reencrypt, passthough, ... > > I need to see what we can do regarding this. > > 3.[sig-network][Conformance][Feature:Route] Path based routing should work for "edge+Allow" route and "passthrough" route [Suite:openshift/conformance/parallel/minimal] > > yep, same as 1., the http part could still work, while the https part > conflicts. > > > I care primarily about 2. which is important for ACME requests.
*** Bug 2002205 has been marked as a duplicate of this bug. ***
OK I was able to reproduce the issue -> oc get route NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD hello-openshift hello-openshift-test4.apps.pdhamdhe909.qe.devcluster.openshift.com hello-openshift 8080 edge/Redirect None hello-openshift-http-allow hello-openshift-test4.apps.pdhamdhe909.qe.devcluster.openshift.com /test hello-openshift 8080 None The following is redirecting which is not a correct behaviour -> curl -Ik http://hello-openshift-test4.apps.pdhamdhe909.qe.devcluster.openshift.com/test HTTP/1.1 302 Found content-length: 0 location: https://hello-openshift-test4.apps.pdhamdhe909.qe.devcluster.openshift.com/test cache-control: no-cache The following looks a correct behaviour -> curl -Ik https://hello-openshift-test4.apps.pdhamdhe909.qe.devcluster.openshift.com HTTP/1.1 200 OK date: Thu, 09 Sep 2021 12:25:26 GMT content-length: 17 content-type: text/plain; charset=utf-8 set-cookie: b070b57379605dc53d051e15a32208bf=f77047feb2807aff88f79d47220dff0f; path=/; HttpOnly; Secure; SameSite=None cache-control: private curl -Ik http://hello-openshift-test4.apps.pdhamdhe909.qe.devcluster.openshift.com HTTP/1.1 302 Found content-length: 0 location: https://hello-openshift-test4.apps.pdhamdhe909.qe.devcluster.openshift.com/ cache-control: no-cache Now I will be testing my fix for this and get back to you once things are working fine.
The issue is fixed by PR https://github.com/openshift/router/pull/343 and now will be tested by QE. @aiyengar can you please test this ? [miheer@localhost router]$ oc new-project service-route-issue Now using project "service-route-issue" on server "https://api.ci-ln-mzj0m02-f76d1.origin-ci-int-gce.dev.openshift.com:6443". You can add applications to this project with the 'new-app' command. For example, try: oc new-app rails-postgresql-example to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname [miheer@localhost router]$ oc create -f https://raw.githubusercontent.com/openshift/origin/master/examples/hello-openshift/hello-pod.json pod/hello-openshift created [miheer@localhost router]$ oc label pod hello-openshift app=hello-openshift pod/hello-openshift labeled [miheer@localhost router]$ oc create service clusterip hello-openshift --tcp=80:8080 --tcp=443:8443 service/hello-openshift created [miheer@localhost router]$ oc create -f route.yaml route.route.openshift.io/hello-openshift created [miheer@localhost router]$ oc get routes NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD hello-openshift hello-openshift-service-route-issue.apps.ci-ln-mzj0m02-f76d1.origin-ci-int-gce.dev.openshift.com hello-openshift 8080 edge/Redirect None [miheer@localhost router]$ [miheer@localhost router]$ [miheer@localhost router]$ [miheer@localhost router]$ vi route-with-insecure-allow.yaml [miheer@localhost router]$ oc create -f route-with-insecure-allow.yaml route.route.openshift.io/hello-openshift-http-allow created [miheer@localhost router]$ oc get routes NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD hello-openshift hello-openshift-service-route-issue.apps.ci-ln-mzj0m02-f76d1.origin-ci-int-gce.dev.openshift.com hello-openshift 8080 edge/Redirect None hello-openshift-http-allow hello-openshift-service-route-issue.apps.ci-ln-mzj0m02-f76d1.origin-ci-int-gce.dev.openshift.com /test hello-openshift 8080 None [miheer@localhost router]$ [miheer@localhost router]$ [miheer@localhost router]$ [miheer@localhost router]$ curl -Ik http://hello-openshift-service-route-issue.apps.ci-ln-mzj0m02-f76d1.origin-ci-int-gce.dev.openshift.com HTTP/1.1 302 Found content-length: 0 location: https://hello-openshift-service-route-issue.apps.ci-ln-mzj0m02-f76d1.origin-ci-int-gce.dev.openshift.com/ cache-control: no-cache [miheer@localhost router]$ curl -Ik http://hello-openshift-service-route-issue.apps.ci-ln-mzj0m02-f76d1.origin-ci-int-gce.dev.openshift.com/test HTTP/1.1 200 OK date: Sat, 11 Sep 2021 03:03:38 GMT content-length: 17 content-type: text/plain; charset=utf-8 set-cookie: adbec88c545fa91f71ca5c51323dccf8=a572e0a22148d35e8ad873bce8a3dbe3; path=/; HttpOnly cache-control: private [miheer@localhost router]$ [miheer@localhost router]$ curl -Ik https://hello-openshift-service-route-issue.apps.ci-ln-mzj0m02-f76d1.origin-ci-int-gce.dev.openshift.com/ HTTP/1.1 200 OK date: Sat, 11 Sep 2021 03:08:13 GMT content-length: 17 content-type: text/plain; charset=utf-8 set-cookie: 29df4ae9f33eb1d50113228c1151c2c9=a572e0a22148d35e8ad873bce8a3dbe3; path=/; HttpOnly; Secure; SameSite=None cache-control: private [miheer@localhost router]$
melvinjoseph@mjoseph-mac Downloads % oc get all NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 172.30.0.1 <none> 443/TCP 72m service/openshift ExternalName <none> kubernetes.default.svc.cluster.local <none> 68m melvinjoseph@mjoseph-mac Downloads % oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.10.0-0.ci.test-2021-09-15-083445-ci-ln-574nt6k-latest True False 54m Cluster version is 4.10.0-0.ci.test-2021-09-15-083445-ci-ln-574nt6k-latest melvinjoseph@mjoseph-mac Downloads % oc new-project service-route-issue Now using project "service-route-issue" on server "https://api.ci-ln-574nt6k-d5d6b.origin-ci-int-aws.dev.rhcloud.com:6443". You can add applications to this project with the 'new-app' command. For example, try: oc new-app rails-postgresql-example to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname melvinjoseph@mjoseph-mac Downloads % oc create -f https://raw.githubusercontent.com/openshift/origin/master/examples/hello-openshift/hello-pod.json pod/hello-openshift created melvinjoseph@mjoseph-mac Downloads % oc label pod hello-openshift app=hello-openshift pod/hello-openshift labeled melvinjoseph@mjoseph-mac Downloads % oc get pod NAME READY STATUS RESTARTS AGE hello-openshift 1/1 Running 0 43s melvinjoseph@mjoseph-mac Downloads % oc create service clusterip hello-openshift --tcp=80:8080 --tcp=443:8443 service/hello-openshift created melvinjoseph@mjoseph-mac Downloads % vi route.yaml melvinjoseph@mjoseph-mac Downloads % oc create -f route.yaml route.route.openshift.io/hello-openshift created melvinjoseph@mjoseph-mac Downloads % oc get routes NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD hello-openshift hello-openshift-service-route-issue.apps.ci-ln-574nt6k-d5d6b.origin-ci-int-aws.dev.rhcloud.com hello-openshift 8080 edge None melvinjoseph@mjoseph-mac Downloads % vi route-with-insecure-allow.yaml melvinjoseph@mjoseph-mac Downloads % oc create -f route-with-insecure-allow.yaml route.route.openshift.io/hello-openshift-http-allow created melvinjoseph@mjoseph-mac Downloads % oc get routes NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD hello-openshift hello-openshift-service-route-issue.apps.ci-ln-574nt6k-d5d6b.origin-ci-int-aws.dev.rhcloud.com hello-openshift 8080 edge None hello-openshift-http-allow hello-openshift-service-route-issue.apps.ci-ln-574nt6k-d5d6b.origin-ci-int-aws.dev.rhcloud.com /test hello-openshift 8080 None melvinjoseph@mjoseph-mac Downloads % curl -Ik http://hello-openshift-service-route-issue.apps.ci-ln-574nt6k-d5d6b.origin-ci-int-aws.dev.rhcloud.com/test HTTP/1.1 200 OK date: Wed, 15 Sep 2021 10:23:49 GMT content-length: 17 content-type: text/plain; charset=utf-8 set-cookie: adbec88c545fa91f71ca5c51323dccf8=707d0543c00a76c5dcba1bd51ae5c6a9; path=/; HttpOnly cache-control: private melvinjoseph@mjoseph-mac Downloads % curl -Ik https://hello-openshift-service-route-issue.apps.ci-ln-574nt6k-d5d6b.origin-ci-int-aws.dev.rhcloud.com/ HTTP/1.1 200 OK date: Wed, 15 Sep 2021 10:24:41 GMT content-length: 17 content-type: text/plain; charset=utf-8 set-cookie: 29df4ae9f33eb1d50113228c1151c2c9=707d0543c00a76c5dcba1bd51ae5c6a9; path=/; HttpOnly; Secure; SameSite=None cache-control: private
Verified: Tested
melvinjoseph@mjoseph-mac Downloads % oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.10.0-0.ci.test-2021-09-27-061328-ci-ln-03l14wt-latest True False 39m Cluster version is 4.10.0-0.ci.test-2021-09-27-061328-ci-ln-03l14wt-latest melvinjoseph@mjoseph-mac Downloads % oc new-project service-route-issue Now using project "service-route-issue" on server "https://api.ci-ln-03l14wt-f76d1.origin-ci-int-gce.dev.openshift.com:6443". You can add applications to this project with the 'new-app' command. For example, try: oc new-app rails-postgresql-example to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname melvinjoseph@mjoseph-mac Downloads % oc create -f https://raw.githubusercontent.com/openshift/origin/master/examples/hello-openshift/hello-pod.json pod/hello-openshift created melvinjoseph@mjoseph-mac Downloads % oc label pod hello-openshift app=hello-openshift pod/hello-openshift labeled melvinjoseph@mjoseph-mac Downloads % oc get po NAME READY STATUS RESTARTS AGE hello-openshift 1/1 Running 0 13s melvinjoseph@mjoseph-mac Downloads % oc create service clusterip hello-openshift --tcp=80:8080 --tcp=443:8443 service/hello-openshift created melvinjoseph@mjoseph-mac Downloads % vi route.yaml melvinjoseph@mjoseph-mac Downloads % oc get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE hello-openshift ClusterIP 172.30.141.237 <none> 80/TCP,443/TCP 118s melvinjoseph@mjoseph-mac Downloads % oc create -f route.yaml route.route.openshift.io/hello-openshift created melvinjoseph@mjoseph-mac Downloads % oc get route NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD hello-openshift hello-openshift-service-route-issue.apps.ci-ln-03l14wt-f76d1.origin-ci-int-gce.dev.openshift.com hello-openshift 8080 edge None melvinjoseph@mjoseph-mac Downloads % melvinjoseph@mjoseph-mac Downloads % melvinjoseph@mjoseph-mac Downloads % melvinjoseph@mjoseph-mac Downloads % vi route-with-insecure-allow.yaml melvinjoseph@mjoseph-mac Downloads % oc create -f route-with-insecure-allow.yaml route.route.openshift.io/hello-openshift-http-allow created melvinjoseph@mjoseph-mac Downloads % oc get routes NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD hello-openshift hello-openshift-service-route-issue.apps.ci-ln-03l14wt-f76d1.origin-ci-int-gce.dev.openshift.com hello-openshift 8080 edge None hello-openshift-http-allow hello-openshift-service-route-issue.apps.ci-ln-03l14wt-f76d1.origin-ci-int-gce.dev.openshift.com /test hello-openshift 8080 None melvinjoseph@mjoseph-mac Downloads % cat route.yaml apiVersion: route.openshift.io/v1 kind: Route metadata: creationTimestamp: null labels: app: hello-openshift name: hello-openshift spec: host: "" port: targetPort: 8080 tls: termination: edge to: kind: "" name: hello-openshift weight: null% melvinjoseph@mjoseph-mac Downloads % cat route-with-insecure-allow.yaml apiVersion: route.openshift.io/v1 kind: Route metadata: creationTimestamp: null labels: app: hello-openshift name: hello-openshift-http-allow spec: host: "hello-openshift-service-route-issue.apps.ci-ln-03l14wt-f76d1.origin-ci-int-gce.dev.openshift.com" port: targetPort: 8080 path: "/test" to: kind: Service name: hello-openshift weight: null status: ingress: null melvinjoseph@mjoseph-mac Downloads % curl -Ik http://hello-openshift-service-route-issue.apps.ci-ln-03l14wt-f76d1.origin-ci-int-gce.dev.openshift.com HTTP/1.0 503 Service Unavailable pragma: no-cache cache-control: private, max-age=0, no-cache, no-store content-type: text/html melvinjoseph@mjoseph-mac Downloads % curl -Ik https://hello-openshift-service-route-issue.apps.ci-ln-03l14wt-f76d1.origin-ci-int-gce.dev.openshift.com HTTP/1.1 200 OK date: Mon, 27 Sep 2021 08:22:32 GMT content-length: 17 content-type: text/plain; charset=utf-8 set-cookie: 29df4ae9f33eb1d50113228c1151c2c9=2b8ce16cb74e24f4b0e4471c264b346d; path=/; HttpOnly; Secure; SameSite=None cache-control: private melvinjoseph@mjoseph-mac Downloads % curl -Ik http://hello-openshift-service-route-issue.apps.ci-ln-03l14wt-f76d1.origin-ci-int-gce.dev.openshift.com HTTP/1.0 503 Service Unavailable pragma: no-cache cache-control: private, max-age=0, no-cache, no-store content-type: text/html melvinjoseph@mjoseph-mac Downloads % curl -Ik http://hello-openshift-service-route-issue.apps.ci-ln-03l14wt-f76d1.origin-ci-int-gce.dev.openshift.com/test HTTP/1.1 200 OK date: Mon, 27 Sep 2021 08:23:16 GMT content-length: 17 content-type: text/plain; charset=utf-8 set-cookie: adbec88c545fa91f71ca5c51323dccf8=2b8ce16cb74e24f4b0e4471c264b346d; path=/; HttpOnly cache-control: private melvinjoseph@mjoseph-mac Downloads %
the PR has been merged into 4.10.0-0.nightly-2021-09-27-211613, seems the robot cannot move it to VERIFIED automatically, so move it to VERIFIED manually.
Miheer, can you open a new PR with the valid test case from https://github.com/openshift/origin/pull/25676 to make sure we have E2E coverage for this issue? We can link a new test-only PR to this BZ to make the history clear but leave the BZ in VERIFIED state as the fix itself has been verified.
(In reply to Miciah Dashiel Butler Masters from comment #21) > Miheer, can you open a new PR with the valid test case from > https://github.com/openshift/origin/pull/25676 to make sure we have E2E > coverage for this issue? We can link a new test-only PR to this BZ to make > the history clear but leave the BZ in VERIFIED state as the fix itself has > been verified. Sure.
Hi, if there is anything that customers should know about this bug or if there are any important workarounds that should be outlined in the bug fixes section OpenShift Container Platform 4.10 release notes, please update the Doc Type and Doc Text fields. If not, can you please mark it as "no doc update"? Thanks!
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056