Bug 1802311 - Conformance: "should set Forwarded headers appropriately" router test fails on Azure/GCP [NEEDINFO]
Summary: Conformance: "should set Forwarded headers appropriately" router test fails o...
Keywords:
Status: ASSIGNED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Routing
Version: 4.3.z
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
: ---
Assignee: Miciah Dashiel Butler Masters
QA Contact: Hongan Li
URL:
Whiteboard:
: 1811862 1812960 1820277 (view as bug list)
Depends On: 1812960
Blocks: 1820277
TreeView+ depends on / blocked
 
Reported: 2020-02-12 21:15 UTC by slowrie
Modified: 2021-01-05 18:10 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1820274 1820277 1836509 (view as bug list)
Environment:
Last Closed:
Target Upstream Version:
dmace: needinfo? (ccoleman)


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift origin pull 24764 None closed Bug 1802311: e2e: Disable router header test on OVN 2020-12-21 16:12:45 UTC

Description slowrie 2020-02-12 21:15:35 UTC
fail [github.com/openshift/origin/test/extended/router/headers.go:183]: Feb 12 16:07:02.534: Unexpected header: '100.64.0.6' (expected 10.128.2.20); All headers: http.Header{"Accept":[]string{"*/*"}, "Forwarded":[]string{"for=100.64.0.6;host=router-headers.example.com;proto=http;proto-version=\"\""}, "User-Agent":[]string{"curl/7.61.1"}, "X-Forwarded-For":[]string{"100.64.0.6"}, "X-Forwarded-Host":[]string{"router-headers.example.com"}, "X-Forwarded-Port":[]string{"80"}, "X-Forwarded-Proto":[]string{"http"}}

https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-azure-ovn-4.3/428

Comment 1 Benjamin Gilbert 2020-02-14 05:21:39 UTC
Also failing on GCP: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-gcp-ovn-4.3/439

fail [github.com/openshift/origin/test/extended/router/headers.go:183]: Feb 13 20:43:36.168: Unexpected header: '100.64.0.5' (expected 10.131.0.31); All headers: http.Header{"Accept":[]string{"*/*"}, "Forwarded":[]string{"for=100.64.0.5;host=router-headers.example.com;proto=http;proto-version=\"\""}, "User-Agent":[]string{"curl/7.61.1"}, "X-Forwarded-For":[]string{"100.64.0.5"}, "X-Forwarded-Host":[]string{"router-headers.example.com"}, "X-Forwarded-Port":[]string{"80"}, "X-Forwarded-Proto":[]string{"http"}}

The test is currently disabled on all platforms except for Azure/AWS/GCP: https://bugzilla.redhat.com/show_bug.cgi?id=1772125

Comment 2 Alexander Constantinescu 2020-02-17 14:27:23 UTC
I assume this should be assigned to "Component: Routing" given the bug mentioned in #comment 1 : https://bugzilla.redhat.com/show_bug.cgi?id=1772125

Comment 4 Dan Mace 2020-03-06 13:33:52 UTC
Moving this to 4.5 since we don't have time to continue digging into it and I don't know anyone has a justification for blocking the 4.4 release on it.

Comment 5 Dan Mace 2020-03-10 12:29:45 UTC
*** Bug 1811862 has been marked as a duplicate of this bug. ***

Comment 6 Ben Parees 2020-03-10 18:05:19 UTC
this is causing consistent failures in a release informing test job, raising severity.  If the test can't be fixed we should consider whether it needs to be temporarily disabled (while keeping a bug to reenable it), or whether the job in question should really be a release informing job.

Comment 9 Clayton Coleman 2020-03-26 20:45:51 UTC
Test has been disabled, moving back to ASSIGNED to find out why.  This bug may not be deferred from 4.5 without a root cause.

Comment 10 Dan Mace 2020-04-08 11:55:15 UTC
Clayton, can you clarify whether we should block the 4.4 release on a fix? Because the problem's isolated to OVN it wasn't clear to me.

Comment 11 Ben Parees 2020-04-13 13:25:26 UTC
if it's specific to OVN on GCP it doesn't need need to block 4.4 as OVN on AWS+baremetal are the only things we care about in 4.4 as far as i understand it.

Comment 12 Ben Bennett 2020-05-20 13:34:57 UTC
*** Bug 1812960 has been marked as a duplicate of this bug. ***

Comment 13 Miciah Dashiel Butler Masters 2020-05-22 15:07:50 UTC
We've had a lot of discussions around this issue in chat and other Bugzilla reports, so I'll try to resummarize the issue here.

The test is fundamentally flawed.  The intention is to verify that the Forwarded: header has the true source address of a client's request (i.e., the address of the packet as received by the load balancer at the edge of the cluster), but we do not have a reliable way to determine what the client's address is, so we do not have it when we need it to verify the value in the Forwarded: header.

In previous iterations of the test, the client was in a pod on the same cluster, so we were able to get the address from the pod's status, make the request, and compare the pod's address with the response header value.  Making the request from a pod worked by accident on AWS (where the request did not get NATted) but did not work reliably on other platforms (where the request sometimes gets NATted).  Moreover, it did not represent a realistic test scenario (because the request may get hairpinned and never leave the cluster on some platforms), so we changed the test to use the test runner as the client.  However, we do not have a way to determine the test runner's egress address (i.e., the address that the load balancer at the edge of the cluster sees) without using some third-party "what's my IP address"-type service.

So we are currently at an impasse with now clear path to fixing the test.

Comment 14 Miciah Dashiel Butler Masters 2020-05-22 15:13:00 UTC
> So we are currently at an impasse with now clear path to fixing the test.

s/now/no/

Comment 15 Hongan Li 2020-05-25 01:24:28 UTC
Hi Miciah, we use `curl icanhazip.com` to get the client IP in QE test. It that possible to use the same way in e2e test?

Comment 16 Miciah Dashiel Butler Masters 2020-05-26 13:44:22 UTC
(In reply to Hongan Li from comment #15)
> Hi Miciah, we use `curl icanhazip.com` to get the client IP in QE test. It
> that possible to use the same way in e2e test?

Thanks!  We have discussed using something like that, but there were objections to adding a dependency on an external service in CI jobs.  In the short term, if QE is verifying the functionality, then we may be able to punt the CI fix to 4.6.  In the long term, maybe we can set up a "what's my IP address"-type service in the CI cluster itself, but that's a longer discussion.

Is QE verifying that the "Forwarded:" or "X-Forwarded-For:" header has the correct address on the various cloud platforms (in particular, AWS, Azure, and GCP)?

Comment 17 Hongan Li 2020-05-27 08:02:41 UTC
Yes, QE test has covered the case on various cloud platforms.

Just as you said, if the client is the pod inside the Azure/GCP cluster, then "x-forwarded-for" is the pod's IP but not the public IP that get from "what's my IP address" service (like curl icanhazip.com). But for AWS it is not true and the "x-forwarded-for" is the pod's public IP.

If the client is outside the cluster, then the three clouds have same behavior and "x-forwarded-for" is the client's public IP.

Comment 18 Andrew McDermott 2020-05-27 16:09:18 UTC
Moving to 4.6 as not a release blocker or an upgrade blocker. In 4.6 we need to rework the test.

Comment 19 Andrew McDermott 2020-05-28 16:14:22 UTC
*** Bug 1820277 has been marked as a duplicate of this bug. ***

Comment 20 Miciah Dashiel Butler Masters 2020-06-18 19:26:53 UTC
We'll look into setting up a "what's my IP address"-type service in the CI cluster next sprint.

Comment 22 Miciah Dashiel Butler Masters 2020-07-09 05:07:45 UTC
We'll try to make some progress on this in the upcoming sprint.

Comment 23 Miciah Dashiel Butler Masters 2020-07-30 08:31:29 UTC
No progress this sprint.  We'll try to move this forward in the upcoming sprint.

Comment 24 Miciah Dashiel Butler Masters 2020-08-21 05:03:20 UTC
Still no progress.  Moving to the upcoming sprint.

Comment 25 Miciah Dashiel Butler Masters 2020-09-02 17:25:02 UTC
Blocked on https://issues.redhat.com/browse/DPTP-1466.

Comment 26 Andrew McDermott 2020-10-02 15:37:42 UTC
Still waiting on the ticket request in comment #25.

Comment 27 Miciah Dashiel Butler Masters 2020-10-23 15:50:29 UTC
Still blocked on https://issues.redhat.com/browse/DPTP-1466.

Comment 28 Andrew McDermott 2020-11-03 18:37:54 UTC
Dropping the severity as we have manual test coverage (see comment #17). But, equally, we still want to have CI tests so setting severity to high.

Comment 29 Miciah Dashiel Butler Masters 2020-11-14 00:25:05 UTC
Still blocked on https://issues.redhat.com/browse/DPTP-1466.


Note You need to log in before you can comment on or make changes to this bug.