fail [github.com/openshift/origin/test/extended/router/headers.go:183]: Feb 12 16:07:02.534: Unexpected header: '100.64.0.6' (expected 10.128.2.20); All headers: http.Header{"Accept":[]string{"*/*"}, "Forwarded":[]string{"for=100.64.0.6;host=router-headers.example.com;proto=http;proto-version=\"\""}, "User-Agent":[]string{"curl/7.61.1"}, "X-Forwarded-For":[]string{"100.64.0.6"}, "X-Forwarded-Host":[]string{"router-headers.example.com"}, "X-Forwarded-Port":[]string{"80"}, "X-Forwarded-Proto":[]string{"http"}} https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-azure-ovn-4.3/428
Also failing on GCP: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-gcp-ovn-4.3/439 fail [github.com/openshift/origin/test/extended/router/headers.go:183]: Feb 13 20:43:36.168: Unexpected header: '100.64.0.5' (expected 10.131.0.31); All headers: http.Header{"Accept":[]string{"*/*"}, "Forwarded":[]string{"for=100.64.0.5;host=router-headers.example.com;proto=http;proto-version=\"\""}, "User-Agent":[]string{"curl/7.61.1"}, "X-Forwarded-For":[]string{"100.64.0.5"}, "X-Forwarded-Host":[]string{"router-headers.example.com"}, "X-Forwarded-Port":[]string{"80"}, "X-Forwarded-Proto":[]string{"http"}} The test is currently disabled on all platforms except for Azure/AWS/GCP: https://bugzilla.redhat.com/show_bug.cgi?id=1772125
I assume this should be assigned to "Component: Routing" given the bug mentioned in #comment 1 : https://bugzilla.redhat.com/show_bug.cgi?id=1772125
This seems to be a problem only in the OVN jobs on Azure and GCP since 4.3. Almost certainly some sort of SDN issue. We'll do a little more triage. https://testgrid.k8s.io/redhat-openshift-ocp-release-4.3-informing#release-openshift-ocp-installer-e2e-gcp-ovn-4.3 https://testgrid.k8s.io/redhat-openshift-ocp-release-4.3-informing#release-openshift-ocp-installer-e2e-azure-ovn-4.3 https://testgrid.k8s.io/redhat-openshift-ocp-release-4.4-informing#release-openshift-ocp-installer-e2e-gcp-ovn-4.4 https://testgrid.k8s.io/redhat-openshift-ocp-release-4.4-informing#release-openshift-ocp-installer-e2e-azure-ovn-4.4
Moving this to 4.5 since we don't have time to continue digging into it and I don't know anyone has a justification for blocking the 4.4 release on it.
*** Bug 1811862 has been marked as a duplicate of this bug. ***
this is causing consistent failures in a release informing test job, raising severity. If the test can't be fixed we should consider whether it needs to be temporarily disabled (while keeping a bug to reenable it), or whether the job in question should really be a release informing job.
Test has been disabled, moving back to ASSIGNED to find out why. This bug may not be deferred from 4.5 without a root cause.
Clayton, can you clarify whether we should block the 4.4 release on a fix? Because the problem's isolated to OVN it wasn't clear to me.
if it's specific to OVN on GCP it doesn't need need to block 4.4 as OVN on AWS+baremetal are the only things we care about in 4.4 as far as i understand it.
*** Bug 1812960 has been marked as a duplicate of this bug. ***
We've had a lot of discussions around this issue in chat and other Bugzilla reports, so I'll try to resummarize the issue here. The test is fundamentally flawed. The intention is to verify that the Forwarded: header has the true source address of a client's request (i.e., the address of the packet as received by the load balancer at the edge of the cluster), but we do not have a reliable way to determine what the client's address is, so we do not have it when we need it to verify the value in the Forwarded: header. In previous iterations of the test, the client was in a pod on the same cluster, so we were able to get the address from the pod's status, make the request, and compare the pod's address with the response header value. Making the request from a pod worked by accident on AWS (where the request did not get NATted) but did not work reliably on other platforms (where the request sometimes gets NATted). Moreover, it did not represent a realistic test scenario (because the request may get hairpinned and never leave the cluster on some platforms), so we changed the test to use the test runner as the client. However, we do not have a way to determine the test runner's egress address (i.e., the address that the load balancer at the edge of the cluster sees) without using some third-party "what's my IP address"-type service. So we are currently at an impasse with now clear path to fixing the test.
> So we are currently at an impasse with now clear path to fixing the test. s/now/no/
Hi Miciah, we use `curl icanhazip.com` to get the client IP in QE test. It that possible to use the same way in e2e test?
(In reply to Hongan Li from comment #15) > Hi Miciah, we use `curl icanhazip.com` to get the client IP in QE test. It > that possible to use the same way in e2e test? Thanks! We have discussed using something like that, but there were objections to adding a dependency on an external service in CI jobs. In the short term, if QE is verifying the functionality, then we may be able to punt the CI fix to 4.6. In the long term, maybe we can set up a "what's my IP address"-type service in the CI cluster itself, but that's a longer discussion. Is QE verifying that the "Forwarded:" or "X-Forwarded-For:" header has the correct address on the various cloud platforms (in particular, AWS, Azure, and GCP)?
Yes, QE test has covered the case on various cloud platforms. Just as you said, if the client is the pod inside the Azure/GCP cluster, then "x-forwarded-for" is the pod's IP but not the public IP that get from "what's my IP address" service (like curl icanhazip.com). But for AWS it is not true and the "x-forwarded-for" is the pod's public IP. If the client is outside the cluster, then the three clouds have same behavior and "x-forwarded-for" is the client's public IP.
Moving to 4.6 as not a release blocker or an upgrade blocker. In 4.6 we need to rework the test.
*** Bug 1820277 has been marked as a duplicate of this bug. ***
We'll look into setting up a "what's my IP address"-type service in the CI cluster next sprint.
We'll try to make some progress on this in the upcoming sprint.
No progress this sprint. We'll try to move this forward in the upcoming sprint.
Still no progress. Moving to the upcoming sprint.
Blocked on https://issues.redhat.com/browse/DPTP-1466.
Still waiting on the ticket request in comment #25.
Still blocked on https://issues.redhat.com/browse/DPTP-1466.
Dropping the severity as we have manual test coverage (see comment #17). But, equally, we still want to have CI tests so setting severity to high.
*** Bug 1871939 has been marked as a duplicate of this bug. ***
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days