Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1921857

Summary: Test flake when hitting router in e2e tests due to one router not being up to date
Product: OpenShift Container Platform Reporter: Clayton Coleman <ccoleman>
Component: NetworkingAssignee: Clayton Coleman <ccoleman>
Networking sub component: router QA Contact: Arvind iyengar <aiyengar>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: aiyengar, amcdermo, aos-bugs, sgreene
Version: 4.7   
Target Milestone: ---   
Target Release: 4.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
[sig-network][Feature:Router] The HAProxy router should set Forwarded headers appropriately
Last Closed: 2021-02-24 15:57:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Clayton Coleman 2021-01-28 18:02:59 UTC 
https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-sdn-network-stress-4.7/1353930135601942528

[sig-network][Feature:Router] The HAProxy router should set Forwarded headers appropriately [Suite:openshift/conformance/parallel]

Flaked with

fail [github.com/openshift/origin/test/extended/router/headers.go:112]: Unexpected error:
    <*errors.errorString | 0xc002ce8970>: {
        s: "host command failed: error running /usr/bin/kubectl ...

because the curl command returned a 503.  The stanza above that line is supposed to have verified the route and backends were available.  This happens because the first router instance has the backend, we progress to this line, and then our next test is load balanced to the second router instance and the backend isn't loaded, so the test fails.

AWS randomly load balances, and we've hit this before in other tests, so we should be able to require that N requests in a row pass before we continue (high probability all router instances are updated).  This helper is used in several tests so it should clear a number of flakes.

https://bugzilla.redhat.com/show_bug.cgi?id=1871939 was opened about a different issue, this should only show up when router/headers.go:112 returns 503 which is much less frequent.
Comment 2 Arvind iyengar 2021-02-02 09:29:40 UTC 
The Flaks has ceased to occur in the latest runs post the inclusion of the fix hence marking this bug as 'verified':

https://testgrid.k8s.io/redhat-openshift-ocp-release-4.7-informing#release-openshift-ocp-installer-e2e-aws-fips-4.7
Comment 5 errata-xmlrpc 2021-02-24 15:57:12 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633