Bug 1926146

Summary: [sig-network-edge][Conformance][Area:Networking][Feature:Router] The HAProxy router should be able to connect to a service that is idled because a GET on the route will unidle it
Product: OpenShift Container Platform Reporter: Jan Chaloupka <jchaloup>
Component: NetworkingAssignee: Andrew McDermott <amcdermo>
Networking sub component: router QA Contact: Arvind iyengar <aiyengar>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: high CC: aiyengar, amcdermo, aos-bugs, bperkins, jluhrsen, slaznick
Version: 4.7   
Target Milestone: ---   
Target Release: 4.8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
[sig-network-edge][Conformance][Area:Networking][Feature:Router] The HAProxy router should be able to connect to a service that is idled because a GET on the route will unidle it
Last Closed: 2021-07-27 22:42:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1927953, 1944204    

Description Jan Chaloupka 2021-02-08 10:51:13 UTC 
test:
[sig-network-edge][Conformance][Area:Networking][Feature:Router] The HAProxy router should be able to connect to a service that is idled because a GET on the route will unidle it 

is failing frequently in CI, see search results:
https://search.ci.openshift.org/?maxAge=168h&context=1&type=bug%2Bjunit&name=&maxMatches=5&maxBytes=20971520&groupBy=job&search=%5C%5Bsig-network-edge%5C%5D%5C%5BConformance%5C%5D%5C%5BArea%3ANetworking%5C%5D%5C%5BFeature%3ARouter%5C%5D+The+HAProxy+router+should+be+able+to+connect+to+a+service+that+is+idled+because+a+GET+on+the+route+will+unidle+it


https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-gcp-rt-4.7/1358650050154074112

The test fails after g.By("Validating that the idle annotations have been removed from the endpoints") step after timing out after 3 minutes:
```
fail [github.com/openshift/origin/test/extended/router/idle.go:105]: Unexpected error:
    <*errors.errorString | 0xc0002d29b0>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
occurred
```
Comment 1 Andrew McDermott 2021-02-08 18:15:11 UTC 
The CI test runs locally in a loop without failure and with other
origin tests running. My first pass will be just bumping the timeouts
for the things we wait on in the test - currently set at 3m and 1m -
perhaps on a test cluster the latter is not enough.

Marking as blocker- for the moment.
Comment 2 Standa Laznicka 2021-02-10 08:29:56 UTC 
This test is failing 100% cases on proxy:
https://testgrid.k8s.io/redhat-openshift-ocp-release-4.7-informing#periodic-ci-openshift-release-master-ocp-4.7-e2e-aws-proxy
https://testgrid.k8s.io/redhat-openshift-ocp-release-4.8-informing#periodic-ci-openshift-release-master-ocp-4.8-e2e-aws-proxy

I can only assume you forgot to add `Proxy: http.ProxyFromEnvironment` to your transport. Please fix ASAP.
Comment 3 Standa Laznicka 2021-02-10 09:25:19 UTC 
The test is also suffering from 100% failure rate on s390x libvirt installs with "waiting for condition" (I did not proceed further as to why that is):
https://testgrid.k8s.io/redhat-openshift-ocp-release-4.7-informing#release-openshift-origin-installer-e2e-remote-libvirt-s390x-4.7
Comment 4 Andrew McDermott 2021-02-10 17:41:08 UTC 
(In reply to Standa Laznicka from comment #2)
> This test is failing 100% cases on proxy:
> https://testgrid.k8s.io/redhat-openshift-ocp-release-4.7-informing#periodic-
> ci-openshift-release-master-ocp-4.7-e2e-aws-proxy
> https://testgrid.k8s.io/redhat-openshift-ocp-release-4.8-informing#periodic-
> ci-openshift-release-master-ocp-4.8-e2e-aws-proxy
> 
> I can only assume you forgot to add `Proxy: http.ProxyFromEnvironment` to
> your transport. Please fix ASAP.

Proxy passing: https://github.com/openshift/origin/pull/25874#issuecomment-776885859
Comment 6 jamo luhrsen 2021-02-16 00:01:18 UTC 
I noticed the PR attached to this bug was merged 4 days ago, but the test case is still permafailing in this job:
https://testgrid.k8s.io/redhat-openshift-ocp-release-4.7-informing#periodic-ci-openshift-release-master-ocp-4.7-e2e-aws-proxy

I'll re-open it for now, but maybe we want a new bug?
Comment 7 jamo luhrsen 2021-02-22 19:25:23 UTC 
(In reply to jamo luhrsen from comment #6)
> I noticed the PR attached to this bug was merged 4 days ago, but the test
> case is still permafailing in this job:
> https://testgrid.k8s.io/redhat-openshift-ocp-release-4.7-informing#periodic-
> ci-openshift-release-master-ocp-4.7-e2e-aws-proxy
> 
> I'll re-open it for now, but maybe we want a new bug?

still failing. any update?
Comment 8 jamo luhrsen 2021-02-22 19:37:20 UTC 
(In reply to jamo luhrsen from comment #7)
> (In reply to jamo luhrsen from comment #6)
> > I noticed the PR attached to this bug was merged 4 days ago, but the test
> > case is still permafailing in this job:
> > https://testgrid.k8s.io/redhat-openshift-ocp-release-4.7-informing#periodic-
> > ci-openshift-release-master-ocp-4.7-e2e-aws-proxy
> > 
> > I'll re-open it for now, but maybe we want a new bug?
> 
> still failing. any update?

looks like this is the PR to resolve this for 4.7:
https://github.com/openshift/origin/pull/25892
Comment 9 Andrew McDermott 2021-02-25 09:46:17 UTC 
(In reply to jamo luhrsen from comment #7)
> (In reply to jamo luhrsen from comment #6)
> > I noticed the PR attached to this bug was merged 4 days ago, but the test
> > case is still permafailing in this job:
> > https://testgrid.k8s.io/redhat-openshift-ocp-release-4.7-informing#periodic-
> > ci-openshift-release-master-ocp-4.7-e2e-aws-proxy
> > 
> > I'll re-open it for now, but maybe we want a new bug?
> 
> still failing. any update?

The PR originally referenced only made it into 4.8:

 https://github.com/openshift/origin/pull/25874

Updated the BZ links; https://github.com/openshift/origin/pull/25892 is the backport PR for 4.7.
Comment 10 Andrew McDermott 2021-02-25 10:13:28 UTC 
(In reply to jamo luhrsen from comment #6)
> I noticed the PR attached to this bug was merged 4 days ago, but the test
> case is still permafailing in this job:
> https://testgrid.k8s.io/redhat-openshift-ocp-release-4.7-informing#periodic-
> ci-openshift-release-master-ocp-4.7-e2e-aws-proxy
> 
> I'll re-open it for now, but maybe we want a new bug?

This change only landed in 4.8.

I see the test passing in: https://testgrid.k8s.io/redhat-openshift-ocp-release-4.8-informing#periodic-ci-openshift-release-master-ocp-4.8-e2e-aws-proxy

Please can we revalidate this as the test is passing in 4.8?

The 4.7 backport is here: https://bugzilla.redhat.com/show_bug.cgi?id=1927953
Comment 12 Arvind iyengar 2021-03-05 08:02:04 UTC 
In the recent runs, there are no further failures noticed: 
------
https://testgrid.k8s.io/redhat-openshift-ocp-release-4.8-informing#periodic-ci-openshift-release-master-nightly-4.8-e2e-aws-proxy
https://testgrid.k8s.io/redhat-openshift-ocp-release-4.8-informing#periodic-ci-openshift-release-master-nightly-4.8-e2e-aws-fips
------

Hence marking as 'verified'
Comment 13 Andrew McDermott 2021-03-29 10:54:45 UTC 
Correcting target release based on https://github.com/openshift/origin/pull/25893#issuecomment-777860169
Comment 19 errata-xmlrpc 2021-07-27 22:42:10 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438