Bug 2000754

Summary: IPerf2 tests should be lower
Product: OpenShift Container Platform Reporter: Stephen Benjamin <stbenjam>
Component: InstallerAssignee: Pierre Prinetti <pprinett>
Installer sub component: OpenShift on OpenStack QA Contact: Jon Uriarte <juriarte>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: m.andre, mbooth, pprinett, sippy
Version: 4.9Keywords: Reopened
Target Milestone: ---   
Target Release: 4.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 2010225 (view as bug list) Environment:
job=periodic-ci-openshift-release-master-ci-4.9-e2e-openstack-parallel=all job=release-openshift-ocp-installer-e2e-aws-upi-4.10=all job=release-openshift-ocp-installer-e2e-aws-upi-4.9=all job=release-openshift-origin-installer-old-rhcos-e2e-aws-4.10=all [sig-network] Networking IPerf2 [Feature:Networking-Performance] should run iperf2
Last Closed: 2022-03-10 16:07:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2010225    

Description Stephen Benjamin 2021-09-02 19:55:14 UTC
periodic-ci-openshift-release-master-ci-4.9-e2e-openstack-parallel

is failing frequently in CI, see:
https://sippy.ci.openshift.org/sippy-ng/jobs/4.9/analysis?filters=%7B%22items%22%3A%5B%7B%22columnField%22%3A%22name%22%2C%22operatorValue%22%3A%22equals%22%2C%22value%22%3A%22periodic-ci-openshift-release-master-ci-4.9-e2e-openstack-parallel%22%7D%5D%7D&tests=%5Bsig-network%5D%20Networking%20IPerf2%20%5BFeature%3ANetworking-Performance%5D%20should%20run%20iperf2%20%5BSuite%3Aopenshift%2Fconformance%2Fparallel%5D%20%5BSuite%3Ak8s%5D

and

https://testgrid.k8s.io/redhat-openshift-ocp-release-4.9-informing#periodic-ci-openshift-release-master-ci-4.9-e2e-openstack-parallel

Example job:

https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.9-e2e-openstack-parallel/1433006778466963456

[sig-network] Networking IPerf2 [Feature:Networking-Performance] should run iperf2 [Suite:openshift/conformance/parallel] [Suite:k8s] expand_less


fail [github.com/onsi/ginkgo.0-origin.0+incompatible/internal/leafnodes/runner.go:113]: Sep  1 10:54:07.533: iperf2 MB/s received below baseline of 90 for client hdqi2rrj-96d0b-2zm5q-worker-0-kl4r9 to server hdqi2rrj-96d0b-2zm5q-worker-0-b52nq: 59

Comment 1 egarcia 2021-09-08 15:17:09 UTC
It seems like these jobs would have otherwise passed if the Iperf2 job was not running. We don't care to test the networking performance of our CI hardware anyway, so we are moving to skip this test.

Comment 2 egarcia 2021-09-08 18:24:40 UTC
post: https://github.com/kubernetes/kubernetes/pull/104845

Comment 3 egarcia 2021-09-09 13:37:48 UTC
Backport to kubernetes 1.22

Comment 5 Matthew Booth 2021-09-14 16:11:40 UTC
The upstream fix has been backported to 1.22, but has not yet been included in a tagged release. The most recent tag was 1.22.1, so we can expect the fix to be included in 1.22.2.

I believe the path to resolving this bug is:

1. Commit a0bebe70ef268e097d775a8e76a725e81a548fad is included in upstream tag 1.22.2
2. openshift/kubernetes:master rebases on upstream tag 1.22.2
3. openshift/origin revendors k8s.io/kubernetes on openshift/kubernetes:master

Ideally all of these things should happen automatically over the next few weeks.

Comment 6 Stephen Benjamin 2021-09-27 14:47:53 UTC
*** Bug 2006346 has been marked as a duplicate of this bug. ***

Comment 7 Stephen Benjamin 2021-09-27 14:54:10 UTC
> Ideally all of these things should happen automatically over the next few weeks.

This is failing quite a bit, not on just the openstack job -- can we bring the fix from upstream directly? https://github.com/openshift/kubernetes/pull/980

Comment 10 Martin André 2021-10-01 08:34:52 UTC
Even though the openshift/kubernetes change merged 2 days ago, we're still seeing issues in our CI:
https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.10-e2e-openstack-parallel/1443518614392016896

I assume it needs to be re-vendored in openshift/origin now?

Comment 13 Pierre Prinetti 2021-10-04 08:44:46 UTC
Looks fine, I am not seeing new runs failed because of iPerf2 for now.

Comment 14 Martin André 2021-10-04 09:10:41 UTC
Moving back to ON_QA as we need to move the bug to VERIFIED in order for the backport to be valid.

Comment 15 Martin André 2021-10-04 09:12:04 UTC
Self-verifying. This iperf2 test is no longer an issue on 4.10 as shown by:

https://search.ci.openshift.org/?search=should+run+iperf2&maxAge=48h&context=1&type=bug%2Bjunit&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

Comment 16 ShiftStack Bugwatcher 2021-11-25 16:12:22 UTC
Removing the Triaged keyword because:

* the QE automation assessment (flag qe_test_coverage) is missing

Comment 20 errata-xmlrpc 2022-03-10 16:07:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056