Bug 2000754 - IPerf2 tests should be lower
Summary: IPerf2 tests should be lower
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.9
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.10.0
Assignee: Pierre Prinetti
QA Contact: Jon Uriarte
URL:
Whiteboard:
: 2006346 (view as bug list)
Depends On:
Blocks: 2010225
TreeView+ depends on / blocked
 
Reported: 2021-09-02 19:55 UTC by Stephen Benjamin
Modified: 2022-03-10 16:07 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 2010225 (view as bug list)
Environment:
job=periodic-ci-openshift-release-master-ci-4.9-e2e-openstack-parallel=all job=release-openshift-ocp-installer-e2e-aws-upi-4.10=all job=release-openshift-ocp-installer-e2e-aws-upi-4.9=all job=release-openshift-origin-installer-old-rhcos-e2e-aws-4.10=all [sig-network] Networking IPerf2 [Feature:Networking-Performance] should run iperf2
Last Closed: 2022-03-10 16:07:01 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github kubernetes kubernetes pull 104845 0 None Merged e2e iperf2 change threshold to 10MBps = 80 Mbps 2021-10-01 08:09:46 UTC
Github kubernetes kubernetes pull 104859 0 None Merged Automated cherry pick of #104845: e2e iperf2 change threshold to 10MBps = 80 Mbps 2021-10-01 08:09:49 UTC
Github openshift kubernetes pull 980 0 None Merged Bug 2000754: UPSTREAM: 104865: e2e iperf2 change threshold to 10MBps = 80 Mbps 2021-10-01 08:09:50 UTC
Github openshift origin pull 26491 0 None Merged Bug 2000754: vendor: e2e iperf2 change threshold to 10MBps 2021-10-04 08:45:03 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:07:30 UTC

Description Stephen Benjamin 2021-09-02 19:55:14 UTC
periodic-ci-openshift-release-master-ci-4.9-e2e-openstack-parallel

is failing frequently in CI, see:
https://sippy.ci.openshift.org/sippy-ng/jobs/4.9/analysis?filters=%7B%22items%22%3A%5B%7B%22columnField%22%3A%22name%22%2C%22operatorValue%22%3A%22equals%22%2C%22value%22%3A%22periodic-ci-openshift-release-master-ci-4.9-e2e-openstack-parallel%22%7D%5D%7D&tests=%5Bsig-network%5D%20Networking%20IPerf2%20%5BFeature%3ANetworking-Performance%5D%20should%20run%20iperf2%20%5BSuite%3Aopenshift%2Fconformance%2Fparallel%5D%20%5BSuite%3Ak8s%5D

and

https://testgrid.k8s.io/redhat-openshift-ocp-release-4.9-informing#periodic-ci-openshift-release-master-ci-4.9-e2e-openstack-parallel

Example job:

https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.9-e2e-openstack-parallel/1433006778466963456

[sig-network] Networking IPerf2 [Feature:Networking-Performance] should run iperf2 [Suite:openshift/conformance/parallel] [Suite:k8s] expand_less


fail [github.com/onsi/ginkgo.0-origin.0+incompatible/internal/leafnodes/runner.go:113]: Sep  1 10:54:07.533: iperf2 MB/s received below baseline of 90 for client hdqi2rrj-96d0b-2zm5q-worker-0-kl4r9 to server hdqi2rrj-96d0b-2zm5q-worker-0-b52nq: 59

Comment 1 egarcia 2021-09-08 15:17:09 UTC
It seems like these jobs would have otherwise passed if the Iperf2 job was not running. We don't care to test the networking performance of our CI hardware anyway, so we are moving to skip this test.

Comment 2 egarcia 2021-09-08 18:24:40 UTC
post: https://github.com/kubernetes/kubernetes/pull/104845

Comment 3 egarcia 2021-09-09 13:37:48 UTC
Backport to kubernetes 1.22

Comment 5 Matthew Booth 2021-09-14 16:11:40 UTC
The upstream fix has been backported to 1.22, but has not yet been included in a tagged release. The most recent tag was 1.22.1, so we can expect the fix to be included in 1.22.2.

I believe the path to resolving this bug is:

1. Commit a0bebe70ef268e097d775a8e76a725e81a548fad is included in upstream tag 1.22.2
2. openshift/kubernetes:master rebases on upstream tag 1.22.2
3. openshift/origin revendors k8s.io/kubernetes on openshift/kubernetes:master

Ideally all of these things should happen automatically over the next few weeks.

Comment 6 Stephen Benjamin 2021-09-27 14:47:53 UTC
*** Bug 2006346 has been marked as a duplicate of this bug. ***

Comment 7 Stephen Benjamin 2021-09-27 14:54:10 UTC
> Ideally all of these things should happen automatically over the next few weeks.

This is failing quite a bit, not on just the openstack job -- can we bring the fix from upstream directly? https://github.com/openshift/kubernetes/pull/980

Comment 10 Martin André 2021-10-01 08:34:52 UTC
Even though the openshift/kubernetes change merged 2 days ago, we're still seeing issues in our CI:
https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.10-e2e-openstack-parallel/1443518614392016896

I assume it needs to be re-vendored in openshift/origin now?

Comment 13 Pierre Prinetti 2021-10-04 08:44:46 UTC
Looks fine, I am not seeing new runs failed because of iPerf2 for now.

Comment 14 Martin André 2021-10-04 09:10:41 UTC
Moving back to ON_QA as we need to move the bug to VERIFIED in order for the backport to be valid.

Comment 15 Martin André 2021-10-04 09:12:04 UTC
Self-verifying. This iperf2 test is no longer an issue on 4.10 as shown by:

https://search.ci.openshift.org/?search=should+run+iperf2&maxAge=48h&context=1&type=bug%2Bjunit&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

Comment 16 ShiftStack Bugwatcher 2021-11-25 16:12:22 UTC
Removing the Triaged keyword because:

* the QE automation assessment (flag qe_test_coverage) is missing

Comment 20 errata-xmlrpc 2022-03-10 16:07:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.