Bug 1956049 - subctl benchmark throughput iperf3 error
Summary: subctl benchmark throughput iperf3 error
Keywords:
Status: POST
Alias: None
Product: Red Hat Advanced Cluster Management for Kubernetes
Classification: Red Hat
Component: Submariner
Version: rhacm-2.2.z
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: ---
Assignee: Sridhar Gaddam
QA Contact: Noam Manos
Christopher Dawson
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-05-02 10:36 UTC by Noam Manos
Modified: 2021-05-04 07:53 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:
ming: rhacm-2.2.z+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github open-cluster-management backlog issues 12151 0 None None None 2021-05-03 16:11:55 UTC
Github submariner-io shipyard pull 541 0 None open Increase connect-timeout in benchmark tests 2021-05-03 14:59:18 UTC
Github submariner-io shipyard pull 543 0 None closed Increase connect-timeout in benchmark tests 2021-05-04 07:53:21 UTC

Description Noam Manos 2021-05-02 10:36:53 UTC
Description of problem:
subctl benchmark latency works, but subctl benchmark throughput fails on:
iperf3: error - unable to send control message: Bad file descriptor

Version-Release number of selected component (if applicable):
subctl 0.9

How reproducible:
?

Steps to Reproduce:
Run subctl benchmark throughput

Actual results:

$ subctl benchmark latency /mnt/skynet-data/pkomarov-env/pkomarov-cluster-a/auth/kubeconfig /mnt/skynet-data/pkomarov-env/ocpup/.config/cl2/auth/kubeconfig

MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.139.1.40 (10.139.) port 0 AF_INET : first burst 0
Minimum Latency Microseconds,Mean Latency Microseconds,Maximum Latency Microseconds,Stddev Latency Microseconds,Transaction Rate Tran/s
18567,19067.27,43144,1544.19,52.344
Minimum Latency Microseconds:	18567
Mean Latency Microseconds:	19067.27
Maximum Latency Microseconds:	43144
Stddev Latency Microseconds:	1544.19
Transaction Rate Tran/s:	52.344
MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.136.0.88 (10.136.) port 0 AF_INET : first burst 0
Minimum Latency Microseconds,Mean Latency Microseconds,Maximum Latency Microseconds,Stddev Latency Microseconds,Transaction Rate Tran/s
19105,19659.30,38983,1280.60,50.767
Minimum Latency Microseconds:	19105
Mean Latency Microseconds:	19659.30
Maximum Latency Microseconds:	38983
Stddev Latency Microseconds:	1280.60
Transaction Rate Tran/s:	50.767
 
$ subctl benchmark throughput /mnt/skynet-data/pkomarov-env/pkomarov-cluster-a/auth/kubeconfig /mnt/skynet-data/pkomarov-env/ocpup/.config/cl2/auth/kubeconfig

iperf3: error - unable to send control message: Bad file descriptor
[going to retry]
iperf3: error - unable to send control message: Bad file descriptor
[going to retry]
iperf3: error - unable to send control message: Bad file descriptor
[going to retry]
iperf3: error - unable to send control message: Bad file descriptor
[going to retry]

iperf3: error - unable to send control message: Bad file descriptor
[going to retry]
iperf3: error - unable to send control message: Bad file descriptor
[going to retry]
iperf3: error - unable to send control message: Bad file descriptor
[going to retry]
iperf3: error - unable to send control message: Bad file descriptor
[going to retry]

Comment 1 Nir Yechiel 2021-05-02 15:53:32 UTC
There are no logs nor access information for this particular environment, but I did some further testing and here is my findings:

1. Locally with kind, this error is not seen -- tested with both devel and 0.9.0.

2. With another OCP environment provided by QE, this error is indeed seen. This was with downstream 0.8, on clusters running OCP 4.5.0 and 4.7.8, both with OpenShiftSDN. I can confirm that Submariner is deployed properly, tunnels are up, and E2E tests are running without failures. So this seems to be specific to the downstream images or OCP.

Comment 2 Maayan Friedman 2021-05-03 06:00:40 UTC
When testing with KIND, if one of test pod will end up running on the control-plane the test (throughput or latency) will end up with error

Comment 3 Sridhar Gaddam 2021-05-03 15:05:17 UTC
Issue is fixed in the upstream devel branch via the following PR - https://github.com/submariner-io/shipyard/pull/541

Comment 4 Sridhar Gaddam 2021-05-04 07:53:25 UTC
Backported and merged in the upstream release 0.9 branch 
https://github.com/submariner-io/shipyard/pull/543


Note You need to log in before you can comment on or make changes to this bug.