Bug 1956049

Summary: subctl benchmark throughput iperf3 error
Product: Red Hat Advanced Cluster Management for Kubernetes Reporter: Noam Manos <nmanos>
Component: SubmarinerAssignee: Sridhar Gaddam <sgaddam>
Status: CLOSED ERRATA QA Contact: Noam Manos <nmanos>
Severity: medium Docs Contact: Christopher Dawson <cdawson>
Priority: unspecified    
Version: rhacm-2.3CC: nyechiel, sgaddam, smattar, tfreger
Target Milestone: ---Flags: smattar: rhacm-2.3+
Target Release: rhacm-2.3   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-08-05 13:18:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Noam Manos 2021-05-02 10:36:53 UTC
Description of problem:
subctl benchmark latency works, but subctl benchmark throughput fails on:
iperf3: error - unable to send control message: Bad file descriptor

Version-Release number of selected component (if applicable):
subctl 0.9

How reproducible:
?

Steps to Reproduce:
Run subctl benchmark throughput

Actual results:

$ subctl benchmark latency /mnt/skynet-data/pkomarov-env/pkomarov-cluster-a/auth/kubeconfig /mnt/skynet-data/pkomarov-env/ocpup/.config/cl2/auth/kubeconfig

MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.139.1.40 (10.139.) port 0 AF_INET : first burst 0
Minimum Latency Microseconds,Mean Latency Microseconds,Maximum Latency Microseconds,Stddev Latency Microseconds,Transaction Rate Tran/s
18567,19067.27,43144,1544.19,52.344
Minimum Latency Microseconds:	18567
Mean Latency Microseconds:	19067.27
Maximum Latency Microseconds:	43144
Stddev Latency Microseconds:	1544.19
Transaction Rate Tran/s:	52.344
MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.136.0.88 (10.136.) port 0 AF_INET : first burst 0
Minimum Latency Microseconds,Mean Latency Microseconds,Maximum Latency Microseconds,Stddev Latency Microseconds,Transaction Rate Tran/s
19105,19659.30,38983,1280.60,50.767
Minimum Latency Microseconds:	19105
Mean Latency Microseconds:	19659.30
Maximum Latency Microseconds:	38983
Stddev Latency Microseconds:	1280.60
Transaction Rate Tran/s:	50.767
 
$ subctl benchmark throughput /mnt/skynet-data/pkomarov-env/pkomarov-cluster-a/auth/kubeconfig /mnt/skynet-data/pkomarov-env/ocpup/.config/cl2/auth/kubeconfig

iperf3: error - unable to send control message: Bad file descriptor
[going to retry]
iperf3: error - unable to send control message: Bad file descriptor
[going to retry]
iperf3: error - unable to send control message: Bad file descriptor
[going to retry]
iperf3: error - unable to send control message: Bad file descriptor
[going to retry]

iperf3: error - unable to send control message: Bad file descriptor
[going to retry]
iperf3: error - unable to send control message: Bad file descriptor
[going to retry]
iperf3: error - unable to send control message: Bad file descriptor
[going to retry]
iperf3: error - unable to send control message: Bad file descriptor
[going to retry]

Comment 1 Nir Yechiel 2021-05-02 15:53:32 UTC
There are no logs nor access information for this particular environment, but I did some further testing and here is my findings:

1. Locally with kind, this error is not seen -- tested with both devel and 0.9.0.

2. With another OCP environment provided by QE, this error is indeed seen. This was with downstream 0.8, on clusters running OCP 4.5.0 and 4.7.8, both with OpenShiftSDN. I can confirm that Submariner is deployed properly, tunnels are up, and E2E tests are running without failures. So this seems to be specific to the downstream images or OCP.

Comment 2 Maayan Friedman 2021-05-03 06:00:40 UTC
When testing with KIND, if one of test pod will end up running on the control-plane the test (throughput or latency) will end up with error

Comment 3 Sridhar Gaddam 2021-05-03 15:05:17 UTC
Issue is fixed in the upstream devel branch via the following PR - https://github.com/submariner-io/shipyard/pull/541

Comment 4 Sridhar Gaddam 2021-05-04 07:53:25 UTC
Backported and merged in the upstream release 0.9 branch 
https://github.com/submariner-io/shipyard/pull/543

Comment 8 Noam Manos 2021-07-06 07:37:08 UTC
Verified:
https://qe-jenkins-csb-skynet.apps.ocp4.prod.psi.redhat.com/job/Submariner-0.9-OSP-AWS-SDN/133/Test-Report/


$ subctl benchmark latency /mnt/skynet-data/nmanos-env/nmanos-cluster-a/auth/kubeconfig /mnt/skynet-data/nmanos-env/ocpup/.config/cl1/auth/kubeconfig  --verbose

Performing latency tests
Creating kubernetes clients
Setting new cluster ID "nmanos-cluster-a", previous cluster ID was "api-nmanos-cluster-a-devcluster-openshift-com:6443"
Setting new cluster ID "default-cl1", previous cluster ID was "api-default-cl1-devcluster-openshift-com:6443"
Creating lighthouse clients
Creating submariner clients
Creating namespace objects with basename "latency"
Generated namespace "e2e-tests-latency-xjnld" in cluster "nmanos-cluster-a" to execute the tests in
Creating namespace "e2e-tests-latency-xjnld" in cluster "default-cl1"
Performing latency tests from Gateway pod to Gateway pod
Creating a Nettest Server Pod on "default-cl1"
Nettest Server Pod "latency-server-podsgg76" was created on node "default-cl1-r7zsb-worker-0-j4knf"
Nettest Client Pod "latency-client-podgqfxt" was created on cluster "nmanos-cluster-a", node "ip-10-0-47-83.us-west-1.compute.internal"; connect to server pod ip "10.231.0.28"
Waiting for the client pod "latency-client-podgqfxt" to exit, returning what client sent
Jul  6 09:20:29.041: INFO: Pod "latency-client-podgqfxt" output:
MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.231.0.28 (10.231.) port 0 AF_INET : first burst 0
Minimum Latency Microseconds,Mean Latency Microseconds,Maximum Latency Microseconds,Stddev Latency Microseconds,Transaction Rate Tran/s
67820,68620.04,138601,5840.33,14.473

Minimum Latency Microseconds:	67820
Mean Latency Microseconds:	68620.04
Maximum Latency Microseconds:	138601
Stddev Latency Microseconds:	5840.33
Transaction Rate Tran/s:	14.473
Performing latency tests from Non-Gateway pod to Non-Gateway pod
Creating a Nettest Server Pod on "default-cl1"
Nettest Server Pod "latency-server-podf9k76" was created on node "default-cl1-r7zsb-worker-0-l657j"
Nettest Client Pod "latency-client-pod6lbn5" was created on cluster "nmanos-cluster-a", node "ip-10-0-233-8.us-west-1.compute.internal"; connect to server pod ip "10.228.2.26"
Waiting for the client pod "latency-client-pod6lbn5" to exit, returning what client sent
Jul  6 09:20:49.482: INFO: Pod "latency-client-pod6lbn5" output:
MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.228.2.26 (10.228.) port 0 AF_INET : first burst 0
Minimum Latency Microseconds,Mean Latency Microseconds,Maximum Latency Microseconds,Stddev Latency Microseconds,Transaction Rate Tran/s
69052,70018.97,139278,5861.52,14.183

Minimum Latency Microseconds:	69052
Mean Latency Microseconds:	70018.97
Maximum Latency Microseconds:	139278
Stddev Latency Microseconds:	5861.52
Transaction Rate Tran/s:	14.183
Deleting namespace "e2e-tests-latency-xjnld" on cluster "nmanos-cluster-a"
Deleting namespace "e2e-tests-latency-xjnld" on cluster "default-cl1"

$ subctl benchmark throughput /mnt/skynet-data/nmanos-env/nmanos-cluster-a/auth/kubeconfig /mnt/skynet-data/nmanos-env/ocpup/.config/cl1/auth/kubeconfig  --verbose

Performing throughput tests
Creating kubernetes clients
Setting new cluster ID "nmanos-cluster-a", previous cluster ID was "api-nmanos-cluster-a-devcluster-openshift-com:6443"
Setting new cluster ID "default-cl1", previous cluster ID was "api-default-cl1-devcluster-openshift-com:6443"
Creating lighthouse clients
Creating submariner clients
Creating namespace objects with basename "throughput"
Generated namespace "e2e-tests-throughput-6vlsb" in cluster "nmanos-cluster-a" to execute the tests in
Creating namespace "e2e-tests-throughput-6vlsb" in cluster "default-cl1"
Performing throughput tests from Gateway pod on cluster "nmanos-cluster-a" to Gateway pod on cluster "default-cl1"
Creating a Nettest Server Pod on "default-cl1"
Nettest Server Pod "nettest-server-pod68f7w" was created on node "default-cl1-r7zsb-worker-0-j4knf"
Nettest Client Pod "nettest-client-podpbqmg" was created on cluster "nmanos-cluster-a", node "ip-10-0-47-83.us-west-1.compute.internal"; connect to server pod ip "10.231.0.29"
Waiting for the client pod "nettest-client-podpbqmg" to exit, returning what client sent
Jul  6 09:21:11.264: INFO: Pod "nettest-client-podpbqmg" output:
 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 15]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 17]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 19]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 21]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 23]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[SUM]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[  7]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[  9]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[ 11]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[ 13]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[ 15]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[ 17]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[ 19]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[ 21]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[ 23]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[SUM]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec   10             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[  7]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[  9]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 11]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 13]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 15]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 17]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 19]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 21]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 23]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[SUM]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[  5]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[  7]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[  7]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[  9]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[  9]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[ 11]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[ 11]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[ 13]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[ 13]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[ 15]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[ 15]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[ 17]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[ 17]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[ 19]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[ 19]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[ 21]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[ 21]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[ 23]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[ 23]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[SUM]   0.00-10.00  sec  3.73 MBytes  3.13 Mbits/sec   50             sender
[SUM]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver

iperf Done.

 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 15]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 17]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 19]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 21]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 23]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[SUM]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[  7]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[  9]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[ 11]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[ 13]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[ 15]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[ 17]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[ 19]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[ 21]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[ 23]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[SUM]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec   10             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[  7]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[  9]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 11]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 13]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 15]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 17]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 19]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 21]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 23]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[SUM]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[  5]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[  7]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[  7]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[  9]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[  9]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[ 11]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[ 11]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[ 13]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[ 13]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[ 15]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[ 15]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[ 17]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[ 17]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[ 19]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[ 19]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[ 21]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[ 21]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[ 23]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[ 23]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[SUM]   0.00-10.00  sec  3.73 MBytes  3.13 Mbits/sec   50             sender
[SUM]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver

iperf Done.

Performing throughput tests from Non-Gateway pod on cluster "nmanos-cluster-a" to Non-Gateway pod on cluster "default-cl1"
Creating a Nettest Server Pod on "default-cl1"
Nettest Server Pod "nettest-server-pod2ktmt" was created on node "default-cl1-r7zsb-worker-0-l657j"
Nettest Client Pod "nettest-client-pod7tjrm" was created on cluster "nmanos-cluster-a", node "ip-10-0-233-8.us-west-1.compute.internal"; connect to server pod ip "10.228.2.27"
Waiting for the client pod "nettest-client-pod7tjrm" to exit, returning what client sent
Jul  6 09:21:31.699: INFO: Pod "nettest-client-pod7tjrm" output:
 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 15]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 17]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 19]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 21]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 23]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[SUM]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[  7]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[  9]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[ 11]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[ 13]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[ 15]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[ 17]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[ 19]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[ 21]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[ 23]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[SUM]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec   10             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[  7]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[  9]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 11]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 13]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 15]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 17]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 19]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 21]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 23]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[SUM]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[  5]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[  7]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[  7]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[  9]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[  9]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[ 11]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[ 11]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[ 13]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[ 13]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[ 15]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[ 15]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[ 17]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[ 17]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[ 19]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[ 19]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[ 21]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[ 21]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[ 23]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[ 23]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[SUM]   0.00-10.00  sec  3.73 MBytes  3.13 Mbits/sec   50             sender
[SUM]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver

iperf Done.

 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 15]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 17]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 19]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 21]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 23]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[SUM]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[  7]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[  9]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[ 11]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[ 13]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[ 15]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[ 17]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[ 19]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[ 21]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[ 23]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[SUM]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec   10             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[  7]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[  9]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 11]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 13]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 15]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 17]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 19]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 21]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[ 23]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
[SUM]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[  5]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[  7]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[  7]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[  9]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[  9]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[ 11]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[ 11]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[ 13]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[ 13]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[ 15]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[ 15]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[ 17]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[ 17]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[ 19]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[ 19]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[ 21]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[ 21]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[ 23]   0.00-10.00  sec   382 KBytes   313 Kbits/sec    5             sender
[ 23]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver
[SUM]   0.00-10.00  sec  3.73 MBytes  3.13 Mbits/sec   50             sender
[SUM]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  receiver

iperf Done.


subctl version: v0.9.0

Showing information for cluster "api-nmanos-cluster-a-devcluster-openshift-com:6443":
COMPONENT                       REPOSITORY                                            VERSION         
submariner                      registry.redhat.io/rhacm2-tech-preview                v0.9.0          
submariner-operator             registry.redhat.io/rhacm2-tech-preview                v0.9.0          
service-discovery               registry.redhat.io/rhacm2-tech-preview                v0.9.0          

Showing information for cluster "nmanos-cluster-a":
COMPONENT                       REPOSITORY                                            VERSION         
submariner                      registry.redhat.io/rhacm2-tech-preview                v0.9.0          
submariner-operator             registry.redhat.io/rhacm2-tech-preview                v0.9.0          
service-discovery               registry.redhat.io/rhacm2-tech-preview                v0.9.0          

### Images of Pods (in namespace submariner-operator) ###

### lighthouse-agent-rhel8 Image ###
id=registry.redhat.io/rhacm2-tech-preview/lighthouse-agent-rhel8@sha256:1c61d96b4f0a3c6098f9bda127ab577f289b61edf7591f70335a71b664c69e6e
name=rhacm2-tech-preview/lighthouse-agent-rhel8
release=4
url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/lighthouse-agent-rhel8/images/v0.9.0-4
version=v0.9.0

### lighthouse-coredns-rhel8 Image ###
id=registry.redhat.io/rhacm2-tech-preview/lighthouse-coredns-rhel8@sha256:bcfe899f13107981859164072c13276985f971562e671b4edb624f319a8ab0c6
name=rhacm2-tech-preview/lighthouse-coredns-rhel8
release=4
url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/lighthouse-coredns-rhel8/images/v0.9.0-4
version=v0.9.0

### submariner-gateway-rhel8 Image ###
id=registry.redhat.io/rhacm2-tech-preview/submariner-gateway-rhel8@sha256:ae746991a9a8b7044ad6f19cbfcf1813bb97a0368222ebc021f5c994a0e733ed
name=rhacm2-tech-preview/submariner-gateway-rhel8
release=5
url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-gateway-rhel8/images/v0.9.0-5
version=v0.9.0

### submariner-rhel8-operator Image ###
id=registry.redhat.io/rhacm2-tech-preview/submariner-rhel8-operator@sha256:b511651ab9196f47b783e74318829d8e8cde9346c2b13022c384f4d143acbb5e
name=rhacm2-tech-preview/submariner-rhel8-operator
release=6
url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-rhel8-operator/images/v0.9.0-6
version=v0.9.0

### submariner-route-agent-rhel8 Image ###
id=registry.redhat.io/rhacm2-tech-preview/submariner-route-agent-rhel8@sha256:5284e8133764b0958def4dced827bda2566b7b7f0a5d4d04492a94638f88fa9a
name=rhacm2-tech-preview/submariner-route-agent-rhel8
release=5
url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-route-agent-rhel8/images/v0.9.0-5
version=v0.9.0

Comment 10 errata-xmlrpc 2021-08-05 13:18:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (RHEA: Submariner subctl 0.9 - bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:3017