Bug 2170505

The Minimal Steps to Reproduce Issue on RHEL8.6 with kernel 4.18.0-372.9.1.rt7.166.el8.x86_64 and firmware 3.20 is:

1.  # echo 1 > /sys/class/net/enp_xxx/device/sriov_numvfs
2.  # lspci |grep Virtual
3.  # virsh define sriov_vm1.xml 
     # virsh start vm1
     # ssh vm1
4.  Install iperf
5. Disable firewall
6. iperf -s -f g
7. Repeat Step1 - 5 to setup Host2 and VM2
8. iperf -c 192.168.xx.xx(HOST1 Vf ip) -i 1 -f g -P 4 -t 14400 (will see the low throughput numbers )
9. switch to rx by adding "-R" in iperf client, the throughput numbers are good


As mentioned this issue is not seen on RHEL 9.0 kernel 5.14.0-70.30.1.rt21.102.el9_0.x86_64 .

Created attachment 1944696 [details]
Adding TCP throughput result from INTEL using iperf

Adding TCP throughput result from INTEL using iperf

Note that we were also able to reproduce this issue using netperf in place of iperf.
We've also been able to reproduce this issue on the a newer version of the NVM, i.e. NVM 4.01.

Update the result below, much similar like iperf output


RX Command:
•        ./tcp_stream --nolog -T 16 -F 200 -B 81920 -l 30
TX Command:
•        ./tcp_stream --nolog -c -H 192.168.6.88 -T 16 -F 200 -B 81920 -l 30
 
TX Side	RX Side	TCP Throughput
Ubuntu22.04	RHEL8.6	remote_throughput=94143066754
RHEL8.6	Ubuntu22.04	remote_throughput=29425842039

Hi Tim,

thanks for the bug report, I have a couple of things I would like to ask:

1. Do you have a chance to try this with regular kernel (not real-time)? I know that we always test new releases of RHEL for various performance metrics but I think that real-time version of kernel is not one that we test.

2. I am guessing that the VM is important here. On bare metal when you try iperf host to host, can you observe this as well? Btw. just to be 100% sure, the VM is the same system as the host, right?

3. Would it be possible to know a bit more about the machine where you are running this? I know that my colleagues who are testing the performance will probably want to know this as well. Probably number of cores/sockets/cpus, probably something like 'lscpu | grep -i 'socket\|cpu' could help.

4. I think this might not be a problem with iperf since our performance team uses iperf for the testing all the time, but I would like to rule this out for sure. Can you please test the version of iperf from RHEL9 that I've prepared for RHEL8? I've put them here for you:

https://people.redhat.com/mruprich/iperf3-3.9-1.el8_6.x86_64.rpm

Thanks and regards,

Michal

Hi,

We were able to resolve this issue and reach 90+ Gbps for TX after changing to the following iperf3 version:
https://people.redhat.com/mruprich/iperf3-3.9-1.el8_6.x86_64.rpm

We can close out this issue.  Thank you all your help here.

Sincerely,
-Tim

Hi Tim,

just to be on the same page here, the package I sent you is just a test package. Don't expect any support on this version in RHEL8, only if we resolve it properly. That means narrowing down the problem and releasing an official fix.

I would like to suggest to move to RHEL9 where this version of iperf3 is already present.

In case you are fine with what I described here, we can close the bug. Otherwise we should try to work together to get this fixed officially.

Regards,
Michal

Hi,

We're currently confirming with the verification team whether they would require an official fix for RHEL 8, or if this workaround will suffice for their RHEL 8 testing.

Sincerely,
-Tim

Hi Tim,

can I ask you about the status of this bug? Can I close it?

Regards,
Michal

Hi Michal,

Please feel free to close this issue, thank you again for all your ongoing help here.

Sincerely,
-Tim

Thanks, closing.