Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2126615

Summary: [Azure][RHEL-9][CVM][Network Performance] Low TCP throughput between 2 Standard_DC96as_v5 CVMs
Product: Red Hat Enterprise Linux 9 Reporter: Li Tian <litian>
Component: kernelAssignee: Virtualization Maintenance <virt-maint>
kernel sub component: Hyper-V QA Contact: Li Tian <litian>
Status: CLOSED MIGRATED Docs Contact:
Severity: unspecified    
Priority: unspecified CC: decui, eterrell, josete, xuli, yacao, yuxisun
Version: 9.1Keywords: MigratedToJIRA, Performance
Target Milestone: rcFlags: decui: needinfo-
pm-rhel: mirror+
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-09-22 14:26:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Li Tian 2022-09-14 06:39:51 UTC
Description of problem:
TCP throughput between 2 Standard_DC96as_v5 CVMs is lower than advertised. This is reproduced at both Red Hat and Microsoft sides. Below is a test sample from NTTTCP.

Connections=1 : throughput=2.99Gbps cyclesPerByte_sender=1.65 cyclesPerByte_receiver=2.89 Avg_TCP_lat=479.864 pktsPerInterrupt=1.89 conCreatedTime=633 retransSegs=1214
      Connections=2 : throughput=6.10Gbps cyclesPerByte_sender=1.78 cyclesPerByte_receiver=2.89 Avg_TCP_lat=229.669 pktsPerInterrupt=1.88 conCreatedTime=675 retransSegs=1992
      Connections=4 : throughput=11.15Gbps cyclesPerByte_sender=1.48 cyclesPerByte_receiver=2.98 Avg_TCP_lat=6963.867 pktsPerInterrupt=2.42 conCreatedTime=1261 retransSegs=3889
      Connections=8 : throughput=15.04Gbps cyclesPerByte_sender=1.60 cyclesPerByte_receiver=2.74 Avg_TCP_lat=7720.227 pktsPerInterrupt=3.08 conCreatedTime=1036 retransSegs=9885
      Connections=16 : throughput=16.80Gbps cyclesPerByte_sender=1.81 cyclesPerByte_receiver=3.03 Avg_TCP_lat=7712.378 pktsPerInterrupt=2.43 conCreatedTime=1811 retransSegs=18803
      Connections=32 : throughput=16.20Gbps cyclesPerByte_sender=2.10 cyclesPerByte_receiver=3.70 Avg_TCP_lat=8525.040 pktsPerInterrupt=2.00 conCreatedTime=3122 retransSegs=57473
      Connections=64 : throughput=16.05Gbps cyclesPerByte_sender=2.93 cyclesPerByte_receiver=4.61 Avg_TCP_lat=8990.453 pktsPerInterrupt=2.33 conCreatedTime=14756 retransSegs=211787
      Connections=128 : throughput=15.46Gbps cyclesPerByte_sender=4.09 cyclesPerByte_receiver=6.06 Avg_TCP_lat=9951.902 pktsPerInterrupt=3.34 conCreatedTime=30995 retransSegs=558881
      Connections=256 : throughput=14.36Gbps cyclesPerByte_sender=5.69 cyclesPerByte_receiver=9.44 Avg_TCP_lat=11873.313 pktsPerInterrupt=5.48 conCreatedTime=51052 retransSegs=1118302
      Connections=512 : throughput=12.84Gbps cyclesPerByte_sender=6.92 cyclesPerByte_receiver=11.98 Avg_TCP_lat=17647.435 pktsPerInterrupt=7.60 conCreatedTime=58597 retransSegs=11032245
      Connections=1024 : throughput=12.91Gbps cyclesPerByte_sender=12.05 cyclesPerByte_receiver=12.39 Avg_TCP_lat=13498.971 pktsPerInterrupt=44.58 conCreatedTime=99941 retransSegs=2622917

Also notice that of Standard_DC64as_v5 is below advertised too:
Connections=1 : throughput=2.96Gbps cyclesPerByte_sender=1.82 cyclesPerByte_receiver=2.72 Avg_TCP_lat=257.609 pktsPerInterrupt=1.68 conCreatedTime=1188 retransSegs=1275
      Connections=2 : throughput=5.64Gbps cyclesPerByte_sender=1.52 cyclesPerByte_receiver=2.90 Avg_TCP_lat=262.646 pktsPerInterrupt=2.11 conCreatedTime=1240 retransSegs=2265
      Connections=4 : throughput=7.95Gbps cyclesPerByte_sender=1.26 cyclesPerByte_receiver=2.33 Avg_TCP_lat=7391.646 pktsPerInterrupt=2.28 conCreatedTime=1432 retransSegs=4785
      Connections=8 : throughput=12.11Gbps cyclesPerByte_sender=1.50 cyclesPerByte_receiver=2.89 Avg_TCP_lat=597.900 pktsPerInterrupt=2.21 conCreatedTime=1441 retransSegs=17272
      Connections=16 : throughput=12.23Gbps cyclesPerByte_sender=1.57 cyclesPerByte_receiver=3.45 Avg_TCP_lat=9136.253 pktsPerInterrupt=1.97 conCreatedTime=1957 retransSegs=30081
      Connections=32 : throughput=15.70Gbps cyclesPerByte_sender=1.85 cyclesPerByte_receiver=3.75 Avg_TCP_lat=9050.949 pktsPerInterrupt=2.01 conCreatedTime=3195 retransSegs=64960
      Connections=64 : throughput=15.53Gbps cyclesPerByte_sender=2.35 cyclesPerByte_receiver=4.61 Avg_TCP_lat=9142.252 pktsPerInterrupt=2.36 conCreatedTime=11085 retransSegs=202511
      Connections=128 : throughput=14.88Gbps cyclesPerByte_sender=3.24 cyclesPerByte_receiver=6.16 Avg_TCP_lat=10435.140 pktsPerInterrupt=3.27 conCreatedTime=25293 retransSegs=548462
      Connections=256 : throughput=13.64Gbps cyclesPerByte_sender=4.50 cyclesPerByte_receiver=9.63 Avg_TCP_lat=13677.645 pktsPerInterrupt=5.14 conCreatedTime=53623 retransSegs=1105525
      Connections=512 : throughput=12.05Gbps cyclesPerByte_sender=5.30 cyclesPerByte_receiver=11.59 Avg_TCP_lat=17504.668 pktsPerInterrupt=6.28 conCreatedTime=72665 retransSegs=13815629
      Connections=1024 : throughput=10.09Gbps cyclesPerByte_sender=7.65 cyclesPerByte_receiver=18.13 Avg_TCP_lat=29683.466 pktsPerInterrupt=8.21 conCreatedTime=98030 retransSegs=16183693

Version-Release number of selected component (if applicable):
5.14.0-160.el9.x86_64

How reproducible:
100%

Steps to Reproduce:
Create 2 Standard_DC96as_v5 CVMs on same subnet and use iperf3 or NTTTCP to test throughput between them.

Actual results:


Expected results:
Standard_DC64as_v5   20000 Mbps
Standard_DC96as_v5   30000 Mbps

Additional info:
Original thread - https://bugzilla.redhat.com/show_bug.cgi?id=2124751

Comment 1 Li Tian 2023-06-15 08:32:27 UTC
Hello Dexuan,

Just a reminder on this item. We are still having around 16Gbps throughput on DC96as_v5. Thanks.

Comment 2 Dexuan Cui 2023-06-19 23:08:16 UTC
Hi Li, thanks for the reminder! We'll take a look.

Comment 3 RHEL Program Management 2023-09-22 14:23:41 UTC
Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

Comment 4 RHEL Program Management 2023-09-22 14:26:24 UTC
This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.