Bug 1916670
Summary: | [RHEL8.4] VMA incurs performance degradation on UDP test cases when run over network team interface | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 8 | Reporter: | Brian Chae <bchae> |
Component: | libvma | Assignee: | Honggang LI <honli> |
Status: | CLOSED ERRATA | QA Contact: | Brian Chae <bchae> |
Severity: | unspecified | Docs Contact: | |
Priority: | unspecified | ||
Version: | 8.4 | CC: | rdma-dev-team |
Target Milestone: | rc | Keywords: | Regression, Triaged |
Target Release: | 8.4 | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | libvma-9.2.2-2.el8 | Doc Type: | If docs needed, set a value |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2021-05-18 14:46:02 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | |||
Bug Blocks: | 1903942 |
Description
Brian Chae
2021-01-15 11:42:52 UTC
The real issue is UDP performance issue. https://beaker.engineering.redhat.com/jobs/4986532 (RHEL-8.3 GA distro) https://beaker-archive.host.prod.eng.bos.redhat.com/beaker-logs/2021/01/49865/4986532/9397386/120339614/562438453/resultoutputfile.log https://beaker.engineering.redhat.com/jobs/4997634 (RHEL-8.3 GA distro with update RHEL-8.4 libvma-9.2.2 scratch build) https://beaker-archive.host.prod.eng.bos.redhat.com/beaker-logs/2021/01/49976/4997634/9412263/120454213/563145553/resultoutputfile.log [test 1] vma_test sockperf pp -i 172.31.40.119 -t 10 --msg-size=1472 -sockperf: Summary: Latency is 6.420 usec +sockperf: Summary: Latency is 163.464 usec [test 2] vma_test sockperf tp -i 172.31.40.119 -t 10 --msg-size=1472 -sockperf: Summary: Message Rate is 1987027 [msg/sec] -sockperf: Summary: BandWidth is 2789.406 MBps (22315.245 Mbps) +sockperf: Summary: Message Rate is 548598 [msg/sec] +sockperf: Summary: BandWidth is 770.127 MBps (6161.013 Mbps) [test 3] vma_test sockperf tp -i 172.31.40.119 -t 10 --msg-size=1472 --giga-size -sockperf: Summary: Message Rate is 1986003 [msg/sec] -sockperf: Summary: BandWidth is 2.788 GBps (22.304 Gbps) +sockperf: Summary: Message Rate is 544454 [msg/sec] +sockperf: Summary: BandWidth is 0.764 GBps (6.114 Gbps) [test 4] vma_test sockperf pp -i 172.31.40.119 --tcp -t 10 --msg-size=1472 -sockperf: Summary: Latency is 7.058 usec +sockperf: Summary: Latency is 7.219 usec [test 5] vma_test sockperf pp -f /tmp/feed.txt -t 10 -F e --msg-size=1472 -sockperf: Summary: Latency is 8.424 usec +sockperf: Summary: Latency is 12.042 usec [test 6] vma_test sockperf tp -f /tmp/feed.txt -t 10 -F e --msg-size=1472 -sockperf: Summary: Message Rate is 1933027 [msg/sec] -sockperf: Summary: BandWidth is 2713.600 MBps (21708.799 Mbps) +sockperf: Summary: Message Rate is 466121 [msg/sec] +sockperf: Summary: BandWidth is 654.345 MBps (5234.757 Mbps) [test 7] vma_test sockperf pp -i 172.31.43.119 -t 10 --msg-size=1472 -sockperf: Summary: Latency is 22.315 usec +sockperf: Summary: Latency is 176.737 usec [test 8] vma_test sockperf pp -i 172.31.43.119 --tcp -t 10 --msg-size=1472 -sockperf: Summary: Latency is 10.114 usec +sockperf: Summary: Latency is 7.764 usec NOTE: line start with '-' is libvma-9.0.2-1.el8.x86_64 output. line starts with '+' is libvma-9.2.2-1.scratch.el8_3.x86_64 output. (In reply to Honggang LI from comment #4) > https://github.com/Mellanox/libvma/pull/935 Upstream approve this PR. Set devel+ flag. Tested as the following: 1. versions DISTRO=RHEL-8.4.0-20210205.n.0 + [21-02-08 07:54:14] cat /etc/redhat-release Red Hat Enterprise Linux release 8.4 Beta (Ootpa) + [21-02-08 07:54:14] uname -a Linux rdma-dev-20.lab.bos.redhat.com 4.18.0-282.el8.x86_64 #1 SMP Tue Feb 2 14:09:52 EST 2021 x86_64 x86_64 x86_64 GNU/Linux + [21-02-08 07:54:14] cat /proc/cmdline BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-282.el8.x86_64 root=/dev/mapper/rhel_rdma--dev--20-root ro intel_idle.max_cstate=0 processor.max_cstate=0 intel_iommu=on iommu=on console=tty0 rd_NO_PLYMOUTH crashkernel=auto resume=/dev/mapper/rhel_rdma--dev--20-swap rd.lvm.lv=rhel_rdma-dev-20/root rd.lvm.lv=rhel_rdma-dev-20/swap console=ttyS1,115200n81 + [21-02-08 07:54:14] rpm -q rdma-core linux-firmware rdma-core-32.0-4.el8.x86_64 linux-firmware-20201218-102.git05789708.el8.noarch + [21-02-08 07:54:14] tail /sys/class/infiniband/mlx5_2/fw_ver /sys/class/infiniband/mlx5_3/fw_ver /sys/class/infiniband/mlx5_bond_0/fw_ver ==> /sys/class/infiniband/mlx5_2/fw_ver <== 12.23.1020 ==> /sys/class/infiniband/mlx5_3/fw_ver <== 12.23.1020 ==> /sys/class/infiniband/mlx5_bond_0/fw_ver <== 14.25.1020 + [21-02-08 07:54:14] lspci + [21-02-08 07:54:14] grep -i -e ethernet -e infiniband -e omni -e ConnectX 01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 04:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] 04:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] 82:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4] 82:00.1 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4] libvma-9.2.2-2.el8.x86_64 + [21-02-08 07:54:14] vma_setup RDMA hosts: Clients: rdma-dev-20 31: mlx5_team_roce: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000 link/ether 7c:fe:90:cb:76:2a brd ff:ff:ff:ff:ff:ff inet 172.31.40.120/24 brd 172.31.40.255 scope global dynamic noprefixroute mlx5_team_roce valid_lft 3391sec preferred_lft 3391sec inet6 fe80::7efe:90ff:fecb:762a/64 scope link noprefixroute valid_lft forever preferred_lft forever 32: mlx5_team_ro.43@mlx5_team_roce: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000 link/ether 7c:fe:90:cb:76:2a brd ff:ff:ff:ff:ff:ff inet 172.31.43.120/24 brd 172.31.43.255 scope global dynamic noprefixroute mlx5_team_ro.43 valid_lft 3455sec preferred_lft 3455sec inet6 fe80::7efe:90ff:fecb:762a/64 scope link noprefixroute valid_lft forever preferred_lft forever 33: mlx5_team_ro.45@mlx5_team_roce: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000 link/ether 7c:fe:90:cb:76:2a brd ff:ff:ff:ff:ff:ff inet 172.31.45.120/24 brd 172.31.45.255 scope global dynamic noprefixroute mlx5_team_ro.45 valid_lft 3459sec preferred_lft 3459sec inet6 fe80::7efe:90ff:fecb:762a/64 scope link noprefixroute valid_lft forever preferred_lft forever Servers: rdma-dev-19 25: mlx5_bond_roce: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000 link/ether 7c:fe:90:cb:74:3a brd ff:ff:ff:ff:ff:ff inet 172.31.40.119/24 brd 172.31.40.255 scope global dynamic noprefixroute mlx5_bond_roce valid_lft 3259sec preferred_lft 3259sec inet6 fe80::7efe:90ff:fecb:743a/64 scope link noprefixroute valid_lft forever preferred_lft forever 26: mlx5_bond_ro.45@mlx5_bond_roce: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000 link/ether 7c:fe:90:cb:74:3a brd ff:ff:ff:ff:ff:ff inet 172.31.45.119/24 brd 172.31.45.255 scope global dynamic noprefixroute mlx5_bond_ro.45 valid_lft 3346sec preferred_lft 3346sec inet6 fe80::7efe:90ff:fecb:743a/64 scope link noprefixroute valid_lft forever preferred_lft forever 27: mlx5_bond_ro.43@mlx5_bond_roce: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000 link/ether 7c:fe:90:cb:74:3a brd ff:ff:ff:ff:ff:ff inet 172.31.43.119/24 brd 172.31.43.255 scope global dynamic noprefixroute mlx5_bond_ro.43 valid_lft 3342sec preferred_lft 3342sec inet6 fe80::7efe:90ff:fecb:743a/64 scope link noprefixroute valid_lft forever preferred_lft forever 2. Results: MLX5 ROCE: Test results for vma on rdma-dev-20: 4.18.0-282.el8.x86_64, rdma-core-32.0-4.el8, mlx5, roce.45, & mlx5_bond_0 Result | Status | Test ---------+--------+------------------------------------ PASS | 0 | sockperf pingpong multicast PASS | 0 | sockperf throughput multicast PASS | 0 | sockperf throughput unicast PASS | 0 | sockperf pingpong unicast PASS | 0 | sockperf (100 sockets) pingpong multicast PASS | 0 | sockperf (100 sockets) pingpong unicast PASS | 0 | sockperf pingpong multicast pkey/vlan PASS | 0 | sockperf pingpong unicast pkey/vlan Checking for failures and known issues: no test failures Performance INFO: + [21-02-08 07:59:32] vma_test sockperf pp -i 172.31.40.119 -t 10 --msg-size=1472 sockperf: Summary: Latency is 3.656 usec + [21-02-08 07:59:58] vma_test sockperf tp -i 172.31.40.119 -t 10 --msg-size=1472 sockperf: Summary: Message Rate is 1988059 [msg/sec] sockperf: Summary: BandWidth is 2790.854 MBps (22326.834 Mbps) + [21-02-08 08:00:24] vma_test sockperf tp -i 172.31.40.119 -t 10 --msg-size=1472 --giga-size sockperf: Summary: Message Rate is 1988267 [msg/sec] sockperf: Summary: BandWidth is 2.791 GBps (22.329 Gbps) + [21-02-08 08:00:50] vma_test sockperf pp -i 172.31.40.119 --tcp -t 10 --msg-size=1472 sockperf: Summary: Latency is 3.736 usec + [21-02-08 08:01:15] vma_test sockperf pp -f /tmp/feed.txt -t 10 -F e --msg-size=1472 sockperf: Summary: Latency is 4.310 usec + [21-02-08 08:01:42] vma_test sockperf tp -f /tmp/feed.txt -t 10 -F e --msg-size=1472 sockperf: Summary: Message Rate is 1935439 [msg/sec] sockperf: Summary: BandWidth is 2716.986 MBps (21735.887 Mbps) + [21-02-08 08:02:08] vma_test sockperf pp -i 172.31.43.119 -t 10 --msg-size=1472 sockperf: Summary: Latency is 22.688 usec + [21-02-08 08:02:34] vma_test sockperf pp -i 172.31.43.119 --tcp -t 10 --msg-size=1472 sockperf: Summary: Latency is 7.206 usec -------------------------------------- MLX5 IB0 Test results for vma on rdma-dev-20: 4.18.0-282.el8.x86_64, rdma-core-32.0-4.el8, mlx5, ib0, & mlx5_2 Result | Status | Test ---------+--------+------------------------------------ PASS | 0 | sockperf pingpong multicast PASS | 0 | sockperf throughput multicast PASS | 0 | sockperf throughput unicast PASS | 0 | sockperf pingpong unicast PASS | 0 | sockperf (100 sockets) pingpong multicast PASS | 0 | sockperf (100 sockets) pingpong unicast PASS | 0 | sockperf pingpong multicast pkey/vlan PASS | 0 | sockperf pingpong unicast pkey/vlan Checking for failures and known issues: no test failures Performance INFO: + [21-02-08 08:34:05] vma_test sockperf pp -i 172.31.0.119 -t 10 --msg-size=1472 sockperf: Summary: Latency is 2.890 usec + [21-02-08 08:34:31] vma_test sockperf tp -i 172.31.0.119 -t 10 --msg-size=1472 sockperf: Summary: Message Rate is 3053068 [msg/sec] sockperf: Summary: BandWidth is 4285.923 MBps (34287.385 Mbps) + [21-02-08 08:34:57] vma_test sockperf tp -i 172.31.0.119 -t 10 --msg-size=1472 --giga-size sockperf: Summary: Message Rate is 3513380 [msg/sec] sockperf: Summary: BandWidth is 4.932 GBps (39.457 Gbps) + [21-02-08 08:35:23] vma_test sockperf pp -i 172.31.0.119 --tcp -t 10 --msg-size=1472 sockperf: Summary: Latency is 2.989 usec + [21-02-08 08:35:49] vma_test sockperf pp -f /tmp/feed.txt -t 10 -F e --msg-size=1472 sockperf: Summary: Latency is 3.225 usec + [21-02-08 08:36:15] vma_test sockperf tp -f /tmp/feed.txt -t 10 -F e --msg-size=1472 sockperf: Summary: Message Rate is 2422217 [msg/sec] sockperf: Summary: BandWidth is 3400.329 MBps (27202.632 Mbps) + [21-02-08 08:36:42] vma_test sockperf pp -i 172.31.2.119 -t 10 --msg-size=1472 sockperf: Summary: Latency is 2.871 usec + [21-02-08 08:37:08] vma_test sockperf pp -i 172.31.2.119 --tcp -t 10 --msg-size=1472 sockperf: Summary: Latency is 2.818 usec o The performance numbers are on par with that of RHEL8.3 VMA test results. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (RDMA stack bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:1594 |