Bug 1856292

Summary: [RHEL8.3] udpong does not run successfully on MLX4 ROCE, as well as MLX5 ROCE
Product: Red Hat Enterprise Linux 8 Reporter: Brian Chae <bchae>
Component: rdma-coreAssignee: Honggang LI <honli>
Status: CLOSED NOTABUG QA Contact: Infiniband QE <infiniband-qe>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 8.3CC: ahleihel, hwkernel-mgr, rdma-dev-team
Target Milestone: rc   
Target Release: 8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-08-18 13:57:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Brian Chae 2020-07-13 10:34:39 UTC
Description of problem:

This is the same issue as reported on Bug 1635810. 

===============================

+ [20-07-01 23:01:48] timeout 5m udpong -s 172.31.45.120 -S 1024 -C 10240000
rconnect: Protocol not supported
name      bytes   xfers   total       time     Gb/sec    usec/xfer
+ [20-07-01 23:01:48] RQA_check_result -r 255 -t udpong

=================================

Also, there is a way to run this same test with "-T socket" option. However, with it, the test succeeds on the client but the server side always times out.


Version-Release number of selected component (if applicable):



DISTRO=RHEL-8.3.0-20200701.2
+ [20-07-01 23:00:39] cat /etc/redhat-release
Red Hat Enterprise Linux release 8.3 Beta (Ootpa)
+ [20-07-01 23:00:39] uname -a
Linux rdma-dev-19.lab.bos.redhat.com 4.18.0-221.el8.x86_64 #1 SMP Thu Jun 25 20:58:19 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
+ [20-07-01 23:00:39] cat /proc/cmdline
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-221.el8.x86_64 root=UUID=52c81e7d-0ede-4f05-bfcb-9512396cae2b ro console=tty0 rd_NO_PLYMOUTH intel_idle.max_cstate=0 intel_iommu=on iommu=on processor.max_cstate=0 crashkernel=auto resume=UUID=14886570-838d-4569-920f-69c59f7fb3ea console=ttyS1,115200
+ [20-07-01 23:00:39] rpm -q rdma-core linux-firmware
rdma-core-29.0-3.el8.x86_64
linux-firmware-20200619-99.git3890db36.el8.noarch
+ [20-07-01 23:00:39] tail /sys/class/infiniband/mlx5_2/fw_ver /sys/class/infiniband/mlx5_3/fw_ver /sys/class/infiniband/mlx5_bond_0/fw_ver
==> /sys/class/infiniband/mlx5_2/fw_ver <==
12.23.1020

==> /sys/class/infiniband/mlx5_3/fw_ver <==
12.23.1020

==> /sys/class/infiniband/mlx5_bond_0/fw_ver <==
14.23.1020
+ [20-07-01 23:00:39] lspci
+ [20-07-01 23:00:39] grep -i -e ethernet -e infiniband -e omni -e ConnectX
01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
04:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
04:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
82:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
82:00.1 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]




How reproducible:

100% on both MLX4 ROCE and MLX5 ROCE


Steps to Reproduce:
1. server: udpong -b <server_ip>
2. client: udpong -s <server_ip> -S 1024 -C 10240000
3.

Actual results:

Server
=======
+ [20-07-01 23:01:45] timeout 5m udpong -b 172.31.45.120
Terminated


Client
======

+ [20-07-01 23:01:48] timeout 5m udpong -s 172.31.45.120 -S 1024 -C 10240000
rconnect: Protocol not supported
name      bytes   xfers   total       time     Gb/sec    usec/xfer
+ [20-07-01 23:01:48] RQA_check_result -r 255 -t udpong

Expected results:

passes similar to below:

+ [20-07-01 16:47:01] timeout 5m udpong -s 172.31.0.119 -S 1024 -C 10240000
name      bytes   xfers   total       time     Gb/sec    usec/xfer
custom    1k      10m     9.7g        3.10s     27.03       0.30
+ [20-07-01 16:47:04] RQA_check_result -r 0 -t udpong


Additional info:



with "-T socket" option, updong works on the client side; but the server side hangs...


o Tested on rdma-dev-19/20 ROCE

[root@rdma-dev-19 ~]$ timeout 5m udpong -b 172.31.45.119 -T socket
client login from 172.31.45.120
[root@rdma-dev-19 ~]$ echo $?
124


[root@rdma-dev-20 ~]$ timeout 5m udpong -s 172.31.45.119 -S 1024 -C 10240000 -T socket
name      bytes   xfers   total       time     Gb/sec    usec/xfer
custom    1k      10m     9.7g       16.56s      5.07       1.62
[root@rdma-dev-20 ~]$

The server side hangs...

Comment 1 Honggang LI 2020-07-13 10:47:49 UTC
Mellanox hardware specific issue add Alaa into the CC'ed list.

Comment 2 Alaa Hleihel (NVIDIA Mellanox) 2020-07-14 14:32:36 UTC
Hi,

Do you know if this ever worked on either of mlx4 or mlx5?
does it work on other vendors?

Regards,
Alaa

Comment 3 Alaa Hleihel (NVIDIA Mellanox) 2020-07-16 11:38:25 UTC
> + [20-07-01 23:01:48] timeout 5m udpong -s 172.31.45.120 -S 1024 -C 10240000
> rconnect: Protocol not supporte

By default the test will run with rsocket and it seems to work only on an InfiniBand link layer.
When using option "-T s" it will start using standard tcp/ip sockets, and then it can work on Ethernet link layer as well.
# udpong  -h
...
        [-T test_option]
            s|sockets - use standard tcp/ip sockets
            a|async - asynchronous operation (use poll)
            b|blocking - use blocking calls
            n|nonblocking - use nonblocking calls
            e|echo - server echoes all messages


Regarding the second issue:

> 
> [root@rdma-dev-20 ~]$ timeout 5m udpong -s 172.31.45.119 -S 1024 -C 10240000 -T socket
> name      bytes   xfers   total       time     Gb/sec    usec/xfer
> custom    1k      10m     9.7g       16.56s      5.07       1.62
> [root@rdma-dev-20 ~]$
>
> The server side hangs...

Looking at the code it seems that the test is designed this way, the server remains up and waits for other clients to connect.
The same thing happens also when running over IB link.

Comment 4 Brian Chae 2020-07-31 15:47:31 UTC
(In reply to Alaa Hleihel (Mellanox) from comment #2)
> Hi,
> 
> Do you know if this ever worked on either of mlx4 or mlx5?
> does it work on other vendors?
> 
> Regards,
> Alaa

Going back to RHEL7, we had this issue... But, closed. See bz 1635810. So, it never worked from RHEL7 to 8.3.
We do see the same issue on both MLX4 and MLX5 ROCE devices.

Brian

Comment 5 Alaa Hleihel (NVIDIA Mellanox) 2020-08-03 07:29:00 UTC
(In reply to Brian Chae from comment #4)
> (In reply to Alaa Hleihel (Mellanox) from comment #2)
> > Hi,
> > 
> > Do you know if this ever worked on either of mlx4 or mlx5?
> > does it work on other vendors?
> > 
> > Regards,
> > Alaa
> 
> Going back to RHEL7, we had this issue... But, closed. See bz 1635810. So,
> it never worked from RHEL7 to 8.3.

I cannot access BZ 1635810, can you CC me on it?

> We do see the same issue on both MLX4 and MLX5 ROCE devices.

I meant, does it work on other vendors (other than Mellanox mlx5/mlx4)?

Comment 6 Honggang LI 2020-08-03 07:40:06 UTC
(In reply to Alaa Hleihel (Mellanox) from comment #5)
> I cannot access BZ 1635810, can you CC me on it?
Done.

Comment 7 Alaa Hleihel (NVIDIA Mellanox) 2020-08-03 12:47:51 UTC
Thanks, Honggang!

Based on the comments in that BZ, especially https://bugzilla.redhat.com/show_bug.cgi?id=1635810#c8 , 
I see that this is just not supported in rscoket, and it's not an mlx5/4 specific issue.

I think that this BZ can be closed as well.

Regards,
Alaa

Comment 8 Brian Chae 2020-08-18 11:27:13 UTC
(In reply to Alaa Hleihel (Mellanox) from comment #5)
> (In reply to Brian Chae from comment #4)
> > (In reply to Alaa Hleihel (Mellanox) from comment #2)
> > > Hi,
> > > 
> > > Do you know if this ever worked on either of mlx4 or mlx5?
> > > does it work on other vendors?
> > > 
> > > Regards,
> > > Alaa
> > 
> > Going back to RHEL7, we had this issue... But, closed. See bz 1635810. So,
> > it never worked from RHEL7 to 8.3.
> 
> I cannot access BZ 1635810, can you CC me on it?
> 
> > We do see the same issue on both MLX4 and MLX5 ROCE devices.
> 
> I meant, does it work on other vendors (other than Mellanox mlx5/mlx4)?

Alaa, the same issue is observed on BXNT ROCE device, as well.

Comment 9 Brian Chae 2020-08-18 11:31:20 UTC
(In reply to Brian Chae from comment #8)
> (In reply to Alaa Hleihel (Mellanox) from comment #5)
> > (In reply to Brian Chae from comment #4)
> > > (In reply to Alaa Hleihel (Mellanox) from comment #2)
> > > > Hi,
> > > > 
> > > > Do you know if this ever worked on either of mlx4 or mlx5?
> > > > does it work on other vendors?
> > > > 
> > > > Regards,
> > > > Alaa
> > > 
> > > Going back to RHEL7, we had this issue... But, closed. See bz 1635810. So,
> > > it never worked from RHEL7 to 8.3.
> > 
> > I cannot access BZ 1635810, can you CC me on it?
> > 
> > > We do see the same issue on both MLX4 and MLX5 ROCE devices.
> > 
> > I meant, does it work on other vendors (other than Mellanox mlx5/mlx4)?
> 
> Alaa, the same issue is observed on BXNT ROCE device, as well.

However, "udpong" succeeds on HFI1 OPA0

Comment 10 Honggang LI 2020-08-18 12:58:45 UTC
(In reply to Brian Chae from comment #9)

> However, "udpong" succeeds on HFI1 OPA0

That's OK. OPA is InfiniBand hardware, not ROCE (Ethernet) hardware.

Comment 11 Brian Chae 2020-08-18 13:57:49 UTC
Afer discussing with Honggang, we decided to close this bug.

Comment 12 Alaa Hleihel (NVIDIA Mellanox) 2020-08-19 07:30:22 UTC
ack, thanks