Bug 2126080
| Summary: | [RHEL9.1] UCX fails in many tests when tested on MLX5 ROCE / IB devices | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | Brian Chae <bchae> |
| Component: | ucx | Assignee: | Michal Schmidt <mschmidt> |
| Status: | CLOSED MIGRATED | QA Contact: | Afom T. Michael <tmichael> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 9.1 | CC: | rdma-dev-team, zguo |
| Target Milestone: | rc | Keywords: | MigratedToJIRA, Regression |
| Target Release: | --- | Flags: | pm-rhel:
mirror+
|
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-09-21 14:40:33 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 2091421 | ||
The same issue was observed in MLX5 IB devices, as well, when testing RHEL-9.2 CTC#2 for tier2 RDMA.
regression suite over rdma-ib-mlx5 pool[ RHEL-9.2.0-20230115.7: rdma-perf-02/03 ]
ucx/ucx/ test results on rdma-perf-02/rdma-perf-03 & Beaker job J:7437662:
5.14.0-234.el9.x86_64, rdma-core-41.0-3.el9, mlx5, ib0, ConnectX-5 & mlx5_0
ucx
FAIL | 254 | ucp worker info for a
FAIL | 254 | ucp worker info for r
FAIL | 254 | ucp worker info for t
FAIL | 254 | ucp worker info for m
FAIL | 254 | ucp worker info for ae
FAIL | 254 | ucp worker info for re
FAIL | 254 | ucp worker info for te
FAIL | 254 | ucp worker info for me
FAIL | 254 | ucp worker info for aw
FAIL | 254 | ucp worker info for rw
FAIL | 254 | ucp worker info for tw
FAIL | 254 | ucp worker info for mw
FAIL | 255 | ucx_perftest tag_lat
FAIL | 255 | ucx_perftest tag_bw
FAIL | 255 | ucx_perftest ucp_put_lat
FAIL | 255 | ucx_perftest ucp_put_bw
FAIL | 255 | ucx_perftest ucp_get
FAIL | 135 | openmpi ucx osu_bw
Sometimes, above results are gone and most of them pass. Sometimes a few ucx_perftests fails with the following tracebacks. + [23-06-04 12:58:47] timeout --preserve-status --kill-after=5m 3m ucx_perftest -t tag_lat -c 1 172.31.0.182 +--------------+--------------+------------------------------+---------------------+-----------------------+ | | | latency (usec) | bandwidth (MB/s) | message rate (msg/s) | +--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+ | Stage | # iterations | 50.0%ile | average | overall | average | overall | average | overall | +--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+ [rdma-perf-03:49562:0:49562] ib_mlx5_log.c:171 Local QP operation on mlx5_0:1/IB (synd 0x2 vend 0x68 hw_synd 0/31) [rdma-perf-03:49562:0:49562] ib_mlx5_log.c:171 RC QP 0x143 wqe[19666]: SEND --e [inl len 18] [rqpn 0x143 dlid=6 sl=0 port=1 src_path_bits=0] ==== backtrace (tid: 49562) ==== 0 /lib64/libucs.so.0(ucs_handle_error+0x2e4) [0x7fba2fe45314] 1 /lib64/libucs.so.0(ucs_fatal_error_message+0xbc) [0x7fba2fe454cc] 2 /lib64/libucs.so.0(ucs_log_default_handler+0x839) [0x7fba2fe467e9] 3 /lib64/libucs.so.0(ucs_log_dispatch+0xe5) [0x7fba2fe41b55] 4 /lib64/ucx/libuct_ib.so.0(uct_ib_mlx5_completion_with_err+0x5e3) [0x7fba2f9a4ee3] 5 /lib64/ucx/libuct_ib.so.0(+0x3b3a4) [0x7fba2f9ba3a4] 6 /lib64/ucx/libuct_ib.so.0(uct_ib_mlx5_check_completion+0x35) [0x7fba2f9a5285] 7 /lib64/ucx/libuct_ib.so.0(+0x39672) [0x7fba2f9b8672] 8 /lib64/libucp.so.0(ucp_worker_progress+0x2a) [0x7fba2feec48a] 9 ucx_perftest(+0x2f5bc) [0x557c2c64d5bc] 10 ucx_perftest(+0x21772) [0x557c2c63f772] 11 ucx_perftest(+0x8c4a) [0x557c2c626c4a] 12 ucx_perftest(+0x5aa4) [0x557c2c623aa4] 13 /lib64/libc.so.6(+0x3feb0) [0x7fba2fa3feb0] 14 /lib64/libc.so.6(__libc_start_main+0x80) [0x7fba2fa3ff60] 15 ucx_perftest(+0x63b5) [0x557c2c6243b5] ================================= timeout: the monitored command dumped core ./runtest.sh: line 170: 49561 Aborted $TMOUT ucx_perftest -t $test $specific_args -c 1 $SERVER_IPV4 + [23-06-04 12:58:50] RQA_check_result -r 134 -t 'ucx_perftest tag_lat' + [23-06-04 13:02:25] timeout --preserve-status --kill-after=5m 3m ucx_perftest -t tag_bw -c 1 172.31.0.182 +--------------+--------------+------------------------------+---------------------+-----------------------+ | | | overhead (usec) | bandwidth (MB/s) | message rate (msg/s) | +--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+ | Stage | # iterations | 50.0%ile | average | overall | average | overall | average | overall | +--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+ [rdma-perf-03:49662:0:49662] ib_mlx5_log.c:171 Local QP operation on mlx5_0:1/IB (synd 0x2 vend 0x68 hw_synd 0/31) [rdma-perf-03:49662:0:49662] ib_mlx5_log.c:171 RC QP 0x14e wqe[17768]: SEND --e [inl len 18] [rqpn 0x14e dlid=6 sl=0 port=1 src_path_bits=0] ==== backtrace (tid: 49662) ==== 0 /lib64/libucs.so.0(ucs_handle_error+0x2e4) [0x7fcac906f314] 1 /lib64/libucs.so.0(ucs_fatal_error_message+0xbc) [0x7fcac906f4cc] 2 /lib64/libucs.so.0(ucs_log_default_handler+0x839) [0x7fcac90707e9] 3 /lib64/libucs.so.0(ucs_log_dispatch+0xe5) [0x7fcac906bb55] 4 /lib64/ucx/libuct_ib.so.0(uct_ib_mlx5_completion_with_err+0x5e3) [0x7fcac8ba4ee3] 5 /lib64/ucx/libuct_ib.so.0(+0x3b3a4) [0x7fcac8bba3a4] 6 /lib64/ucx/libuct_ib.so.0(uct_ib_mlx5_check_completion+0x35) [0x7fcac8ba5285] 7 /lib64/ucx/libuct_ib.so.0(+0x39672) [0x7fcac8bb8672] 8 /lib64/libucp.so.0(ucp_worker_progress+0x2a) [0x7fcac911648a] 9 ucx_perftest(+0x322c9) [0x564dccf002c9] 10 ucx_perftest(+0x20ed2) [0x564dcceeeed2] 11 ucx_perftest(+0x8c4a) [0x564dcced6c4a] 12 ucx_perftest(+0x5aa4) [0x564dcced3aa4] 13 /lib64/libc.so.6(+0x3feb0) [0x7fcac8c3feb0] 14 /lib64/libc.so.6(__libc_start_main+0x80) [0x7fcac8c3ff60] 15 ucx_perftest(+0x63b5) [0x564dcced43b5] ================================= timeout: the monitored command dumped core ./runtest.sh: line 170: 49661 Aborted $TMOUT ucx_perftest -t $test $specific_args -c 1 $SERVER_IPV4 + [23-06-04 13:02:27] RQA_check_result -r 134 -t 'ucx_perftest tag_bw' ================================= beaker job ID: https://beaker.engineering.redhat.com/jobs/7929031 regression suite over rdma-ib-mlx5 pool[ RHEL-9.3.0-20230531.0 - rdma-perf-02/03 : UCX,opensm-functional,fabtests ] Refer to the beaker result log : https://beaker-archive.hosts.prod.psi.bos.redhat.com/beaker-logs/2023/06/79290/7929031/14026222/161241148/754198875/resultoutputfile.log Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug. This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there. Due to differences in account names between systems, some fields were not replicated. Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information. To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer. You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like: "Bugzilla Bug" = 1234567 In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information. |
Description of problem: When tested on all variants of MLX5 ROCE HCAs, the following UCX tests failed: FAIL | 254 | ucp worker info for a FAIL | 254 | ucp worker info for r FAIL | 254 | ucp worker info for t FAIL | 254 | ucp worker info for m FAIL | 254 | ucp worker info for ae FAIL | 254 | ucp worker info for re FAIL | 254 | ucp worker info for te FAIL | 254 | ucp worker info for me FAIL | 254 | ucp worker info for aw FAIL | 254 | ucp worker info for rw FAIL | 254 | ucp worker info for tw FAIL | 254 | ucp worker info for mw FAIL | 255 | ucx_perftest tag_lat FAIL | 255 | ucx_perftest tag_bw FAIL | 255 | ucx_perftest ucp_put_lat FAIL | 255 | ucx_perftest ucp_put_bw FAIL | 255 | ucx_perftest ucp_get FAIL | 135 | openmpi ucx osu_bw This is a regression issue when comparedd with RHEL-9.1.0-20220524.0, we well as build for CTC#2 testing cycle ( however, this build was not availabe during Beta compose testing cycle ) Version-Release number of selected component (if applicable): Clients: rdma-dev-22 Servers: rdma-dev-21 DISTRO=RHEL-9.1.0-20220910.0 + [22-09-11 20:02:57] cat /etc/redhat-release Red Hat Enterprise Linux release 9.1 Beta (Plow) + [22-09-11 20:02:57] uname -a Linux rdma-dev-22.rdma.lab.eng.rdu2.redhat.com 5.14.0-162.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Sep 5 10:44:43 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux + [22-09-11 20:02:57] cat /proc/cmdline BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-162.el9.x86_64 root=UUID=376371e8-0b44-45c2-8687-191dbb3737bc ro intel_idle.max_cstate=0 processor.max_cstate=0 intel_iommu=on iommu=on console=tty0 rd_NO_PLYMOUTH crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=UUID=beb6c243-17c9-4210-ba33-d2c0b4062b8a console=ttyS1,115200n81 + [22-09-11 20:02:57] rpm -q rdma-core linux-firmware rdma-core-41.0-3.el9.x86_64 linux-firmware-20220708-127.el9.noarch + [22-09-11 20:02:57] tail /sys/class/infiniband/mlx5_0/fw_ver /sys/class/infiniband/mlx5_1/fw_ver /sys/class/infiniband/mlx5_2/fw_ver ==> /sys/class/infiniband/mlx5_0/fw_ver <== 12.28.2006 ==> /sys/class/infiniband/mlx5_1/fw_ver <== 12.28.2006 ==> /sys/class/infiniband/mlx5_2/fw_ver <== 12.28.2006 + [22-09-11 20:02:57] lspci + [22-09-11 20:02:57] grep -i -e ethernet -e infiniband -e omni -e ConnectX 01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe 01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe 02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe 02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe 04:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4] 82:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4] 82:00.1 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4] Installed: ucx-cma-1.13.0-1.el9.x86_64 ucx-ib-1.13.0-1.el9.x86_64 ucx-rdmacm-1.13.0-1.el9.x86_64 How reproducible: 100% Steps to Reproduce: 1. Install the above RHEL-9.1.0-20220910.0 build 2. Install & execute kernel-kernel-infiniband-ucx test script 3. Watch ucx result on client side Actual results: + [22-09-11 20:03:03] timeout --preserve-status --kill-after=5m 3m ucx_info -u a -w [1662940983.707711] [rdma-dev-22:259734:0] mm_posix.c:187 UCX ERROR Not enough memory to write total of 4292720 bytes. Please check that /dev/shm or the directory you specified has more available memory. [1662940983.707961] [rdma-dev-22:259734:0] uct_mem.c:155 UCX ERROR failed to allocate 4292720 bytes using md posix for mm_recv_desc: Out of memory [1662940983.707966] [rdma-dev-22:259734:0] mpool.c:226 UCX ERROR Failed to allocate memory pool (name=mm_recv_desc) chunk: Out of memory [1662940983.707969] [rdma-dev-22:259734:0] mm_iface.c:801 UCX ERROR failed to get the first receive descriptor <Failed to create UCP worker> + [22-09-11 20:03:03] RQA_check_result -r 254 -t 'ucp worker info for a' Expected results: results from RHEL-9.1.0-20220524.0 build + [22-09-11 19:11:54] timeout --preserve-status --kill-after=5m 3m ucx_info -u a -w # # UCP worker 'rdma-perf-03:255480' # # address: 743 bytes # atomics: 17:dc_mlx5/mlx5_0:1, 18:rc_mlx5/mlx5_0:1 # # memory: 18.90MB, file descriptors: 43 # create time: 156.139 ms # + [22-09-11 19:11:58] RQA_check_result -r 0 -t 'ucp worker info for a' Additional info: