This bug has been migrated to another issue tracking site. It has been closed here and may no longer be being monitored.

If you would like to get updates for this issue, or to participate in it, you may do so at Red Hat Issue Tracker .
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2177492 - [RHEL9.2] fabtests result in many core files
Summary: [RHEL9.2] fabtests result in many core files
Keywords:
Status: CLOSED MIGRATED
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: fabtests
Version: 9.2
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Kamal Heib
QA Contact: zguo
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-03-12 10:52 UTC by Brian Chae
Modified: 2023-09-21 13:39 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-09-21 13:39:45 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker   RHEL-6072 0 None Migrated None 2023-09-21 13:39:44 UTC
Red Hat Issue Tracker RHELPLAN-151500 0 None None None 2023-03-12 10:52:47 UTC

Description Brian Chae 2023-03-12 10:52:23 UTC
Description of problem:

SIGABRT core files were observed during the fabtests on RHEL-9.2.0 Beta compose.

Version-Release number of selected component (if applicable):


Clients: rdma-dev-21
Servers: rdma-dev-20

DISTRO=RHEL-9.2.0-20230309.10
+ [23-03-11 08:59:05] cat /etc/redhat-release
Red Hat Enterprise Linux release 9.2 Beta (Plow)

+ [23-03-11 08:59:05] uname -a
Linux rdma-dev-21.rdma.lab.eng.rdu2.redhat.com 5.14.0-284.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Feb 27 20:08:54 EST 2023 x86_64 x86_64 x86_64 GNU/Linux

+ [23-03-11 08:59:05] cat /proc/cmdline
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-284.el9.x86_64 root=UUID=941c727f-9f57-43d6-8de9-0af7db8bf888 ro intel_idle.max_cstate=0 processor.max_cstate=0 intel_iommu=on iommu=on console=tty0 rd_NO_PLYMOUTH crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=UUID=11327d46-3e02-467d-b44e-086447bf8566 console=ttyS1,115200n81

+ [23-03-11 08:59:05] rpm -q rdma-core linux-firmware
rdma-core-44.0-2.el9.x86_64
linux-firmware-20230210-132.el9.noarch

+ [23-03-11 08:59:05] tail /sys/class/infiniband/mlx5_0/fw_ver /sys/class/infiniband/mlx5_1/fw_ver /sys/class/infiniband/mlx5_2/fw_ver
==> /sys/class/infiniband/mlx5_0/fw_ver <==
12.28.2006

==> /sys/class/infiniband/mlx5_1/fw_ver <==
12.28.2006

==> /sys/class/infiniband/mlx5_2/fw_ver <==
12.28.2006
+ [23-03-11 08:59:05] lspci
+ [23-03-11 08:59:05] grep -i -e ethernet -e infiniband -e omni -e ConnectX
01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
04:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
82:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
82:00.1 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]


Installed:
  fabtests-1.17.0-2.el9.x86_64                                                  
  python3-attrs-20.3.0-7.el9.noarch                                             
  python3-iniconfig-1.1.1-7.el9.noarch                                          
  python3-packaging-20.9-5.el9.noarch                                           
  python3-pluggy-0.13.1-7.el9.noarch                                            
  python3-py-1.10.0-6.el9.noarch                                                
  python3-pyparsing-2.4.7-9.el9.noarch                                          
  python3-pytest-6.2.2-6.el9.noarch                                             
  python3-toml-0.10.2-6.el9.noarch                                              
  ruby-3.0.4-160.el9_0.x86_64                                                   
  ruby-default-gems-3.0.4-160.el9_0.noarch                                      
  ruby-libs-3.0.4-160.el9_0.x86_64                                              
  rubygem-bigdecimal-3.0.0-160.el9_0.x86_64                                     
  rubygem-bundler-2.2.33-160.el9_0.noarch                                       
  rubygem-io-console-0.5.7-160.el9_0.x86_64                                     
  rubygem-json-2.5.1-160.el9_0.x86_64                                           
  rubygem-psych-3.3.2-160.el9_0.x86_64                                          
  rubygem-rdoc-6.3.3-160.el9_0.noarch                                           
  rubygems-3.2.33-160.el9_0.noarch                                              

How reproducible:


Steps to Reproduce:
1. run the fatests with the above packages on the above MLX5 IB0
2.
3.

Actual results:

In both RDMA server and client hosts, the following core files were observed.

TIME                           PID UID GID SIG     COREFILE EXE                               SIZE
Sat 2023-03-11 09:53:13 EST  64015   0   0 SIGABRT present  /usr/bin/fi_av_xfer             272.6K
Sat 2023-03-11 09:53:20 EST  64064   0   0 SIGABRT present  /usr/bin/fi_av_xfer             274.0K
Sat 2023-03-11 09:53:31 EST  64182   0   0 SIGABRT present  /usr/bin/fi_cq_data             274.6K
Sat 2023-03-11 09:53:38 EST  64228   0   0 SIGABRT present  /usr/bin/fi_cq_data             275.2K
Sat 2023-03-11 09:53:45 EST  64274   0   0 SIGABRT present  /usr/bin/fi_dgram               271.5K
Sat 2023-03-11 09:53:52 EST  64318   0   0 SIGABRT present  /usr/bin/fi_dgram_waitset       270.0K
Sat 2023-03-11 09:54:04 EST  64470   0   0 SIGABRT present  /usr/bin/fi_poll                271.8K
Sat 2023-03-11 09:54:10 EST  64518   0   0 SIGABRT present  /usr/bin/fi_poll                272.1K
Sat 2023-03-11 09:54:17 EST  64564   0   0 SIGABRT present  /usr/bin/fi_rdm                 270.8K
Sat 2023-03-11 09:54:24 EST  64608   0   0 SIGABRT present  /usr/bin/fi_rdm                 271.6K
Sat 2023-03-11 09:54:31 EST  64652   0   0 SIGABRT present  /usr/bin/fi_rdm_rma_event       269.7K
Sat 2023-03-11 09:54:39 EST  64696   0   0 SIGABRT present  /usr/bin/fi_rdm_rma_trigger     274.2K
Sat 2023-03-11 09:54:50 EST  64813   0   0 SIGABRT present  /usr/bin/fi_shared_ctx          676.3K
Sat 2023-03-11 09:55:05 EST  65038   0   0 SIGABRT present  /usr/bin/fi_shared_ctx          675.2K
Sat 2023-03-11 09:55:12 EST  65084   0   0 SIGABRT present  /usr/bin/fi_rdm_tagged_peek     270.5K
Sat 2023-03-11 09:55:21 EST  65168   0   0 SIGABRT present  /usr/bin/fi_rdm_shared_av       270.6K
Sat 2023-03-11 09:55:46 EST  65293   0   0 SIGABRT present  /usr/bin/fi_multi_mr            304.6K
Sat 2023-03-11 09:55:58 EST  65374   0   0 SIGABRT present  /usr/bin/fi_multi_ep            715.9K
Sat 2023-03-11 09:56:07 EST  65419   0   0 SIGABRT present  /usr/bin/fi_recv_cancel         303.8K
Sat 2023-03-11 09:56:17 EST  65500   0   0 SIGABRT present  /usr/bin/fi_unexpected_msg      275.5K
Sat 2023-03-11 09:56:26 EST  65544   0   0 SIGABRT present  /usr/bin/fi_msg_inject          303.7K
Sat 2023-03-11 09:56:34 EST  65591   0   0 SIGABRT present  /usr/bin/fi_msg_inject          304.6K
Sat 2023-03-11 09:56:42 EST  65637   0   0 SIGABRT present  /usr/bin/fi_msg_inject          304.4K
Sat 2023-03-11 09:56:51 EST  65681   0   0 SIGABRT present  /usr/bin/fi_msg_inject          304.3K
Sat 2023-03-11 09:56:59 EST  65726   0   0 SIGABRT present  /usr/bin/fi_bw                  307.2K
Sat 2023-03-11 09:57:07 EST  65770   0   0 SIGABRT present  /usr/bin/fi_bw                  306.1K
Sat 2023-03-11 09:57:16 EST  65851   0   0 SIGABRT present  /usr/bin/fi_rdm_multi_client    273.6K
Sat 2023-03-11 09:57:23 EST  65896   0   0 SIGABRT present  /usr/bin/fi_rdm_multi_client    274.4K
Sat 2023-03-11 09:57:46 EST  66267   0   0 SIGABRT present  /usr/bin/fi_rma_bw              302.1K
Sat 2023-03-11 09:57:55 EST  66312   0   0 SIGABRT present  /usr/bin/fi_rma_bw              301.6K
Sat 2023-03-11 09:58:03 EST  66356   0   0 SIGABRT present  /usr/bin/fi_rma_bw              301.8K
Sat 2023-03-11 09:58:11 EST  66401   0   0 SIGABRT present  /usr/bin/fi_rma_bw              302.0K
Sat 2023-03-11 09:58:20 EST  66446   0   0 SIGABRT present  /usr/bin/fi_rma_bw              304.7K
Sat 2023-03-11 09:58:28 EST  66490   0   0 SIGABRT present  /usr/bin/fi_rma_bw              303.7K
Sat 2023-03-11 09:58:37 EST  66534   0   0 SIGABRT present  /usr/bin/fi_rdm_atomic          305.7K
Sat 2023-03-11 09:58:45 EST  66579   0   0 SIGABRT present  /usr/bin/fi_rdm_atomic          304.9K
Sat 2023-03-11 09:58:55 EST  66662   0   0 SIGABRT present  /usr/bin/fi_multi_recv          275.3K
Sat 2023-03-11 09:59:06 EST  66743   0   0 SIGABRT present  /usr/bin/fi_rdm_pingpong        304.3K
Sat 2023-03-11 09:59:14 EST  66790   0   0 SIGABRT present  /usr/bin/fi_rdm_pingpong        305.9K
Sat 2023-03-11 09:59:23 EST  66834   0   0 SIGABRT present  /usr/bin/fi_rdm_pingpong        304.2K
Sat 2023-03-11 09:59:31 EST  66881   0   0 SIGABRT present  /usr/bin/fi_rdm_pingpong        304.2K
Sat 2023-03-11 09:59:47 EST  67074   0   0 SIGABRT present  /usr/bin/fi_rdm_tagged_pingpong 305.5K
Sat 2023-03-11 09:59:55 EST  67119   0   0 SIGABRT present  /usr/bin/fi_rdm_tagged_pingpong 305.4K
Sat 2023-03-11 10:00:03 EST  67165   0   0 SIGABRT present  /usr/bin/fi_rdm_tagged_pingpong 305.6K
Sat 2023-03-11 10:00:12 EST  67211   0   0 SIGABRT present  /usr/bin/fi_rdm_tagged_pingpong 306.1K
Sat 2023-03-11 10:00:20 EST  67255   0   0 SIGABRT present  /usr/bin/fi_rdm_tagged_bw       303.1K
Sat 2023-03-11 10:00:29 EST  67300   0   0 SIGABRT present  /usr/bin/fi_rdm_tagged_bw       306.0K
Sat 2023-03-11 10:00:37 EST  67344   0   0 SIGABRT present  /usr/bin/fi_rdm_tagged_bw       303.6K
Sat 2023-03-11 10:00:46 EST  67389   0   0 SIGABRT present  /usr/bin/fi_rdm_tagged_bw       303.6K
Sat 2023-03-11 10:00:55 EST  67434   0   0 SIGABRT present  /usr/bin/fi_dgram_pingpong      306.3K
Sat 2023-03-11 10:01:05 EST  67535   0   0 SIGABRT present  /usr/bin/fi_multinode           273.8K
Sat 2023-03-11 10:01:05 EST  67529   0   0 SIGABRT present  /usr/bin/fi_multinode           273.9K
Sat 2023-03-11 10:01:13 EST  67647   0   0 SIGABRT present  /usr/bin/fi_multinode           271.9K
Sat 2023-03-11 10:01:13 EST  67622   0   0 SIGABRT present  /usr/bin/fi_multinode           276.0K
Sat 2023-03-11 10:17:41 EST  71058   0   0 SIGABRT present  /usr/bin/fi_unexpected_msg      274.9K
Sat 2023-03-11 15:03:42 EST 164478   0   0 SIGABRT present  /usr/bin/fi_av_xfer             274.1K
Sat 2023-03-11 15:03:50 EST 164537   0   0 SIGABRT present  /usr/bin/fi_av_xfer             273.9K
Sat 2023-03-11 15:04:00 EST 164692   0   0 SIGABRT present  /usr/bin/fi_cq_data             273.3K
Sat 2023-03-11 15:04:07 EST 164750   0   0 SIGABRT present  /usr/bin/fi_cq_data             271.1K
Sat 2023-03-11 15:04:14 EST 164810   0   0 SIGABRT present  /usr/bin/fi_dgram               271.2K
Sat 2023-03-11 15:04:21 EST 164868   0   0 SIGABRT present  /usr/bin/fi_dgram_waitset       270.5K
Sat 2023-03-11 15:04:33 EST 165073   0   0 SIGABRT present  /usr/bin/fi_poll                271.8K
Sat 2023-03-11 15:04:40 EST 165131   0   0 SIGABRT present  /usr/bin/fi_poll                271.4K
Sat 2023-03-11 15:04:47 EST 165189   0   0 SIGABRT present  /usr/bin/fi_rdm                 271.8K
Sat 2023-03-11 15:04:54 EST 165246   0   0 SIGABRT present  /usr/bin/fi_rdm                 270.1K
Sat 2023-03-11 15:05:01 EST 165304   0   0 SIGABRT present  /usr/bin/fi_rdm_rma_event       270.6K
Sat 2023-03-11 15:05:09 EST 165361   0   0 SIGABRT present  /usr/bin/fi_rdm_rma_trigger     275.5K
Sat 2023-03-11 15:05:19 EST 165515   0   0 SIGABRT present  /usr/bin/fi_shared_ctx          677.7K
Sat 2023-03-11 15:05:35 EST 165813   0   0 SIGABRT present  /usr/bin/fi_shared_ctx          680.3K
Sat 2023-03-11 15:05:42 EST 165870   0   0 SIGABRT present  /usr/bin/fi_rdm_tagged_peek     271.0K
Sat 2023-03-11 15:05:51 EST 165976   0   0 SIGABRT present  /usr/bin/fi_rdm_shared_av       270.5K
Sat 2023-03-11 15:06:15 EST 166130   0   0 SIGABRT present  /usr/bin/fi_multi_mr            304.5K
Sat 2023-03-11 15:06:28 EST 166237   0   0 SIGABRT present  /usr/bin/fi_multi_ep            718.1K
Sat 2023-03-11 15:06:34 EST 166293   0   0 SIGABRT present  /usr/bin/fi_recv_cancel         305.0K
Sat 2023-03-11 15:06:45 EST 166400   0   0 SIGABRT present  /usr/bin/fi_unexpected_msg      275.3K
Sat 2023-03-11 15:06:53 EST 166458   0   0 SIGABRT present  /usr/bin/fi_msg_inject          303.7K
Sat 2023-03-11 15:07:02 EST 166516   0   0 SIGABRT present  /usr/bin/fi_msg_inject          306.0K
Sat 2023-03-11 15:07:10 EST 166575   0   0 SIGABRT present  /usr/bin/fi_msg_inject          306.8K
Sat 2023-03-11 15:07:18 EST 166633   0   0 SIGABRT present  /usr/bin/fi_msg_inject          304.8K
Sat 2023-03-11 15:07:27 EST 166690   0   0 SIGABRT present  /usr/bin/fi_bw                  307.7K
Sat 2023-03-11 15:07:35 EST 166747   0   0 SIGABRT present  /usr/bin/fi_bw                  307.1K
Sat 2023-03-11 15:07:43 EST 166855   0   0 SIGABRT present  /usr/bin/fi_rdm_multi_client    274.2K
Sat 2023-03-11 15:07:50 EST 166912   0   0 SIGABRT present  /usr/bin/fi_rdm_multi_client    274.2K
Sat 2023-03-11 15:08:14 EST 167402   0   0 SIGABRT present  /usr/bin/fi_rma_bw              304.3K
Sat 2023-03-11 15:08:22 EST 167461   0   0 SIGABRT present  /usr/bin/fi_rma_bw              303.0K
Sat 2023-03-11 15:08:31 EST 167518   0   0 SIGABRT present  /usr/bin/fi_rma_bw              301.9K
Sat 2023-03-11 15:08:39 EST 167576   0   0 SIGABRT present  /usr/bin/fi_rma_bw              304.6K
Sat 2023-03-11 15:08:48 EST 167633   0   0 SIGABRT present  /usr/bin/fi_rma_bw              302.3K
Sat 2023-03-11 15:08:56 EST 167692   0   0 SIGABRT present  /usr/bin/fi_rma_bw              301.8K
Sat 2023-03-11 15:09:04 EST 167749   0   0 SIGABRT present  /usr/bin/fi_rdm_atomic          305.7K
Sat 2023-03-11 15:09:13 EST 167808   0   0 SIGABRT present  /usr/bin/fi_rdm_atomic          303.9K
Sat 2023-03-11 15:09:22 EST 167913   0   0 SIGABRT present  /usr/bin/fi_multi_recv          274.4K
Sat 2023-03-11 15:09:33 EST 168019   0   0 SIGABRT present  /usr/bin/fi_rdm_pingpong        306.8K
Sat 2023-03-11 15:09:42 EST 168076   0   0 SIGABRT present  /usr/bin/fi_rdm_pingpong        303.0K
Sat 2023-03-11 15:09:50 EST 168134   0   0 SIGABRT present  /usr/bin/fi_rdm_pingpong        304.5K
Sat 2023-03-11 15:09:59 EST 168192   0   0 SIGABRT present  /usr/bin/fi_rdm_pingpong        302.9K
Sat 2023-03-11 15:10:14 EST 168445   0   0 SIGABRT present  /usr/bin/fi_rdm_tagged_pingpong 303.9K
Sat 2023-03-11 15:10:22 EST 168502   0   0 SIGABRT present  /usr/bin/fi_rdm_tagged_pingpong 306.0K
Sat 2023-03-11 15:10:31 EST 168559   0   0 SIGABRT present  /usr/bin/fi_rdm_tagged_pingpong 306.9K
Sat 2023-03-11 15:10:39 EST 168617   0   0 SIGABRT present  /usr/bin/fi_rdm_tagged_pingpong 306.3K
Sat 2023-03-11 15:10:48 EST 168676   0   0 SIGABRT present  /usr/bin/fi_rdm_tagged_bw       306.3K
Sat 2023-03-11 15:10:56 EST 168733   0   0 SIGABRT present  /usr/bin/fi_rdm_tagged_bw       303.4K
Sat 2023-03-11 15:11:04 EST 168791   0   0 SIGABRT present  /usr/bin/fi_rdm_tagged_bw       305.8K
Sat 2023-03-11 15:11:13 EST 168849   0   0 SIGABRT present  /usr/bin/fi_rdm_tagged_bw       307.0K
Sat 2023-03-11 15:11:22 EST 168906   0   0 SIGABRT present  /usr/bin/fi_dgram_pingpong      308.1K
Sat 2023-03-11 15:11:32 EST 169022   0   0 SIGABRT present  /usr/bin/fi_multinode           275.1K
Sat 2023-03-11 15:11:32 EST 169019   0   0 SIGABRT present  /usr/bin/fi_multinode           272.6K
Sat 2023-03-11 15:11:40 EST 169126   0   0 SIGABRT present  /usr/bin/fi_multinode           273.3K
Sat 2023-03-11 15:11:40 EST 169124   0   0 SIGABRT present  /usr/bin/fi_multinode           276.2K
Sat 2023-03-11 15:28:08 EST 172866   0   0 SIGABRT present  /usr/bin/fi_unexpected_msg      273.9K
total 35012



Expected results:


Additional info:

Comment 1 zguo 2023-05-22 03:29:17 UTC
Also can catch this issue on irdma iwarp.

https://beaker.engineering.redhat.com/jobs/7866063

Comment 2 RHEL Program Management 2023-09-21 13:39:27 UTC
Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

Comment 3 RHEL Program Management 2023-09-21 13:39:45 UTC
This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.


Note You need to log in before you can comment on or make changes to this bug.