Bug 1945997

Description of problem:

Core is dumped consistently while running "/usr/sbin/dump_fts" for infiniban-diags for RDMA hosts.


Version-Release number of selected component (if applicable):

DISTRO=RHEL-8.4.0-20210309.1

+ [21-04-02 16:29:07] cat /etc/redhat-release
Red Hat Enterprise Linux release 8.4 Beta (Ootpa)

+ [21-04-02 16:29:07] uname -a
Linux rdma-virt-02.lab.bos.redhat.com 4.18.0-293.el8.x86_64 #1 SMP Mon Mar 1 10:04:09 EST 2021 x86_64 x86_64 x86_64 GNU/Linux

+ [21-04-02 16:29:07] cat /proc/cmdline
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-293.el8.x86_64 root=UUID=34b90771-4181-41fd-88ae-42b3bbd44439 ro intel_idle.max_cstate=0 processor.max_cstate=0 intel_iommu=on iommu=on console=tty0 rd_NO_PLYMOUTH crashkernel=auto resume=UUID=d9d59ac4-2aa3-4f3c-b5e6-81b2812a7afa console=ttyS1,115200n81

+ [21-04-02 16:29:07] rpm -q rdma-core linux-firmware
rdma-core-32.0-4.el8.x86_64
linux-firmware-20201218-102.git05789708.el8.noarch

+ [21-04-02 16:29:07] tail /sys/class/infiniband/mlx5_0/fw_ver /sys/class/infiniband/mlx5_1/fw_ver /sys/class/infiniband/mlx5_bond_0/fw_ver
==> /sys/class/infiniband/mlx5_0/fw_ver <==
12.28.2006

==> /sys/class/infiniband/mlx5_1/fw_ver <==
12.28.2006

==> /sys/class/infiniband/mlx5_bond_0/fw_ver <==
14.29.2002
+ [21-04-02 16:29:07] lspci
+ [21-04-02 16:29:07] grep -i -e ethernet -e infiniband -e omni -e ConnectX
02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
03:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
03:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
04:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
04:00.1 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
05:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
05:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]

[root@rdma-virt-02 var]$ ibstatus
Infiniband device 'mlx5_0' port 1 status:
        default gid:     fe80:0000:0000:0000:e41d:2d03:00e7:0ff6
        base lid:        0x13
        sm lid:          0xd
        state:           4: ACTIVE
        phys state:      5: LinkUp
        rate:            56 Gb/sec (4X FDR)
        link_layer:      InfiniBand

Infiniband device 'mlx5_1' port 1 status:
        default gid:     fe80:0000:0000:0001:e41d:2d03:00e7:0ff7
        base lid:        0x26
        sm lid:          0x1
        state:           4: ACTIVE
        phys state:      5: LinkUp
        rate:            100 Gb/sec (4X EDR)
        link_layer:      InfiniBand

Infiniband device 'mlx5_bond_0' port 1 status:
        default gid:     fe80:0000:0000:0000:e61d:2dff:fefd:a72a
        base lid:        0x0
        sm lid:          0x0
        state:           4: ACTIVE
        phys state:      5: LinkUp
        rate:            25 Gb/sec (1X EDR)
        link_layer:      Ethernet





How reproducible:

100%

Steps to Reproduce:
1. Have RHEL8.4 is up booted, up and running in one of RDMA lab hosts - rdma-virt-02 in this case
2. run the following IB diag command

/usr/sbin/dump_fts -P 1


3.

Actual results:
Running python3 As root: 
TIME                            PID   UID   GID SIG COREFILE  EXE
Fri 2021-04-02 16:30:50 EDT   35026     0     0  11 present   /usr/sbin/dump_fts
total 72
-rw-r-----. 1 root root 66731 Apr  2 16:30 core.dump_fts.0.e2b000c7c8fb45baa1e1f29de0cd318f.35026.1617395450000000.lz4 <<<=======

b'Red Hat Enterprise Linux release 8.4 Beta (Ootpa)\n'

Firmware Bug, please contact your hardware vendor.
Apr 02 12:17:22 rdma-virt-02.lab.bos.redhat.com kernel: ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored

Errors found during boot. Please check if it's a bug.
Errors found during boot. Please check if it's a bug.
Errors found during boot. Please check if it's a bug.
Errors found during boot. Please check if it's a bug.
Errors found during boot. Please check if it's a bug.
Errors found during boot. Please check if it's a bug.
Errors found during boot. Please check if it's a bug.
Errors found during boot. Please check if it's a bug.
Errors found during boot. Please check if it's a bug.
Errors found during boot. Please check if it's a bug.
Apr 02 12:17:23 rdma-virt-02.lab.bos.redhat.com kernel: ERST: Error Record Serialization Table (ERST) support is initialized.

Apr 02 16:17:33 rdma-virt-02.lab.bos.redhat.com kernel: ACPI Error: No handler for Region [SYSI] (00000000547761c4) [IPMI] (20200925/evregion-132)

Apr 02 16:17:33 rdma-virt-02.lab.bos.redhat.com kernel: ACPI Error: Region IPMI (ID=7) has no handler (20200925/exfldio-265)

Apr 02 16:17:33 rdma-virt-02.lab.bos.redhat.com kernel: ACPI Error: Aborting method \_SB.PMI0._GHL due to previous error (AE_NOT_EXIST) (20200925/psparse-531)

Apr 02 16:17:33 rdma-virt-02.lab.bos.redhat.com kernel: ACPI Error: Aborting method \_SB.PMI0._PMC due to previous error (AE_NOT_EXIST) (20200925/psparse-531)

Apr 02 16:17:33 rdma-virt-02.lab.bos.redhat.com kernel: ACPI Error: AE_NOT_EXIST, Evaluating _PMC (20200925/power_meter-756)

Apr 02 16:18:51 rdma-virt-02.lab.bos.redhat.com NetworkManager[1480]: <error> [1617394731.9644] platform-linux: sysctl: failed to set 'bonding/ad_actor_system' to '00:00:00:00:00:00': (22) Invalid argument

Apr 02 16:30:50 rdma-virt-02.lab.bos.redhat.com kernel: dump_fts[35026]: segfault at 20 ip 000055692c235b7c sp 00007ffda925a4c0 error 4 in dump_fts[55692c233000+8000] <<<=============================



Expected results:

Should have successfully run the command without any core dump


Additional info:


GDB core decode
===============

[root@rdma-virt-02 coredump]$ gdb /tmp/core.dump_fts.0.e2b000c7c8fb45baa1e1f29de0cd318f.35026.1617395450000000
GNU gdb (GDB) Red Hat Enterprise Linux 8.2-15.el8
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
[New LWP 35026]
Reading symbols from /usr/sbin/dump_fts...Reading symbols from /usr/lib/debug/usr/sbin/dump_fts-32.0-4.el8.x86_64.debug...done.
done.
Core was generated by `/usr/sbin/dump_fts -P 1'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  dump_lid (str_len=200, portguid=0x7ffda925a558, base_port_lid=<synthetic pointer>, 
    last_port_lid=<synthetic pointer>, fabric=0x55692cdbb720, valid=1, lid=33, 
    str=0x7ffda925a680 ": (Channel Adapter portguid 0x248a07030049d4f1: 'rdma-dev-21 mlx5_2')")
    at infiniband-diags/dump_fts.c:272
272             nodeguid = port->node->guid;
(gdb) where
#0  dump_lid (str_len=200, portguid=0x7ffda925a558, base_port_lid=<synthetic pointer>, 
    last_port_lid=<synthetic pointer>, fabric=0x55692cdbb720, valid=1, lid=33, 
    str=0x7ffda925a680 ": (Channel Adapter portguid 0x248a07030049d4f1: 'rdma-dev-21 mlx5_2')")
    at infiniband-diags/dump_fts.c:272
#1  dump_unicast_tables (node=0x55692cdc2b10, startl=0, endl=58, mad_port=0x55692cdc9ce0, 
    fabric=0x55692cdbb720) at infiniband-diags/dump_fts.c:366
#2  0x00007f7afb56abd0 in ibnd_iter_nodes_type (fabric=<optimized out>, 
    func=0x55692c236690 <process_switch>, node_type=<optimized out>, user_data=0x55692cdbb720)
    at libibnetdisc/ibnetdisc.c:930
#3  0x000055692c235577 in main (argc=<optimized out>, argv=<optimized out>)
    at infiniband-diags/dump_fts.c:481
(gdb)