Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1732548

Summary:

glibc: Segmentation fault in libc-2.28.so when running IMB-MPI1

Product:

Red Hat Enterprise Linux 8

Reporter:

Adrian Suhov <v-adsuho>

Component:

glibc

Assignee:

Florian Weimer <fweimer>

Status:

CLOSED INSUFFICIENT_DATA

QA Contact:

qe-baseos-tools-bugs

Severity:

unspecified

Docs Contact:

Priority:

unspecified

Version:

8.0

CC:

ashankar, codonell, dj, fweimer, jopoulso, juhlee, mnewsome, pfrankli, vkuznets

Target Milestone:

Target Release:

8.0

Hardware:

x86_64

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2019-08-01 14:43:59 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
dmesg output for the bad VM	none

Description Adrian Suhov 2019-07-23 16:04:56 UTC

Created attachment 1592917 [details]
dmesg output for the bad VM

Description of problem:
This issue occurred when trying to test RDMA/Infiniband on Azure. The VM size used was HC44rs. Intel MPI version 2018.3.222 was used for testing

Version-Release number of selected component (if applicable):
2.28

How reproducible:
100%

Steps to Reproduce:
1. Setup 2 VMs on Azure, HC44rs VM size. Make sure they're one the same subnet so they can see each other over the network.
2. Install MLNX OFED driver 4.5-1.0.1.0. Disable firewall and SELinux. 
3. Change LOAD_EIPOIB to 'yes' in /etc/infiniband/openib.conf
4. Download and install Intel MPI.
5. Reboot both VMs
6. After reboot, if everything is ok on the setup side, the command that will cause the fail is this:
/opt/intel/compilers_and_libraries_2018.3.222/linux/mpi/intel64/bin/mpirun -hosts 10.0.0.4,10.0.0.5 -ppn 2 -n 2 -env I_MPI_FABRICS ofa -env I_MPI_DAPL_PROVIDER=ofa-v2-ib0 -env SECS_PER_SAMPLE=600 /opt/intel/compilers_and_libraries_2018.3.222/linux/mpi/intel64/bin/IMB-MPI1 pingpong

Actual results:
===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 15272 RUNNING AT 10.0.0.5
=   EXIT CODE: 11
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
   Intel(R) MPI Library troubleshooting guide:
      https://software.intel.com/node/561764
===================================================================================

When looking inside dmesg, we can see segmentation fault errors:

[   42.725166] IMB-MPI1[2756]: segfault at 0 ip 00007f4281332b06 sp 00007ffd1b340190 error 4 in libc-2.28.so[7f42812a
[   42.725189] IMB-MPI1[2757]: segfault at 0 ip 00007f2d7822cb06 sp 00007ffc36ce0310 error 4 in libc-2.28.so[7f2d781a
[   42.737587] Code: 00 00 00 90 f3 0f 1e fa 48 8d 15 9d 5f 33 00 e9 00 00 00 00 f3 0f 1e fa 41 54 49 89 f4 55 48 89
[   42.774843] Code: 00 00 00 90 f3 0f 1e fa 48 8d 15 9d 5f 33 00 e9 00 00 00 00 f3 0f 1e fa 41 54 49 89 f4 55 48 89
[   43.905730] EXT4-fs (sdb1): mounted filesystem with ordered data mode. Opts: (null)
[   43.971992] Adding 2097148k swap on /mnt/resource/swapfile.  Priority:-2 extents:6 across:2260988k FS
[   53.259670] IMB-MPI1[2956]: segfault at 0 ip 00007f0e121dbb06 sp 00007ffff37e0950 error 4 in libc-2.28.so[7f0e1215
[   53.259691] IMB-MPI1[2957]: segfault at 0 ip 00007fced43a5b06 sp 00007ffc87ba6420 error 4 in libc-2.28.so[7fced431
[   53.272286] Code: 00 00 00 90 f3 0f 1e fa 48 8d 15 9d 5f 33 00 e9 00 00 00 00 f3 0f 1e fa 41 54 49 89 f4 55 48 89
[   53.286612] Code: 00 00 00 90 f3 0f 1e fa 48 8d 15 9d 5f 33 00 e9 00 00 00 00 f3 0f 1e fa 41 54 49 89 f4 55 48 89
[   59.719505] hv_balloon: Max. dynamic memory size: 360448 MB
[   63.795583] IMB-MPI1[3110]: segfault at 0 ip 00007f40fb537b06 sp 00007ffc366982a0 error 4 in libc-2.28.so[7f40fb4a
[   63.795606] IMB-MPI1[3111]: segfault at 0 ip 00007f661e633b06 sp 00007ffef41b7a10 error 4
[   63.808239] Code: 00 00 00 90 f3 0f 1e fa 48 8d 15 9d 5f 33 00 e9 00 00 00 00 f3 0f 1e fa 41 54 49 89 f4 55 48 89
[   63.841781]  in libc-2.28.so[7f661e5a9000+1ba000]
[   63.847642] Code: 00 00 00 90 f3 0f 1e fa 48 8d 15 9d 5f 33 00 e9 00 00 00 00 f3 0f 1e fa 41 54 49 89 f4 55 48 89
[   74.380840] IMB-MPI1[3269]: segfault at 0 ip 00007f4f57891b06 sp 00007ffff0a1aa80 error 4 in libc-2.28.so[7f4f5780
[   74.380861] IMB-MPI1[3270]: segfault at 0 ip 00007fbc8c8b9b06 sp 00007ffd8ae67180 error 4
[   74.392970] Code: 00 00 00 90 f3 0f 1e fa 48 8d 15 9d 5f 33 00 e9 00 00 00 00 f3 0f 1e fa 41 54 49 89 f4 55 48 89
[   74.392970]  in libc-2.28.so[7fbc8c82f000+1ba000]
[   74.429882] Code: 00 00 00 90 f3 0f 1e fa 48 8d 15 9d 5f 33 00 e9 00 00 00 00 f3 0f 1e fa 41 54 49 89 f4 55 48 89
[   84.916997] IMB-MPI1[3426]: segfault at 0 ip 00007f019d1e5b06 sp 00007fff6bd686a0 error 4 in libc-2.28.so[7f019d15
[   84.917024] IMB-MPI1[3427]: segfault at 0 ip 00007fa83abeab06 sp 00007ffcd2f418f0 error 4
[   84.929223] Code: 00 00 00 90 f3 0f 1e fa 48 8d 15 9d 5f 33 00 e9 00 00 00 00 f3 0f 1e fa 41 54 49 89 f4 55 48 89
[   84.962649]  in libc-2.28.so[7fa83ab60000+1ba000]
[   84.968561] Code: 00 00 00 90 f3 0f 1e fa 48 8d 15 9d 5f 33 00 e9 00 00 00 00 f3 0f 1e fa 41 54 49 89 f4 55 48 89
[   95.453505] IMB-MPI1[3499]: segfault at 0 ip 00007f3999beab06 sp 00007fff51e97690 error 4 in libc-2.28.so[7f3999b6
[   95.453527] IMB-MPI1[3500]: segfault at 0 ip 00007f42ac183b06 sp 00007ffd53f02410 error 4
[   95.465435] Code: 00 00 00 90 f3 0f 1e fa 48 8d 15 9d 5f 33 00 e9 00 00 00 00 f3 0f 1e fa 41 54 49 89 f4 55 48 89
[   95.465436]  in libc-2.28.so[7f42ac0f9000+1ba000]
[   95.503767] Code: 00 00 00 90 f3 0f 1e fa 48 8d 15 9d 5f 33 00 e9 00 00 00 00 f3 0f 1e fa 41 54 49 89 f4 55 48 89

Expected results:
The benchmark should return exit code 0 and dmesg should be clean.

Comment 2 Vitaly Kuznetsov 2019-07-24 15:32:47 UTC

Hi Adrian,

do I understand correctly that to enable RDMA on Azure one has to at least:

1) Install out-of-the-box 'LIS' drivers containing 'vmbus_rdma' driver

2) Install out-of-the-box Mellanox 'OFED' drivers?

or am I missing some recent changes? In case I'm not this will likely remain completely unsupported by Red Hat.
We can, however, look at the glibc issue if it is somehow reproducible without these external components.

Comment 3 Ju Lee 2019-07-24 20:37:35 UTC

Hi Vitaly,

Our test automation covers many different distro in Azure, and here is a quick summary. RHEL and CentOS will need OFED driver installation and LIS installation. Also there are the required packages and waagent. You will find detail information in Github, https://github.com/LIS/LISAv2/blob/master/Testscripts/Linux/SetupRDMA.sh#L65.

You can find the test command in this script line, https://github.com/LIS/LISAv2/blob/master/Testscripts/Linux/TestRDMA_MultiVM.sh#L710 

We have done many different configurations in my testing, and Intel MPI (2018.3.222) failed in RHEL 8.0 + 4.18.0-80.4.2. LIS does not participate in this case. However, IBM Platform MPI, Open MPI, MVAPICH MPI passed in the same RHEL 8.0 VM though.

Comment 4 Florian Weimer 2019-07-25 08:15:51 UTC

I'm sorry, but Red Hat does not support the Mellanox OFED drivers.  You will have to reproduce this issue with the supported openmpi-based stack, with an untainted kernel, before we can debug this issue.

Feel free to post a backtrace with debugging symbols from the crash, and I can see if it is potentially glibc-related.  In msot cases when we see crashes in libc.so.6, it is the result of applications passsing NULL pointers to functions such as strlen, so the crash is due to an application bug.

Comment 5 Florian Weimer 2019-08-01 14:43:59 UTC

Without a backtrace or coredump, we are not able to assist with diagnosing this issue.  Sorry.