Bug 1749608 - The libfabric-1.8.0 breaks mpirun with containers
Summary: The libfabric-1.8.0 breaks mpirun with containers
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: libfabric
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Honggang LI
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On: 1731749
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-09-06 01:41 UTC by Honggang LI
Modified: 2019-09-14 16:33 UTC (History)
9 users (show)

Fixed In Version: libfabric-1.8.0-3.fc32 libfabric-1.8.0-3.fc31
Clone Of: 1731749
Environment:
Last Closed: 2019-09-06 02:35:33 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)

Description Honggang LI 2019-09-06 01:41:29 UTC
+++ This bug was initially created as a clone of Bug #1731749 +++

I am using the following steps to set up a container:

$ cat mpi-test/Dockerfile
FROM registry.access.redhat.com/ubi8/ubi:latest

RUN yum -y install openmpi && \\
    yum clean all && \\
    rpm -Uhv http://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/libfabric/1.6.2/1.el8/x86_64/libfabric-1.6.2-1.el8.x86_64.rpm --force && \\
    useradd --create-home --shell /bin/bash mpi

COPY ring /home/mpi/ring
USER mpi
WORKDIR /home/mpi

$ cd mpi-test/
$ podman build --tag=mpi-test:latest .

Running the container with:

$ mpirun --mca btl_vader_single_copy_mechanism none \\
   --mca orte_tmpdir_base /tmp/podman-mpirun \\
   podman run --env-host \\
     -v /tmp/podman-mpirun:/tmp/podman-mpirun \\
     --userns=keep-id \\
     --net=host --pid=host --ipc=host \\
     mpi-test /home/mpi/ring

Gives me following segfault with 1.8.0:

Jul 19 18:05:55 rhel08 systemd-coredump[31236]: Process 31223 (ring) of user 1000 dumped core.
                                                
                                                Stack trace of thread 31223:
                                                #0  0x00007f2dc6379719 ofi_mem_init (libfabric.so.1)
                                                #1  0x00007f2dc636e0c0 fi_ini (libfabric.so.1)
                                                #2  0x00007f2dc636ed15 fi_getinfo (libfabric.so.1)
                                                #3  0x00007f2dc6373a8e fi_getinfo (libfabric.so.1)
                                                #4  0x00007f2dc66b4f4e usnic_component_init (mca_btl_usnic.so)
                                                #5  0x00007f2dd418415d mca_btl_base_select (libopen-pal.so.40)
                                                #6  0x00007f2dcc2302f6 mca_bml_r2_component_init (mca_bml_r2.so)
                                                #7  0x00007f2dd4d0af94 mca_bml_base_init (libmpi.so.40)
                                                #8  0x00007f2dd4cb8474 ompi_mpi_init (libmpi.so.40)
                                                #9  0x00007f2dd4ce86b2 PMPI_Init (libmpi.so.40)
                                                #10 0x00000000004009ad n/a (/home/mpi/ring)
                                                #11 0x00007f2dd46a6813 n/a (/usr/lib64/libc-2.28.so)

Downgrading to libfabric 1.6.2 makes the segfault go away.

Comment 1 Honggang LI 2019-09-06 02:35:33 UTC
Fixed in libfabric-1.8.0-3.fc32 . Also backported the patches for libfabric-1.8.0-3.fc31 .

Comment 2 Fedora Update System 2019-09-06 02:47:12 UTC
FEDORA-2019-31c2c1e58e has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2019-31c2c1e58e

Comment 3 Fedora Update System 2019-09-06 10:46:10 UTC
libfabric-1.8.0-3.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-31c2c1e58e

Comment 4 Fedora Update System 2019-09-14 00:10:08 UTC
libfabric-1.8.0-3.fc31 has been pushed to the Fedora 31 stable repository. If problems still persist, please make note of it in this bug report.

Comment 5 Fedora Update System 2019-09-14 16:33:40 UTC
libfabric-1.8.0-3.fc31 has been pushed to the Fedora 31 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.