Bug 1749608

Summary: The libfabric-1.8.0 breaks mpirun with containers
Product: [Fedora] Fedora Reporter: Honggang LI <honli>
Component: libfabricAssignee: Honggang LI <honli>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: areber, honli, hwkernel-mgr, jarod, krzysztof.czurylo, mschmidt, orion, rdma-dev-team, zguo
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libfabric-1.8.0-3.fc32 libfabric-1.8.0-3.fc31 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1731749 Environment:
Last Closed: 2019-09-06 02:35:33 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1731749    
Bug Blocks:    

Description Honggang LI 2019-09-06 01:41:29 UTC 
+++ This bug was initially created as a clone of Bug #1731749 +++

I am using the following steps to set up a container:

$ cat mpi-test/Dockerfile
FROM registry.access.redhat.com/ubi8/ubi:latest

RUN yum -y install openmpi && \\
    yum clean all && \\
    rpm -Uhv http://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/libfabric/1.6.2/1.el8/x86_64/libfabric-1.6.2-1.el8.x86_64.rpm --force && \\
    useradd --create-home --shell /bin/bash mpi

COPY ring /home/mpi/ring
USER mpi
WORKDIR /home/mpi

$ cd mpi-test/
$ podman build --tag=mpi-test:latest .

Running the container with:

$ mpirun --mca btl_vader_single_copy_mechanism none \\
   --mca orte_tmpdir_base /tmp/podman-mpirun \\
   podman run --env-host \\
     -v /tmp/podman-mpirun:/tmp/podman-mpirun \\
     --userns=keep-id \\
     --net=host --pid=host --ipc=host \\
     mpi-test /home/mpi/ring

Gives me following segfault with 1.8.0:

Jul 19 18:05:55 rhel08 systemd-coredump[31236]: Process 31223 (ring) of user 1000 dumped core.
                                                
                                                Stack trace of thread 31223:
                                                #0  0x00007f2dc6379719 ofi_mem_init (libfabric.so.1)
                                                #1  0x00007f2dc636e0c0 fi_ini (libfabric.so.1)
                                                #2  0x00007f2dc636ed15 fi_getinfo (libfabric.so.1)
                                                #3  0x00007f2dc6373a8e fi_getinfo (libfabric.so.1)
                                                #4  0x00007f2dc66b4f4e usnic_component_init (mca_btl_usnic.so)
                                                #5  0x00007f2dd418415d mca_btl_base_select (libopen-pal.so.40)
                                                #6  0x00007f2dcc2302f6 mca_bml_r2_component_init (mca_bml_r2.so)
                                                #7  0x00007f2dd4d0af94 mca_bml_base_init (libmpi.so.40)
                                                #8  0x00007f2dd4cb8474 ompi_mpi_init (libmpi.so.40)
                                                #9  0x00007f2dd4ce86b2 PMPI_Init (libmpi.so.40)
                                                #10 0x00000000004009ad n/a (/home/mpi/ring)
                                                #11 0x00007f2dd46a6813 n/a (/usr/lib64/libc-2.28.so)

Downgrading to libfabric 1.6.2 makes the segfault go away.
Comment 1 Honggang LI 2019-09-06 02:35:33 UTC 
Fixed in libfabric-1.8.0-3.fc32 . Also backported the patches for libfabric-1.8.0-3.fc31 .
Comment 2 Fedora Update System 2019-09-06 02:47:12 UTC 
FEDORA-2019-31c2c1e58e has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2019-31c2c1e58e
Comment 3 Fedora Update System 2019-09-06 10:46:10 UTC 
libfabric-1.8.0-3.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-31c2c1e58e
Comment 4 Fedora Update System 2019-09-14 00:10:08 UTC 
libfabric-1.8.0-3.fc31 has been pushed to the Fedora 31 stable repository. If problems still persist, please make note of it in this bug report.
Comment 5 Fedora Update System 2019-09-14 16:33:40 UTC 
libfabric-1.8.0-3.fc31 has been pushed to the Fedora 31 stable repository. If problems still persist, please make note of it in this bug report.