Bug 1705296 - Cannot build vtk for Python 3.8: nothing provides libmpi.so.12()(64bit) needed by python3-mpi4py-mpich
Summary: Cannot build vtk for Python 3.8: nothing provides libmpi.so.12()(64bit) neede...
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: mpich
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Zbigniew Jędrzejewski-Szmek
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On: 1705301
Blocks: PYTHON38
TreeView+ depends on / blocked
 
Reported: 2019-05-01 23:09 UTC by Miro Hrončok
Modified: 2019-05-08 03:13 UTC (History)
5 users (show)

Fixed In Version: mpich-3.2.1-11.fc31 openmpi-4.0.1-2.fc31
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-05-08 03:13:56 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Miro Hrončok 2019-05-01 23:09:25 UTC
I have an interesting problem, when I try to build vtk for Python 3.8:

     # /usr/bin/dnf builddep --installroot /var/lib/mock/900764-fedora-rawhide-x86_64-1556745192.100416/root/ --releasever 31 --setopt=deltarpm=False --disableplugin=local --disableplugin=spacewalk --disableplugin=local --disableplugin=spacewalk /var/lib/mock/900764-fedora-rawhide-x86_64-1556745192.100416/root//builddir/build/SRPMS/vtk-8.2.0-4.fc31.src.rpm
    Error: 
     Problem 1: cannot install the best candidate for the job
      - nothing provides libmpi.so.12()(64bit) needed by python3-mpi4py-mpich-3.0.1-3.fc31.x86_64
     Problem 2: cannot install the best candidate for the job
      - nothing provides libmpi.so.40()(64bit) needed by python3-mpi4py-openmpi-3.0.1-3.fc31.x86_64


Indeed, I have rebuilt mpi4py for Python 3.8 and it requires:

    $ dnf repoquery --repo=python3.8 --requires python3-mpi4py-openmpi
    libc.so.6(GLIBC_2.14)(64bit)
    libdl.so.2()(64bit)
    libdl.so.2(GLIBC_2.2.5)(64bit)
    libmpi.so.40()(64bit)
    libpthread.so.0()(64bit)
    libpython3.8m.so.1.0()(64bit)
    mpi4py-common = 3.0.1-3.fc31
    python(abi) = 3.8
    python3-openmpi(x86-64)
    rtld(GNU_HASH)

    $ dnf repoquery --repo=python3.8 --requires python3-mpi4py-mpich
    libc.so.6(GLIBC_2.14)(64bit)
    libdl.so.2()(64bit)
    libdl.so.2(GLIBC_2.2.5)(64bit)
    libmpi.so.12()(64bit)
    libpthread.so.0()(64bit)
    libpython3.8m.so.1.0()(64bit)
    mpi4py-common = 3.0.1-3.fc31
    python(abi) = 3.8
    python3-mpich(x86-64)
    rtld(GNU_HASH)


However, in rawhide, it requires:

    $ dnf repoquery --repo=rawhide --requires python3-mpi4py-openmpi
    Nepodařilo se načíst plugin: py3query
    Poslední kontrola metadat: před 0:14:02, Čt 2. května 2019, 00:48:02 CEST.
    libc.so.6(GLIBC_2.14)(64bit)
    libdl.so.2()(64bit)
    libdl.so.2(GLIBC_2.2.5)(64bit)
    libmpi.so.40()(64bit)(openmpi-x86_64)
    libpthread.so.0()(64bit)
    libpython3.7m.so.1.0()(64bit)
    mpi4py-common = 3.0.1-2.fc31
    python(abi) = 3.7
    python3-openmpi(x86-64)
    rtld(GNU_HASH)

    $ dnf repoquery --repo=rawhide --requires python3-mpi4py-mpich
    Nepodařilo se načíst plugin: py3query
    Poslední kontrola metadat: před 0:14:09, Čt 2. května 2019, 00:48:02 CEST.
    libc.so.6(GLIBC_2.14)(64bit)
    libdl.so.2()(64bit)
    libdl.so.2(GLIBC_2.2.5)(64bit)
    libmpi.so.12()(64bit)(mpich-x86_64)
    libpthread.so.0()(64bit)
    libpython3.7m.so.1.0()(64bit)
    mpi4py-common = 3.0.1-2.fc31
    python(abi) = 3.7
    python3-mpich(x86-64)
    rtld(GNU_HASH)


Notice the difference between:

    libmpi.so.40()(64bit) and libmpi.so.40()(64bit)(openmpi-x86_64)
    libmpi.so.12()(64bit) and libmpi.so.12()(64bit)(mpich-x86_64)


Is this something that's relevant to python 3.8, or is something else broken entirely?

I've tried to rebuild mpi4py in rawhide Koji, to see what it would require, but it FTBFS on 3 architectures:

https://koji.fedoraproject.org/koji/taskinfo?taskID=34566237

It however includes the suffix on the not-failed ones:

    libmpi.so.12()(64bit)(mpich-s390x)

Relevant Copr builds:

https://copr.fedorainfracloud.org/coprs/g/python/python3.8/build/891230/ mpi4py
https://copr.fedorainfracloud.org/coprs/g/python/python3.8/build/900764/ vtk

Comment 1 Miro Hrončok 2019-05-01 23:13:02 UTC
As a blind shot, I'm now trying to rebuild mpich and openmpi against Python 3.8 first, than rebuild mpi4py, vtk.

Comment 2 Zbigniew Jędrzejewski-Szmek 2019-05-02 08:49:39 UTC
> nothing provides libmpi.so.12()(64bit) needed by python3-mpi4py-mpich-3.0.1-3.fc31.x86_64

There's no "global" libmpi.so.12, and it only makes sense when qualified with either
'(openmpi-x86_64)' or '(mpich-x86_64)'. Thus any package that has an unadorned libmpi
Requires or Provides is busted.

For python3-mpi4py-openmpi-0:3.0.1-2.fc31.x86_64 which seems to be the latest build in koji,
I see:
$ dnf repoquery --requires python3-mpi4py-openmpi-0:3.0.1-2.fc31.x86_64 | grep libmpi
libmpi.so.40()(64bit)(openmpi-x86_64)
$ dnf repoquery --requires python3-mpi4py-mpich-0:3.0.1-2.fc31.x86_64 | grep libmpi
libmpi.so.12()(64bit)(mpich-x86_64)

and this looks correct. It seems your python3-mpi4py-mpich-3.0.1-3.fc31.x86_64 is wrong
here.

Comment 3 Miro Hrončok 2019-05-02 09:00:30 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #2)
> It seems your
> python3-mpi4py-mpich-3.0.1-3.fc31.x86_64 is wrong
> here.

Indeed. Do you know based on what are those requires generated?

Comment 4 Zbigniew Jędrzejewski-Szmek 2019-05-02 09:24:34 UTC
mpi.req from rpm-mpi-hooks?

Comment 5 Orion Poplawski 2019-05-07 03:38:45 UTC
For mpi.req to work, mpich and openmpi will have to be rebuilt with python 3.8 first.  This is because it looks for library dependencies to add the requires suffix for in $MPI_PYTHON3_SITEARCH as defined in the respective mpi modules which is set at compile time.

Comment 6 Miro Hrončok 2019-05-07 07:54:21 UTC
What about (something like) this:

https://src.fedoraproject.org/rpms/mpich/pull-request/1

Comment 7 Zbigniew Jędrzejewski-Szmek 2019-05-07 08:59:29 UTC
I think we can guard against this. I opened https://src.fedoraproject.org/rpms/openmpi/pull-request/6
to add
openmpi(built-for-python2) = 2.7
openmpi(built-for-python3) = 3.7

mpi4py could than have BuildRequires: openmpi(built-for-python3) = %{python3_version}.

PTAL.

Comment 8 Zbigniew Jędrzejewski-Szmek 2019-05-07 09:04:58 UTC
Hmm, so there are two alternative approaches. Miro's is more self-contained.
Let's go with that.

Comment 9 Zbigniew Jędrzejewski-Szmek 2019-05-07 09:28:33 UTC
https://src.fedoraproject.org/rpms/openmpi/pull-request/7 also.

Comment 10 Zbigniew Jędrzejewski-Szmek 2019-05-07 09:53:02 UTC
Built in rawhide.

Comment 11 Orion Poplawski 2019-05-08 03:13:56 UTC
I've gone with a slightly different version for openmpi, but thanks for the fix.


Note You need to log in before you can comment on or make changes to this bug.