Bug 1287541

Summary: Shipping libopenblas?64_.so with symbol suffix
Product: [Fedora] Fedora Reporter: Milan Bouchet-Valat <nalimilan>
Component: openblasAssignee: Susi Lehtola <susi.lehtola>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: adm.fkt.physik, orion, susi.lehtola
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openblas-0.2.15-3.fc24 openblas-0.2.15-5.fc22 openblas-0.2.15-5.fc23 openblas-0.2.15-5.el5 openblas-0.2.15-5.el7 openblas-0.2.15-5.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-01-28 18:44:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Updated spec none

Description Milan Bouchet-Valat 2015-12-02 10:15:17 UTC
Julia and possibly other packages like Numpy [1] cannot currently link to the ILP64 OpenBLAS in distribution packages because of symbol name conflicts for libraries that expect LP64 versions. The solution that was found with upstream [2; 3] is to build an ILP64 library with a suffix added to all symbol names. That way, libraries can load the LP64 version as they are used to in parallel without any conflict.

For Julia to use the ILP64 OpenBLAS (as upstream recommends) in Fedora, I'd therefore need the openblas package to include such a library. This can be done by building and installing a new series of libraries using something like:
make INTERFACE64=1 SYMBOLSUFFIX=64_

The OpenBLAS Makefiles are set up so that the libraries will be be called  libopenblas?64_.so.

The suffix isn't really standardized [4]. "64_" it's the one used by Julia upstream, and it seems reasonable -- except if we have a reason to use another one (in which case we could imagine coordinating with Julia upstream).

Thanks!


1: https://github.com/numpy/numpy/issues/5906
2: https://github.com/xianyi/OpenBLAS/pull/459
3: https://github.com/xianyi/OpenBLAS/pull/656
4: https://github.com/xianyi/OpenBLAS/issues/646

Comment 1 Fedora Update System 2015-12-24 05:53:38 UTC
openblas-0.2.15-2.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-c01db2e7ae

Comment 2 Fedora Update System 2015-12-24 05:53:39 UTC
openblas-0.2.15-2.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2015-35a90d4541

Comment 3 Milan Bouchet-Valat 2015-12-24 13:32:37 UTC
Thanks, this works perfectly for Julia!

Comment 4 Fedora Update System 2015-12-25 01:57:48 UTC
openblas-0.2.15-2.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-c01db2e7ae

Comment 5 Fedora Update System 2015-12-28 22:53:50 UTC
openblas-0.2.15-2.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.

Comment 6 Fedora Update System 2015-12-30 20:55:58 UTC
openblas-0.2.15-2.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-35a90d4541

Comment 7 Fedora Update System 2016-01-05 18:16:45 UTC
openblas-0.2.15-2.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2016-6e4e9ae46c

Comment 8 Fedora Update System 2016-01-05 18:16:47 UTC
openblas-0.2.15-2.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2016-b5d8b1465e

Comment 9 Fedora Update System 2016-01-05 23:21:12 UTC
openblas-0.2.15-2.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2016-6e4e9ae46c

Comment 10 Fedora Update System 2016-01-05 23:25:59 UTC
openblas-0.2.15-2.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2016-b5d8b1465e

Comment 11 Milan Bouchet-Valat 2016-01-07 17:22:50 UTC
Unfortunately, I spoke too soon. There was a bug in my .spec file which means ILP64 wasn't actually used. When I really enable it, I get build failures related to LAPACK functions, which aren't correctly added the 64_ suffix. For example:
$ strings /lib64/libopenblasp64_.so.0 | grep dgeqp3
dgeqp3_
LAPACKE_dgeqp364_
LAPACKE_dgeqp3_work64_
LAPACKE_dgeqp3
LAPACKE_dgeqp3_work

while Julia expects the first result should be dgeqp3_64.

I guess this means we also need a suffixed variant of lapack, right?

Comment 12 Orion Poplawski 2016-01-07 17:25:49 UTC
That's my assumption and why I filed bug #1295965

Comment 13 Orion Poplawski 2016-01-07 20:07:10 UTC
Would switching to the bundled lapack code fix this?  This is allowed now.  Just note Provides: bundled(lapack).

Comment 14 Milan Bouchet-Valat 2016-01-07 21:59:45 UTC
AFAIK it would, as that's why Julia upstream is using. Indeed it might be simpler for everybody. Anyway LAPACK is statically linked so there's no real point in using the libs from the lapack package.

Comment 15 Milan Bouchet-Valat 2016-01-07 22:00:51 UTC
(Oh, and would would also mean we wouldn't need to update RHEL's LAPACK to 3.5 or 3.6.)

Comment 16 Susi Lehtola 2016-01-08 01:16:00 UTC
... not to mention allowing to fix (i.e. circumvent) bug #1176025 and bug #1176026.

Comment 17 Fedora Update System 2016-01-08 03:24:07 UTC
openblas-0.2.15-2.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.

Comment 18 Orion Poplawski 2016-01-08 20:55:45 UTC
Created attachment 1112991 [details]
Updated spec

* Fri Jan 08 2016 Orion Poplawski <orion.com> - 0.2.15-3
- Allow conditional build with or without system lapack

How does this seem?

Comment 19 Susi Lehtola 2016-01-08 21:41:54 UTC
If it builds, it's probably fine.

Comment 20 Milan Bouchet-Valat 2016-01-09 21:43:54 UTC
Fine with me too.

Comment 21 Susi Lehtola 2016-01-09 22:32:51 UTC
Although it should be %bcond_with instead of %bcond_without.

Comment 22 Orion Poplawski 2016-01-10 02:58:04 UTC
Right, I figured I'd start with a drop in replacement for the current package and change to %bcond_without where appropriate (everywhere?).

Comment 23 Fedora Update System 2016-01-13 22:15:22 UTC
openblas-0.2.15-5.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2016-9156b5210b

Comment 24 Fedora Update System 2016-01-13 22:15:25 UTC
openblas-0.2.15-5.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-914172f10d

Comment 25 Fedora Update System 2016-01-13 22:15:31 UTC
openblas-0.2.15-5.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2016-49d61ed4db

Comment 26 Fedora Update System 2016-01-13 22:15:34 UTC
openblas-0.2.15-5.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2016-572e2e6c2b

Comment 27 Fedora Update System 2016-01-14 10:21:43 UTC
openblas-0.2.15-5.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2016-9156b5210b

Comment 28 Fedora Update System 2016-01-14 10:24:45 UTC
openblas-0.2.15-5.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2016-49d61ed4db

Comment 29 Fedora Update System 2016-01-14 11:24:44 UTC
openblas-0.2.15-5.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-914172f10d

Comment 30 Fedora Update System 2016-01-14 14:46:22 UTC
openblas-0.2.15-5.el5 has been pushed to the Fedora EPEL 5 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2016-2cabb19b8b

Comment 31 Fedora Update System 2016-01-15 19:53:14 UTC
openblas-0.2.15-5.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-572e2e6c2b

Comment 32 Milan Bouchet-Valat 2016-01-16 16:32:54 UTC
Orion: why do you set NO_LAPACK only when lapacke == 0? AFAICT this means that the ILP64 LAPACK interface is never built on Fedora, which defeats the whole goal. Anyway, OpenBLAS includes LAPACKE, so we can set NO_LAPACK=1 everywhere.

I think this explains why I'm getting crashes when using the Fedora ILP64 OpenBLAS that I'm not seeing with Julia's. I'm currently building a modified package to check this.

Comment 33 Orion Poplawski 2016-01-16 16:40:54 UTC
My reading of the makefile is that it only tests for setting NO_LAPACK, not the value, so that NO_LAPACK=1 and NO_LAPACK=0 have the same effect.  But I may made a mistake or there may be some other issue.

Comment 34 Milan Bouchet-Valat 2016-01-16 17:15:57 UTC
Ah, sorry, I confused NO_LAPACK with NO_LAPACKE. So it looks OK indeed.

I really don't know what's causing the Fedora ILP64 OpenBLAS 0.2.15-5 not to work correctly . It fails both with Julia and with ARPACK. I have a very simple test case in Julia:
$ LD_PRELOAD=/lib64/libopenblas64_.so.0 ./julia 
julia> n = 10
       elty = Float32
       A = triu(rand(elty,n,n))
       x = rand(elty,n)
       A\x
ERROR: Base.LinAlg.LAPACKException(140144782868480)
 in chklapackerror at linalg/lapack.jl:38
 in trtrs! at linalg/lapack.jl:3285
 [inlined code] from linalg/triangular.jl:396
 in \ at linalg/generic.jl:364
 in eval at ./boot.jl:265

The very high value of the error code seems to indicate a size mismatch to me.

Any ideas?

Comment 35 Milan Bouchet-Valat 2016-01-17 17:25:50 UTC
I've tried enabling the tests in a build, and they fail. Maybe not related, though:
https://kojipkgs.fedoraproject.org//work/tasks/6675/12586675/build.log

(Note that I've only enabled the 64-bit serial builds.)

Comment 36 Fedora Update System 2016-01-24 03:18:05 UTC
openblas-0.2.15-5.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.

Comment 37 Fedora Update System 2016-01-24 03:33:16 UTC
openblas-0.2.15-5.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.

Comment 38 Milan Bouchet-Valat 2016-01-28 11:13:22 UTC
Actually, the crashes I saw locally also appear when running the Julia tests:
https://kojipkgs.fedoraproject.org//work/tasks/1902/12701902/build.log

I guess segfaults can indicate either a 32/64 bits mix, or a compilation issue (flags/compiler not tested by upstream).

Comment 39 Susi Lehtola 2016-01-28 18:19:03 UTC
(In reply to Milan Bouchet-Valat from comment #38)
> Actually, the crashes I saw locally also appear when running the Julia tests:
> https://kojipkgs.fedoraproject.org//work/tasks/1902/12701902/build.log
> 
> I guess segfaults can indicate either a 32/64 bits mix, or a compilation
> issue (flags/compiler not tested by upstream).

But this is not an OpenBLAS bug. Look at root.log:

DEBUG util.py:399:   atlas                    x86_64  3.10.2-11.fc24                   build  6.0 M

DEBUG util.py:399:   openblas-threads64_      x86_64  0.2.15-5.fc24                    build  4.0 M

You seem to be mixing libraries, which will of course give you unpredictable behavior.

It's also very odd you don't install any of the -devel packages.

Comment 40 Milan Bouchet-Valat 2016-01-28 18:44:12 UTC
(In reply to Susi Lehtola from comment #39)
> But this is not an OpenBLAS bug. Look at root.log:
> 
> DEBUG util.py:399:   atlas                    x86_64  3.10.2-11.fc24        
> build  6.0 M
> 
> DEBUG util.py:399:   openblas-threads64_      x86_64  0.2.15-5.fc24         
> build  4.0 M
> 
> You seem to be mixing libraries, which will of course give you unpredictable
> behavior.
Well, the whole point of adding a suffix was to avoid conflicts between libraries, so I'm not sure why this would cause crashes. Though of course I agree that atlas shouldn't be installed (it's a dependency of SuiteSparse currently).

> It's also very odd you don't install any of the -devel packages.
Actually, I don't need them since Julia loads OpenBLAS dynamically, using the full SONAME. Else, it would have failed with another error.


That said, I now see the failure is logical since Julia expects SuiteSparse and ARPACK to use the same integer size as BLAS. So I'll retry when these will have been moved to ILP64 too. (Some of the errors I reported above seem harder to explain, though, but I'll confirm that later.)

Comment 41 Fedora Update System 2016-01-29 03:22:09 UTC
openblas-0.2.15-5.el5 has been pushed to the Fedora EPEL 5 stable repository. If problems still persist, please make note of it in this bug report.

Comment 42 Fedora Update System 2016-01-29 06:00:47 UTC
openblas-0.2.15-5.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.

Comment 43 Fedora Update System 2016-02-01 01:53:56 UTC
openblas-0.2.15-5.el6 has been pushed to the Fedora EPEL 6 stable repository. If problems still persist, please make note of it in this bug report.