Bug 1181369 - Packaging violation: bundled LAPACK in ATLAS
Summary: Packaging violation: bundled LAPACK in ATLAS
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: atlas
Version: 21
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Frantisek Kluknavsky
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: 1149032
TreeView+ depends on / blocked
 
Reported: 2015-01-13 02:22 UTC by Susi Lehtola
Modified: 2015-02-19 02:59 UTC (History)
3 users (show)

Fixed In Version: atlas-3.10.1-18.fc21
Clone Of:
Environment:
Last Closed: 2015-02-07 06:30:33 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
Patch against rawhide spec for static linkage (2.61 KB, patch)
2015-01-30 23:54 UTC, Susi Lehtola
no flags Details | Diff

Description Susi Lehtola 2015-01-13 02:22:58 UTC
Starting from Fedora 21, ATLAS uses a bundled LAPACK. This is unacceptable, as there is no reason to use the system version of LAPACK like OpenBLAS does.

The ATLAS build system should be patched to be able to use the system version of LAPACK.

Comment 1 Susi Lehtola 2015-01-14 20:44:27 UTC
Also, since I couldn't find an approval for bundling by the FPC, this is a packaging violation.

Comment 2 Susi Lehtola 2015-01-19 01:47:04 UTC
Looks like you should be able to adopt Debian's patches

https://packages.debian.org/source/sid/atlas

Comment 3 Frantisek Kluknavsky 2015-01-23 23:24:31 UTC
http://koji.fedoraproject.org/koji/taskinfo?taskID=8711138

Comment 4 Susi Lehtola 2015-01-23 23:34:51 UTC
Thanks for looking into this.

You might want to consider linking against the necessary LAPACK routines statically, though..

Comment 5 Susi Lehtola 2015-01-23 23:40:02 UTC
Especially since now linking against the library doesn't work

$ g++ -O2 -Wall lapack_check.cpp -L/usr/lib64/atlas -lsatlas
/usr/bin/ld: /tmp/cco97X0I.o: undefined reference to symbol 'dsyev_'
/lib64/liblapack.so.3: error adding symbols: DSO missing from command line

and then again liblapack is linked against libblas that contains the same BLAS symbols as atlas.

Comment 6 Frantisek Kluknavsky 2015-01-24 00:26:12 UTC
Linking works for me. Lapack is unbundled now, you need lapack-devel installed, dsyev_ is in liblapack.so.

If I understand things correctly, those optimized lapack routines shipped with atlas are linked with atlas statically. The whole system lapack library is linked dynamically. I hope I got the order correctly and optimized routines are called (seems so after a quick test but I am not completely sure).

I am glad for any suggestions how to do this properly, but linking complete lapack statically from lapack-static package is still bundling forbidden in Fedora.

Comment 7 Susi Lehtola 2015-01-24 04:47:07 UTC
(In reply to Frantisek Kluknavsky from comment #6)
> I am glad for any suggestions how to do this properly, but linking complete
> lapack statically from lapack-static package is still bundling forbidden in
> Fedora.

No, static linkage against system libraries is allowed in rare circumstances as here.

Comment 8 Susi Lehtola 2015-01-24 04:51:55 UTC
(In reply to Frantisek Kluknavsky from comment #6)
> Linking works for me. Lapack is unbundled now, you need lapack-devel
> installed, dsyev_ is in liblapack.so.

Yeah but since the libraries are supposed to be bundled, the current package breaks compatibility.

And, if I have understood correctly, since liblapack.so is linked to reference libblas.so, calling dsyev may actually call reference BLAS instead of the optimized version in ATLAS.

The way this has been done earlier is that only the LAPACK routines that aren't implemented by ATLAS (or OpenBLAS) are included from the static library to form the final target library. That way you only need to link to a single library, and won't have any problems with clashing symbols and unpredictable behavior...

Comment 9 Frantisek Kluknavsky 2015-01-28 16:46:49 UTC
For me, dsyev of a 1000x1000 matrix
[1 0 0 0 0 ...]
[0 2 0 0 0 ...]
[0 0 3 0 0 ...]
[0 0 0 4 0 ...]
[0 0 0 0 5 ...]
...
takes 0.75s with lapack, 0.45s with atlas-18 and lapack, 0.45s with atlas-17 (all libraries built locally in mock, atlas from koji is worse than blas sometimes) - dsyev from lapack calls the optimized blas functions from atlas. 
Compatibility with existing binaries is not broken. Build requirements are broken, I updated pkgconfig file and added lapack-devel to dependencies of atlas-devel.
I can not see the advantage of linking against lapack-static at the expense of fragile downstream patch, compared to default upstream method of recompiling lapack from source. I is still bundled, any fix in lapack still requires a rebuild of atlas.
Please watch for breakage or performance regression. I gave it 2 days of testing but one never knows.

Comment 10 Susi Lehtola 2015-01-30 23:54:20 UTC
(In reply to Frantisek Kluknavsky from comment #9)
> I can not see the advantage of linking against lapack-static at the expense
> of fragile downstream patch, compared to default upstream method of
> recompiling lapack from source. I is still bundled, any fix in lapack still
> requires a rebuild of atlas.
> Please watch for breakage or performance regression. I gave it 2 days of
> testing but one never knows.

Well, I think the easier linkage and the avoidance of possible symbol clashes (liblapack.so linked to refence BLAS, also liblapack contains LAPACK routines for which ATLAS optimized versions are available) more than outweighs the bad sides of static linkage; namely, the need to rebuild ATLAS whenever the LAPACK package is (substantially) updated.

Switching to a static link is actually extremely easy; the only thing necessary is to switch from liblapack.so to liblapack_pic.a. Then, upon compilation you get a list of duplicate symbols. Then you just remove these and it works. I've prepared a patch for static linkage (attached).

Comment 11 Susi Lehtola 2015-01-30 23:54:54 UTC
Created attachment 986330 [details]
Patch against rawhide spec for static linkage

Comment 12 Susi Lehtola 2015-02-02 22:25:49 UTC
Would you be OK with this?

Comment 13 Susi Lehtola 2015-02-07 06:30:33 UTC
Since you haven't replied, I assume you don't have a problem with this. If the link breaks in a new release, I can chip in to help.

Comment 14 Fedora Update System 2015-02-08 08:39:19 UTC
atlas-3.10.1-18.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/atlas-3.10.1-18.fc21

Comment 15 Fedora Update System 2015-02-19 02:59:34 UTC
atlas-3.10.1-18.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.