Bug 489887 - Latest scipy fails to load some routines
Latest scipy fails to load some routines
Status: CLOSED NOTABUG
Product: Fedora
Classification: Fedora
Component: scipy (Show other bugs)
10
All Linux
low Severity medium
: ---
: ---
Assigned To: Jef Spaleta
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-03-12 09:48 EDT by David Jansen
Modified: 2009-03-25 05:30 EDT (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-03-25 05:30:13 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description David Jansen 2009-03-12 09:48:38 EDT
Description of problem:

After the update of scipy to version 0.7.0-0.1.b1, some python scripts in use here at our institute, seem to be broken.

Version-Release number of selected component (if applicable):
scipy-0.7.0-0.1.b1 (i386 & x86_64, perhaps others too)

How reproducible:
I could simplify the invokation that fails to simply this:
$ python
Python 2.5.2 (r252:60911, Sep 30 2008, 15:42:03) 
[GCC 4.3.2 20080917 (Red Hat 4.3.2-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import scipy.interpolate
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.5/site-packages/scipy/interpolate/__init__.py", line 13, in <module>
    from rbf import Rbf
  File "/usr/lib64/python2.5/site-packages/scipy/interpolate/rbf.py", line 47, in <module>
    from scipy import linalg
  File "/usr/lib64/python2.5/site-packages/scipy/linalg/__init__.py", line 8, in <module>
    from basic import *
  File "/usr/lib64/python2.5/site-packages/scipy/linalg/basic.py", line 17, in <module>
    from lapack import get_lapack_funcs
  File "/usr/lib64/python2.5/site-packages/scipy/linalg/lapack.py", line 18, in <module>
    from scipy.linalg import clapack
ImportError: /usr/lib64/python2.5/site-packages/scipy/linalg/clapack.so: undefined symbol: clapack_sgesv
>>> 

Reinstalling scipy-0.6.0-7.fc10 makes it all work again. Could it be that the new scipy was built against a different version of lapack that hasn't shown up in the updates yet?
I have this suspicion because ldd reports quite a different set of linked libraries:
old version: $ ldd /usr/lib/python2.5/site-packages/scipy/linalg/clapack.so
	linux-gate.so.1 =>  (0x00481000)
	liblapack.so.3 => /usr/lib/liblapack.so.3 (0x00a5d000)
	libblas.so.3 => /usr/lib/libblas.so.3 (0x00724000)
	libpython2.5.so.1.0 => /usr/lib/libpython2.5.so.1.0 (0x0049e000)
	libgfortran.so.3 => /usr/lib/libgfortran.so.3 (0x00296000)
	libm.so.6 => /lib/libm.so.6 (0x00110000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00139000)
	libc.so.6 => /lib/libc.so.6 (0x0077d000)
	libpthread.so.0 => /lib/libpthread.so.0 (0x00147000)
	libdl.so.2 => /lib/libdl.so.2 (0x00161000)
	libutil.so.1 => /lib/libutil.so.1 (0x00166000)
	/lib/ld-linux.so.2 (0x0060a000)
new version: $ ldd /usr/lib/python2.5/site-packages/scipy/linalg/clapack.so 
	linux-gate.so.1 =>  (0x00110000)
	liblapack.so.3 => /usr/lib/liblapack.so.3 (0x0012a000)
	libf77blas.so.3 => /usr/lib/atlas/libf77blas.so.3 (0x00633000)
	libcblas.so.3 => /usr/lib/atlas/libcblas.so.3 (0x0064f000)
	libatlas.so.3 => /usr/lib/atlas/libatlas.so.3 (0x0066f000)
	libpython2.5.so.1.0 => /usr/lib/libpython2.5.so.1.0 (0x009b1000)
	libgfortran.so.3 => /usr/lib/libgfortran.so.3 (0x00b02000)
	libm.so.6 => /lib/libm.so.6 (0x00bbc000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00be5000)
	libc.so.6 => /lib/libc.so.6 (0x00bf3000)
	libblas.so.3 => /usr/lib/libblas.so.3 (0x00d67000)
	libpthread.so.0 => /lib/libpthread.so.0 (0x00dc0000)
	libdl.so.2 => /lib/libdl.so.2 (0x00dda000)
	libutil.so.1 => /lib/libutil.so.1 (0x00ddf000)
	/lib/ld-linux.so.2 (0xb8012000)
Comment 1 Jef Spaleta 2009-03-13 15:11:48 EDT
Hmm, that is very strange and completely unexpected.

Before I delve into this deeper, can see if this continues to happen with the scipy-0.7.0-2.fc10 package available in updates-testing?

If that package fixes the problem I'm willing to move it to stable.

-jef
Comment 2 David Jansen 2009-03-16 09:29:24 EDT
Same problem with the version from updates-testing:
$ rpm -q scipy
scipy-0.7.0-2.fc10.x86_64
$ python                                                     
Python 2.5.2 (r252:60911, Sep 30 2008, 15:42:03) 
[GCC 4.3.2 20080917 (Red Hat 4.3.2-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import scipy.interpolate
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.5/site-packages/scipy/interpolate/__init__.py", line 13, in <module>
    from rbf import Rbf
  File "/usr/lib64/python2.5/site-packages/scipy/interpolate/rbf.py", line 47, in <module>
    from scipy import linalg
  File "/usr/lib64/python2.5/site-packages/scipy/linalg/__init__.py", line 8, in <module>
    from basic import *
  File "/usr/lib64/python2.5/site-packages/scipy/linalg/basic.py", line 17, in <module>
    from lapack import get_lapack_funcs
  File "/usr/lib64/python2.5/site-packages/scipy/linalg/lapack.py", line 18, in <module>
    from scipy.linalg import clapack
ImportError: /usr/lib64/python2.5/site-packages/scipy/linalg/clapack.so: undefined symbol: clapack_sgesv
>>>
Comment 3 Jef Spaleta 2009-03-16 12:04:42 EDT
Hmm.  I can't reproduce this locally on my 32bit system. I'll see if other people can confirm it.

But in the meantime could you get me the ldd output for clapack.so on your system? 
Here's mine.

ldd /usr/lib/python2.5/site-packages/scipy/linalg/clapack.so
	linux-gate.so.1 =>  (0x00a39000)
	liblapack.so.3 => /usr/lib/atlas/liblapack.so.3 (0x00110000)
	libf77blas.so.3 => /usr/lib/atlas/libf77blas.so.3 (0x006e2000)
	libcblas.so.3 => /usr/lib/atlas/libcblas.so.3 (0x00712000)
	libatlas.so.3 => /usr/lib/atlas/libatlas.so.3 (0x00a3a000)
	libpython2.5.so.1.0 => /usr/lib/libpython2.5.so.1.0 (0x00732000)
	libgfortran.so.3 => /usr/lib/libgfortran.so.3 (0x00e4a000)
	libm.so.6 => /lib/libm.so.6 (0x0064c000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00675000)
	libc.so.6 => /lib/libc.so.6 (0x008b5000)
	libblas.so.3 => /usr/lib/atlas/libblas.so.3 (0x00f04000)
	libpthread.so.0 => /lib/libpthread.so.0 (0x00683000)
	libdl.so.2 => /lib/libdl.so.2 (0x0069d000)
	libutil.so.1 => /lib/libutil.so.1 (0x006a2000)
	/lib/ld-linux.so.2 (0x00892000)

I want to make sure you've got the right atlas package installed.

rpm -qf /usr/lib/atlas/liblapack.so.3 
For me that returns atlas-3.6.0-15.fc10.i386


-jef
Comment 4 Jason Farrell 2009-03-16 12:07:55 EDT
I'm up-to-date with updates-testing and I can't reproduce:

[jaf@nano:1j:178h:0e:0.34MB stuff]$ rpm -q python scipy
python-2.5.2-1.fc10.x86_64
scipy-0.7.0-2.fc10.x86_64
[jaf@nano:1j:179h:0e:0.34MB stuff]$ python
Python 2.5.2 (r252:60911, Sep 30 2008, 15:42:03)
[GCC 4.3.2 20080917 (Red Hat 4.3.2-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import scipy.interpolate
>>> print 'all fine here'
all fine here
>>>
Comment 5 David Jansen 2009-03-16 12:23:52 EDT
On my x86_64 machine where I have updates-testing enabled:

$ ldd /usr/lib64/python2.5/site-packages/scipy/linalg/clapack.so
	linux-vdso.so.1 =>  (0x00007fff375ff000)
	liblapack.so.3 => /usr/lib64/liblapack.so.3 (0x0000000000548000)
	libf77blas.so.3 => /usr/lib64/atlas/libf77blas.so.3 (0x000000000429e000)
	libcblas.so.3 => /usr/lib64/atlas/libcblas.so.3 (0x000000000103a000)
	libatlas.so.3 => /usr/lib64/atlas/libatlas.so.3 (0x0000000002e49000)
	libpython2.5.so.1.0 => /usr/lib64/libpython2.5.so.1.0 (0x0000000006436000)
	libgfortran.so.3 => /usr/lib64/libgfortran.so.3 (0x0000000005513000)
	libm.so.6 => /lib64/libm.so.6 (0x0000000004cc6000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000000000110000)
	libc.so.6 => /lib64/libc.so.6 (0x0000000001df1000)
	libblas.so.3 => /usr/lib64/libblas.so.3 (0x0000000001257000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x0000000005fbf000)
	libdl.so.2 => /lib64/libdl.so.2 (0x0000000005cbf000)
	libutil.so.1 => /lib64/libutil.so.1 (0x00000000014af000)
	/lib64/ld-linux-x86-64.so.2 (0x0000000000e19000)

Just to be sure, I have now installed the version from updates-testing also on the 32-bit machine where the problem was first encountered:
$ ldd /usr/lib/python2.5/site-packages/scipy/linalg/clapack.so
	linux-gate.so.1 =>  (0x00db8000)
	liblapack.so.3 => /usr/lib/liblapack.so.3 (0x00440000)
	libf77blas.so.3 => /usr/lib/atlas/libf77blas.so.3 (0x00e45000)
	libcblas.so.3 => /usr/lib/atlas/libcblas.so.3 (0x00a72000)
	libatlas.so.3 => /usr/lib/atlas/libatlas.so.3 (0x00e61000)
	libpython2.5.so.1.0 => /usr/lib/libpython2.5.so.1.0 (0x00110000)
	libgfortran.so.3 => /usr/lib/libgfortran.so.3 (0x00302000)
	libm.so.6 => /lib/libm.so.6 (0x00261000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x0028a000)
	libc.so.6 => /lib/libc.so.6 (0x00a92000)
	libblas.so.3 => /usr/lib/libblas.so.3 (0x009da000)
	libpthread.so.0 => /lib/libpthread.so.0 (0x00e0e000)
	libdl.so.2 => /lib/libdl.so.2 (0x00c40000)
	libutil.so.1 => /lib/libutil.so.1 (0x00c2a000)
	/lib/ld-linux.so.2 (0x0041d000)

That all looks very similar to your output, as far as I can see.
Comment 6 Jef Spaleta 2009-03-16 14:28:47 EDT
You missed a key difference!

Your system isn't using the lapack.so.3 included from atlas!

My system is using  liblapack.so.3 => /usr/lib/atlas/liblapack.so.3
Your systems are using /usr/lib/liblapack.so.3

See the difference?

scipy is compiled against atlas.

By default on Fedora clients the linker is configured so that atlas is checked first. The atlas package drops in a configuration file in /etc/ld.so.conf.d/
which tells the linker to prefer the atlas directory over the main system directory.  So on my system with both /usr/lib/liblapack.so.3
and /usr/lib/atlas/liblapack.so.3 available the linker prefers the atlas provided liblapack.so.3 

If you are using custom LD_LIBRARY_PATH environment variables or custom configuration in /etc/ld.so.conf.d/ you are going to have problems.

Please check to make sure you haven't made any localized linker configuration changes that are influencing the order of library location discovery on your system.

-jef
Comment 7 David Jansen 2009-03-25 05:30:13 EDT
Correct, we had an LD_LIBRARY_PATH set to get some other software to work. Without it, the scipy package works fine, so this is not a bug in the package after all. I'll consider this issue closed and I will figure out what to do to get the other (non-Fedora) applications to work without an LD_LIBRARY_PATH

Note You need to log in before you can comment on or make changes to this bug.