Description of problem: I ran the following python program: import numpy import scipy.linalg scipy.linalg.svdvals(1j * numpy.random.random((3, 4))) Note: the size of the array is significant, and size (4, 3) doesn't cause a crash. Version-Release number of selected component: python-2.7.3-13.fc18 Additional info: backtrace_rating: 4 cmdline: python bug.py crash_function: array_dealloc executable: /usr/bin/python2.7 kernel: 3.7.2-204.fc18.x86_64 remote_result: NOTFOUND uid: 1000 Truncated backtrace: Thread no. 1 (10 frames) #5 array_dealloc at numpy/core/src/multiarray/arrayobject.c:267 #6 f2py_rout_flapack_zgesdd at build/src.linux-x86_64-2.7/build/src.linux-x86_64-2.7/scipy/linalg/flapackmodule.c:10415 #7 PyObject_Call at /usr/src/debug/Python-2.7.3/Objects/abstract.c:2529 #8 do_call at /usr/src/debug/Python-2.7.3/Python/ceval.c:4316 #9 call_function at /usr/src/debug/Python-2.7.3/Python/ceval.c:4121 #10 PyEval_EvalFrameEx at /usr/src/debug/Python-2.7.3/Python/ceval.c:2740 #11 PyEval_EvalCodeEx at /usr/src/debug/Python-2.7.3/Python/ceval.c:3330 #12 fast_function at /usr/src/debug/Python-2.7.3/Python/ceval.c:4194 #13 call_function at /usr/src/debug/Python-2.7.3/Python/ceval.c:4119 #14 PyEval_EvalFrameEx at /usr/src/debug/Python-2.7.3/Python/ceval.c:2740
Created attachment 689469 [details] File: backtrace
Created attachment 689470 [details] File: build_ids
Created attachment 689471 [details] File: cgroup
Created attachment 689472 [details] File: core_backtrace
Created attachment 689473 [details] File: dso_list
Created attachment 689474 [details] File: environ
Created attachment 689475 [details] File: limits
Created attachment 689476 [details] File: maps
Created attachment 689477 [details] File: open_fds
Created attachment 689478 [details] File: proc_pid_status
Created attachment 689479 [details] File: smolt_data
Created attachment 689480 [details] File: var_log_messages
This package has changed ownership in the Fedora Package Database. Reassigning to the new owner of this component.
This message is a reminder that Fedora 18 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 18. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '18'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 18's end of life. Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 18 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior to Fedora 18's end of life. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
The problem persists in F20.
This seems to be a problem in numpy, not Python. Reassigning.
(In reply to Dan Stahlke from comment #17) > The problem persists in F20. I cannot reproduce it on F21, which should have the same version of numpy.
May be an atlas issue - valgrind reports: >>> scipy.linalg.svdvals(1j * numpy.random.random((3, 4))) ==9314== Invalid write of size 8 ==9314== at 0x115922D0: dlasq2_ (in /usr/lib64/atlas/liblapack.so.3.0) ==9314== by 0x11591B8B: dlasq1_ (in /usr/lib64/atlas/liblapack.so.3.0) ==9314== by 0x114DFDB7: dbdsqr_ (in /usr/lib64/atlas/liblapack.so.3.0) ==9314== by 0x11591146: dlasdq_ (in /usr/lib64/atlas/liblapack.so.3.0) ==9314== by 0x114DC9E1: dbdsdc_ (in /usr/lib64/atlas/liblapack.so.3.0) ==9314== by 0x1179FF7D: zgesdd_ (in /usr/lib64/atlas/liblapack.so.3.0) ==9314== by 0x154AB279: ??? (in /usr/lib64/python2.7/site-packages/scipy/linalg/_flapack.so) ==9314== by 0x4E7B0D2: PyObject_Call (in /usr/lib64/libpython2.7.so.1.0) ==9314== by 0x4F0F37B: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0) ==9314== by 0x4F131DC: PyEval_EvalCodeEx (in /usr/lib64/libpython2.7.so.1.0) ==9314== by 0x4F1188E: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0) ==9314== by 0x4F131DC: PyEval_EvalCodeEx (in /usr/lib64/libpython2.7.so.1.0) ==9314== Address 0x10894b40 is not stack'd, malloc'd or (recently) free'd ..... atlas-3.8.4-14.fc20.x86_64 vs atlas-3.10.1-18.fc21.x86_64
The problem seems to come from lapack bundled in atlas. I have a short reproducer in C that causes the same valgrind error than scipy. Does the same with both lapack or atlas. I am not a fortran expert, I can not guarantee its correctness but it is definitely easier to handle than scipy (Python -> C module machine-generated by f2py loaded into python -> fortran interface of atlas -> core of atlas compiled from machine-generated C, very funny. Attempts to debug it entertained a group of engineers one whole afternoon. Happy April Fools' Day.) #include <stdio.h> #include <stdlib.h> typedef struct {double r,i;} complex_double; //typedef taken from scipy typedef int fint; extern void zgesdd_(char*,int*,int*,complex_double*,int*,double*,complex_double*,int*,complex_double*,int*,complex_double*,int*,double*,int*,int*); //function prototype taken from scipy, seems sane when compared with manual int main() { int i, j; fint m=3; fint n=4; //constants from zgesdd manual fint minmn=m<n?m:n; fint maxmn=m<n?n:m; fint lwork=57; //scipy uses 57, manual says >= 2*min(M,N)+max(M,N) fint lrwork = 5*minmn; fint liwork = 8*minmn; fint info = 0; fint unused = 1; complex_double *a=malloc(n*m*sizeof(complex_double)); double *s=malloc(minmn*sizeof(double)); complex_double *work=malloc(lwork*sizeof(complex_double)); double *rwork=malloc(lrwork*sizeof(double)); fint *iwork=malloc(liwork*sizeof(int)); for (i=0; i<m; ++i) { //maybe columns/rows should be accessed differently but it does not matter now for (j=0; j<n; ++j) { a[i*n+j].r=0; a[i*n+j].i=(double)rand()/rand(); } } zgesdd_("N", &m , &n , a , &m, s, NULL, &unused, NULL, &unused, work, &lwork, rwork, iwork, &info); free(a); free(s); free(work); free(rwork); free(iwork); return 0; }
Strangely, the same in fortran works without valgrind warnings. Has fortran different memory management? program matrix external zgesdd parameter (m=4) parameter (n=3) parameter (ldu=1) parameter (lwork=57) parameter (liwork=8*min(M,N)) parameter (lrwork=5*min(m,n)) c complex*16 a(m, n) complex*16, allocatable :: a(:,:) c double precision s(min(m, n)) double precision, allocatable :: s(:) c complex*16 work(max(1, lwork)) complex*16, allocatable :: work(:) complex*16 unused(1,1) complex*16 constant integer i, j integer info c double precision rwork(max(1, lrwork)) double precision, allocatable :: rwork(:) c integer iwork(liwork) integer, allocatable :: iwork(:) allocate(a(m, n)) allocate(s(min(m, n))) allocate(work(max(1, lwork))) allocate(rwork(max(1, lrwork))) allocate(iwork(liwork)) constant = dcmplx(dble(0), dble(0.5)) do i = 1, m do j = 1, n a(i, j) = constant end do end do info = 0 ws = worksize do i= 1, n a(i, i) = i end do call zgesdd( $"N", m, n, a, m, s, unused, ldu, unused, ldu, work, lwork, rwork, $iwork, info) do i = 1, m do j = 1, n write (*,*) "i: ", i, "j: ", j, "a: ", a(i,j) end do end do do i = 1, min(m,n) write (*,*) "i: ", i, "s: ", s(i) end do write (*,*) 'Info: ', info deallocate(a) deallocate(s) deallocate(work) deallocate(rwork) stop end
Fortran has different array ordering so to make your C equivalent to your fortran, you need: zgesdd_("N", &n , &m , a , &n, s, NULL, &unused, NULL, &unused, work, &lwork, rwork, iwork, &info);
Yes, thanks, in pure fortran it fails in the same way.
http://icl.utk.edu/lapack-forum/viewtopic.php?f=13&t=4687&sid=20893b804ebf7bbe783c557a18951fc2
The LDA argument to zgesdd (5th argument) has to be the same as M, the (2nd argument) or the memory arrangement is all wrong and a crash would be expected. But that's not how scipy is calling zgesdd_: Breakpoint 2, 0x00007fffec766050 in zgesdd_ () from /usr/lib64/atlas/liblapack.so.3 (gdb) up #1 0x00007fffe9e1d27a in f2py_rout__flapack_zgesdd (capi_self=<optimized out>, capi_args=<optimized out>, capi_keywds=<optimized out>, f2py_func=0x7fffec766050 <zgesdd_>) at build/src.linux-x86_64-2.7/build/src.linux-x86_64-2.7/scipy/linalg/_flapackmodule.c:10348 10348 (*f2py_func)((compute_uv?(full_matrices?"A":"S"):"N"),&m,&n,a,&m,s,u,&u0,vt,&vt0,work,&lwork,rwork,iwork,&info); (gdb) list 10343 /*end of frompyobj*/ 10344 #ifdef F2PY_REPORT_ATEXIT 10345 f2py_start_call_clock(); 10346 #endif 10347 /*callfortranroutine*/ 10348 (*f2py_func)((compute_uv?(full_matrices?"A":"S"):"N"),&m,&n,a,&m,s,u,&u0,vt,&vt0,work,&lwork,rwork,iwork,&info); 10349 /*(*f2py_func)(&m,&n,&minmn,&u0,&u1,&vt0,&vt1,a,&compute_uv,&full_matrices,u,s,vt,work,rwork,&lwork,iwork,&info);*/ 10350 if (PyErr_Occurred()) 10351 f2py_success = 0; 10352 #ifdef F2PY_REPORT_ATEXIT (gdb) print m $1 = 3 (gdb) print n $2 = 4 (gdb) print minmn $3 = <optimized out> (gdb) print a $4 = (complex_double *) 0xcd7c40 (gdb) print s $5 = (double *) 0xcb7e40 (gdb) print u $6 = (complex_double *) 0xd37440 (gdb) print u0 $7 = 1 (gdb) print vt $8 = (complex_double *) 0xc1e6e0 (gdb) print vt0 $9 = 1 (gdb) print work $10 = <optimized out> (gdb) print lwork $11 = 57 (gdb) print iwork $12 = (int *) 0xd32760 (gdb) print info $13 = 0 We really need to figure out the malloc'd sizes of the arrays. The fact that this is fixed with a newer atlas, still makes me think something is up with it. Harder to track the lapack code in atlas as we loose the debug information.
This message is a reminder that Fedora 20 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 20. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '20'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 20 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 20 changed to end-of-life (EOL) status on 2015-06-23. Fedora 20 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.