Bug 1030830 - Can't import numpy in python-debug interpreter: ImportError: No module named multiarray
Can't import numpy in python-debug interpreter: ImportError: No module named ...
Status: CLOSED EOL
Product: Fedora
Classification: Fedora
Component: python (Show other bugs)
20
x86_64 Linux
unspecified Severity medium
: ---
: ---
Assigned To: Matej Stuchlik
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-11-15 04:24 EST by Miha
Modified: 2016-01-31 21:15 EST (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-06-29 08:56:01 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Miha 2013-11-15 04:24:03 EST
User-Agent:       Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0
Build Identifier: 

When I'm using python-debug, I can't import numpy. If I use ordinary python interpreter, there is no problem. They both have the same sys.path. 

This problem happens both in Fedora 19 and in up to date vanilla (i.e. no additional packages installed except python-debug and numpy) Fedora 20. Here is the output from Fedora 20:

[miha@localhost ~]$ python-debug
Python 2.7.5 (default, Nov 12 2013, 16:45:15) 
[GCC 4.8.2 20131017 (Red Hat 4.8.2-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.7/site-packages/numpy/__init__.py", line 153, in <module>
    from . import add_newdocs
  File "/usr/lib64/python2.7/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/usr/lib64/python2.7/site-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/usr/lib64/python2.7/site-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/usr/lib64/python2.7/site-packages/numpy/core/__init__.py", line 6, in <module>
    from . import multiarray
ImportError: cannot import name multiarray
[70322 refs]
>>> 


Reproducible: Always
Comment 1 Bohuslav "Slavek" Kabrda 2013-11-18 05:16:57 EST
The reason why this fails is, that the debug build dynload_shlib always looks for module_d.so, instead of just module.so - and there is no multiarray_d.so.

The patch that helps us build the python-debug version comes from Debian and has been adapted by Dave Malcolm in the way to _not_ fall back to multiarray.so, unlike the original Debian patch. According to a comment in python.spec:

#   * Debian's patch to dynload_shlib.c looks for module_d.so, then module.so,
# but this can potentially find a module built against the wrong DSO ABI.  We
# instead search for just module_d.so in a debug build

I don't totally understand this - Dave, could you please explain what you mean by "wrong DSO ABI"?  Does that mean that you wanted to make this bulletproof against cases where Python 2.6 "module.so" would be present and Python 2.7 would try to load "module_d.so"? Some different concerns?
Comment 2 Miha 2013-11-18 07:15:45 EST
Thanks for the response! Do you know of any workaround (hack) I can use to make this work for me?

I tried to copy multiarray.so to multiarray_d.so, but that also fails:

[miha@localhost ~]$ sudo cp /usr/lib64/python2.7/site-packages/numpy/core/multiarray.so /usr/lib64/python2.7/site-packages/numpy/core/multiarray_d.so
[miha@localhost ~]$ python-debug
Python 2.7.5 (default, Nov 12 2013, 16:45:15) 
[GCC 4.8.2 20131017 (Red Hat 4.8.2-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
python-debug: /builddir/build/BUILD/Python-2.7.5/Objects/object.c:65: _Py_AddToAllObjects: Assertion `(op->_ob_prev == ((void *)0)) == (op->_ob_next == ((void *)0))' failed.
Aborted (core dumped)
Comment 3 Dave Malcolm 2013-11-18 11:07:21 EST
(In reply to Bohuslav "Slavek" Kabrda from comment #1)
> The reason why this fails is, that the debug build dynload_shlib always
> looks for module_d.so, instead of just module.so - and there is no
> multiarray_d.so.
> 
> The patch that helps us build the python-debug version comes from Debian and
> has been adapted by Dave Malcolm in the way to _not_ fall back to
> multiarray.so, unlike the original Debian patch. According to a comment in
> python.spec:
> 
> #   * Debian's patch to dynload_shlib.c looks for module_d.so, then
> module.so,
> # but this can potentially find a module built against the wrong DSO ABI.  We
> # instead search for just module_d.so in a debug build
> 
> I don't totally understand this - Dave, could you please explain what you
> mean by "wrong DSO ABI"? 

The layout of the PyObject struct is different between the optimized and the debug build: the debug build adds two extra pointers near the top (_ob_next and _ob_prev), before the ob_refcnt.  There are also some changes to PyTypeObject.  See object.h, and look for #ifdef Py_TRACE_REFS.

If the debug libpython tries to use a module built for the optimized libpython (or vice versa), it will likely immediately segfault - things won't be in memory in their expected locations: even changing the reference count will be accessing the wrong memory location.

> Does that mean that you wanted to make this
> bulletproof against cases where Python 2.6 "module.so" would be present and
> Python 2.7 would try to load "module_d.so"? Some different concerns?
More about bulletproofing optimized vs debug within a release e.g. 2.7 debug vs 2.7 optimized.  As noted above, 2.7 debug modules aren't compatible with 2.7 optimized libpython.
Comment 4 Dave Malcolm 2013-11-18 11:10:43 EST
(In reply to Miha from comment #2)
> Thanks for the response! Do you know of any workaround (hack) I can use to
> make this work for me?
> 
> I tried to copy multiarray.so to multiarray_d.so, but that also fails:
> 
> [miha@localhost ~]$ sudo cp
> /usr/lib64/python2.7/site-packages/numpy/core/multiarray.so
> /usr/lib64/python2.7/site-packages/numpy/core/multiarray_d.so
> [miha@localhost ~]$ python-debug
> Python 2.7.5 (default, Nov 12 2013, 16:45:15) 
> [GCC 4.8.2 20131017 (Red Hat 4.8.2-1)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import numpy
> python-debug: /builddir/build/BUILD/Python-2.7.5/Objects/object.c:65:
> _Py_AddToAllObjects: Assertion `(op->_ob_prev == ((void *)0)) ==
> (op->_ob_next == ((void *)0))' failed.
> Aborted (core dumped)

Yes: simply copying them won't work (for the reasons noted in comment #3.  I had expected a segfault, but you got a C-level assertion failure instead.  In any case it's not going to work that way).

The modules need to be rebuilt against the correct configuration of python to be usable.   The fix may be to update how we package numpy, to build against both configurations, providing a -debug subpackage - though I'm not the numpy/python maintainer.
Comment 5 Bohuslav "Slavek" Kabrda 2013-11-19 02:18:26 EST
Thanks for the detailed info, Dave!
Miha, I'd probably suggest asking maintainer of numpy to create a debug build (maybe reassign this bug). A problem might be that there is no standardized way of building debug builds of extension packages.
I'm currently pretty occupied, but I'll try to take this to FPC before end of the week to get extension debug builds standardized in some way.
Comment 6 Miha 2013-11-19 14:58:48 EST
Thanks for the help! I opened a new bug report for numpy: https://bugzilla.redhat.com/show_bug.cgi?id=1031998
Comment 7 Orion Poplawski 2013-11-27 21:59:11 EST
Instructions as to how to build a debug version of numpy (or other python extensions) would be useful.  I have no idea how to go about it.
Comment 8 Fedora Admin XMLRPC Client 2015-05-12 08:02:25 EDT
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 9 Fedora End Of Life 2015-05-29 05:46:13 EDT
This message is a reminder that Fedora 20 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 20. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '20'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 20 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.
Comment 10 Fedora End Of Life 2015-06-29 08:56:01 EDT
Fedora 20 changed to end-of-life (EOL) status on 2015-06-23. Fedora 20 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Note You need to log in before you can comment on or make changes to this bug.