Bug 467266

Summary: numpy installation causes Python's modules list request to produce a stacktrace
Product: [Fedora] Fedora Reporter: Matteo Castellini <self>
Component: numpyAssignee: Gwyn Ciesla <gwync>
Status: CLOSED WONTFIX QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: medium    
Version: 9CC: a.badger, gwync, jarod, jspaleta, rdieter
Target Milestone: ---   
Target Release: ---   
Hardware: i386   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-07-14 17:10:04 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 246212    
Bug Blocks:    

Description Matteo Castellini 2008-10-16 15:59:33 UTC
Description of problem:
The installation of numpy causes a stacktrace when trying to obtain Python's modules list.


Version-Release number of selected component (if applicable):
numpy-1.1.1-1.fc9.i386


How reproducible:
Always.


Steps to Reproduce:
1. $ python
2. >>> help()
3. help> modules
  
Actual results:
Please wait a moment while I gather a list of all available modules...

Data Dir: /usr/share/gnome-applets/invest-applet
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.5/site.py", line 346, in __call__
    return pydoc.help(*args, **kwds)
  File "/usr/lib/python2.5/pydoc.py", line 1646, in __call__
    self.interact()
  File "/usr/lib/python2.5/pydoc.py", line 1664, in interact
    self.help(request)
  File "/usr/lib/python2.5/pydoc.py", line 1680, in help
    elif request == 'modules': self.listmodules()
  File "/usr/lib/python2.5/pydoc.py", line 1801, in listmodules
    ModuleScanner().run(callback)
  File "/usr/lib/python2.5/pydoc.py", line 1852, in run
    for importer, modname, ispkg in pkgutil.walk_packages():
  File "/usr/lib/python2.5/pkgutil.py", line 125, in walk_packages
    for item in walk_packages(path, name+'.', onerror):
  File "/usr/lib/python2.5/pkgutil.py", line 125, in walk_packages
    for item in walk_packages(path, name+'.', onerror):
  File "/usr/lib/python2.5/pkgutil.py", line 110, in walk_packages
    __import__(name)
  File "/usr/lib/python2.5/site-packages/numpy/distutils/fcompiler/__init__.py", line 59, in <module>
    class FCompiler(CCompiler):
  File "/usr/lib/python2.5/site-packages/numpy/distutils/fcompiler/__init__.py", line 195, in FCompiler
    shared_lib_extension = get_config_var('SO')  # or .dll
  File "/usr/lib/python2.5/distutils/sysconfig.py", line 535, in get_config_var
    return get_config_vars().get(name)
  File "/usr/lib/python2.5/distutils/sysconfig.py", line 493, in get_config_vars
    func()
  File "/usr/lib/python2.5/distutils/sysconfig.py", line 352, in _init_posix
    raise DistutilsPlatformError(my_msg)
distutils.errors.DistutilsPlatformError: invalid Python installation: unable to open /usr/lib/python2.5/config/Makefile (No such file or directory)


Expected results:
Showing the actual Python's modules list.

Comment 1 Gwyn Ciesla 2008-10-16 16:30:11 UTC
Does numpy-1.2.0 in F-9 updates-testing and (soon to be updates) resolve this?

Comment 2 Jef Spaleta 2008-10-16 16:49:30 UTC
I get a traceback concerning /usr/share/gnome-applets/invest-applet
even with numpy not installed on my rawhide box.

Can you reproduce this with numpy uninstalled on your F9 box?

-jef

Comment 3 Gwyn Ciesla 2008-10-16 17:03:44 UTC
My traceback is different, F9, numpy-1.1.1:

help> modules

Please wait a moment while I gather a list of all available modules...

/dev/mapper/control: open failed: Permission denied
Failure to communicate with kernel device-mapper driver.
dm.c: 1577
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.5/site.py", line 346, in __call__
    return pydoc.help(*args, **kwds)
  File "/usr/lib/python2.5/pydoc.py", line 1646, in __call__
    self.interact()
  File "/usr/lib/python2.5/pydoc.py", line 1664, in interact
    self.help(request)
  File "/usr/lib/python2.5/pydoc.py", line 1680, in help
    elif request == 'modules': self.listmodules()
  File "/usr/lib/python2.5/pydoc.py", line 1801, in listmodules
    ModuleScanner().run(callback)
  File "/usr/lib/python2.5/pydoc.py", line 1852, in run
    for importer, modname, ispkg in pkgutil.walk_packages():
  File "/usr/lib/python2.5/pkgutil.py", line 110, in walk_packages
    __import__(name)
  File "/usr/lib/python2.5/site-packages/block/__init__.py", line 23, in <module>
    from device import MultiPath, RaidDev, RaidSet, BlockDev, DeviceMaps, \
  File "/usr/lib/python2.5/site-packages/block/device.py", line 209, in <module>
    class MPNameCache(_IUD):
  File "/usr/lib/python2.5/site-packages/block/device.py", line 214, in MPNameCache
    for map in _dm.maps():
MemoryError

Comment 4 Jef Spaleta 2008-10-16 17:34:36 UTC
Let's be clear.
Different traceback with numpy uninstalled?

Perhaps this is a deep python bug that multiple modules trip over?

-jef

Comment 5 Matteo Castellini 2008-10-16 17:44:26 UTC
Comment #1: installing numpy-1.2.0-1.fc9.i386 doesn't seem to solve this problem.

Comment #2, #4: applying the same procedure with numpy uninstalled solves the problem, as invoking "modules" doesn't produce the stacktrace any more.

Comment #3: your stacktrace is likely caused by python-pyblock (bug #246212).

Comment 6 Gwyn Ciesla 2008-10-16 17:46:37 UTC
Hmm, different machine, no numpy, no stacktrace.

Original machine, uninstalled numpy (and all it's deps. . .grrrr), and I get the stacktrace I posted in #3, same thing.  

I think Jef may be right, this might not be a numpy bug.  Still playing with it. . .

Comment 7 Jef Spaleta 2008-10-16 18:03:48 UTC
I found a dep issue.  On rawhide atleast if i install python-devel and numpy into a clean mock buildroot.. problem goes away.

We'll need to do more testing across other python modules to catch similar situations.  I'm going to make a suggestion to devel-list about how python maintainers can do a simple test with mock and we will go from there.

But for the moment...on rawhide (and F9-updates-testing I assume)...for numpy...something in the module logic for numpy in rawhide atleast really really wants to access /usr/lib/python2.5/config/Makefile  and throws an exception if its not available.

So what exactly do we do about that? Make python-devel a requirement for numpy runtime? Or do we patch the exception condition so it fails silently? Or what?

-jef

Comment 8 Jef Spaleta 2008-10-16 18:20:35 UTC
Okay here's how you confirm if a specific python module package is affected with mock

1) set up the mock localbuild root
mock -r <whatever_release> init    

2) chroot into the root of that mock environment

3) Run the python help module test, which should return clean
python
help()
modules
(Ctrl-D a few times to get back out of the python and chroot sessions)

4) install the module you think is a problem
mock -r <whatever_release> install <py_package>  

5) Do step 2 and 3 again see if its a traceback and if there is something is wonky with the python modules in the installed package.


-jef

Comment 9 Gwyn Ciesla 2008-10-16 18:51:11 UTC
My gut is to band-aid it with the Requires for now, while working on the real fix.  How many python modules are we talking about?  Might we need to coordinate a larger effort?

Comment 10 Jef Spaleta 2008-10-16 18:56:37 UTC
Possibly a lot of modules.... and it will take a big effort.  I posted to -devel-list.

Even the mock based procedure is not going to be enough. An exception in pygtk2 is triggered if the display is not found. So doing this under mock for anything that pulls in pygtk2 will traceback at the point, hiding any exception after the pygtk2 import is attempted. So on my rawhide client gnome-applets triggers a traceback that is masked by the pygtk2 display exception if I try to test gnome-applets under mock.  Madness!

Its sort of amazing that this functionality ever works actually. I think if a module throws an exception on import... this module listing procedure blows up.
The fix may very well be to tell python to just silently let all exceptions happen on module imports when building that listing.

-jef

Comment 11 Toshio Ernie Kuratomi 2008-10-16 20:45:10 UTC
Here's a really interesting one: fusion-icon-gtk-0.1.0-0.2.5e2dc9git.fc9.noarch

it starts running itself when modules() is scanning....

Comment 12 Gwyn Ciesla 2008-12-29 21:11:47 UTC
Where are we with this, in general?

Comment 13 Gwyn Ciesla 2009-03-05 20:06:51 UTC
Just tested this on F-10, got the following. . .

help> modules

Please wait a moment while I gather a list of all available modules...

/dev/mapper/control: open failed: Permission denied
Failure to communicate with kernel device-mapper driver.
dm.c: 1577
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.5/site.py", line 346, in __call__
    return pydoc.help(*args, **kwds)
  File "/usr/lib/python2.5/pydoc.py", line 1646, in __call__
    self.interact()
  File "/usr/lib/python2.5/pydoc.py", line 1664, in interact
    self.help(request)
  File "/usr/lib/python2.5/pydoc.py", line 1680, in help
    elif request == 'modules': self.listmodules()
  File "/usr/lib/python2.5/pydoc.py", line 1801, in listmodules
    ModuleScanner().run(callback)
  File "/usr/lib/python2.5/pydoc.py", line 1852, in run
    for importer, modname, ispkg in pkgutil.walk_packages():
  File "/usr/lib/python2.5/pkgutil.py", line 110, in walk_packages
    __import__(name)
  File "/usr/lib/python2.5/site-packages/block/__init__.py", line 23, in <module>
    from device import MultiPath, RaidDev, RaidSet, BlockDev, DeviceMaps, \
  File "/usr/lib/python2.5/site-packages/block/device.py", line 209, in <module>
    class MPNameCache(_IUD):
  File "/usr/lib/python2.5/site-packages/block/device.py", line 214, in MPNameCache
    for map in _dm.maps():
MemoryError

Comment 14 Matteo Castellini 2009-03-05 20:16:11 UTC
(In reply to comment #13)
> Just tested this on F-10, got the following. . .
> 
> help> modules
> 
> Please wait a moment while I gather a list of all available modules...
> 
> /dev/mapper/control: open failed: Permission denied
> Failure to communicate with kernel device-mapper driver.
> dm.c: 1577
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/usr/lib/python2.5/site.py", line 346, in __call__
>     return pydoc.help(*args, **kwds)
>   File "/usr/lib/python2.5/pydoc.py", line 1646, in __call__
>     self.interact()
>   File "/usr/lib/python2.5/pydoc.py", line 1664, in interact
>     self.help(request)
>   File "/usr/lib/python2.5/pydoc.py", line 1680, in help
>     elif request == 'modules': self.listmodules()
>   File "/usr/lib/python2.5/pydoc.py", line 1801, in listmodules
>     ModuleScanner().run(callback)
>   File "/usr/lib/python2.5/pydoc.py", line 1852, in run
>     for importer, modname, ispkg in pkgutil.walk_packages():
>   File "/usr/lib/python2.5/pkgutil.py", line 110, in walk_packages
>     __import__(name)
>   File "/usr/lib/python2.5/site-packages/block/__init__.py", line 23, in
> <module>
>     from device import MultiPath, RaidDev, RaidSet, BlockDev, DeviceMaps, \
>   File "/usr/lib/python2.5/site-packages/block/device.py", line 209, in
> <module>
>     class MPNameCache(_IUD):
>   File "/usr/lib/python2.5/site-packages/block/device.py", line 214, in
> MPNameCache
>     for map in _dm.maps():
> MemoryError  

This problem has already been reported in bugreport #246212

Comment 15 Gwyn Ciesla 2009-03-05 20:28:39 UTC
Ok.  So unless we can confirm this issue is fixed, we'll need to wait on 246212 to know for sure?  Adding blocker.

Comment 16 Jef Spaleta 2009-03-06 00:21:47 UTC
(In reply to comment #13)

with numpy-1.2.0-1.fc10.i386
and python-devel-2.5.2-1.fc10.i386
installed

no problem. The latest numpy update implied as built in bug 488464  for F-10 which makes numpy require python-devel hasn't landed in on my mirrors yet.

Nothing has changed with regard to this problem. 

There are lots of ways to get exceptions on module loading even if all package deps are fulfilled which are hardware situational or user configuration specific. 
If you can get into a situation where a module throws any exception on import, then this functionality will break.

Take a real close look at pydoc.py  and see if the listmodules function makes use of the safeimport function also defined in pydoc.py. I don't think it does... and I think that's the underlying problem here. Though I don't know how to correct it without talking to upstream about how to rewrite the listmodules function in pydoc.py to do a safe import.


-jef

Comment 17 Bug Zapper 2009-06-10 02:59:00 UTC
This message is a reminder that Fedora 9 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 9.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '9'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 9's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 9 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 18 Bug Zapper 2009-07-14 17:10:04 UTC
Fedora 9 changed to end-of-life (EOL) status on 2009-07-10. Fedora 9 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.