Bug 471617 - Missing path in the virtual environment.
Missing path in the virtual environment.
Status: CLOSED WORKSFORME
Product: Fedora
Classification: Fedora
Component: python-virtualenv (Show other bugs)
9
x86_64 Linux
medium Severity medium
: ---
: ---
Assigned To: Steve Milner
Fedora Extras Quality Assurance
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-11-14 12:24 EST by Thomas Moschny
Modified: 2015-04-24 15:01 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-04-24 15:01:17 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Thomas Moschny 2008-11-14 12:24:01 EST
Description of problem:
Inside a virtual environment created with the 'virtualenv' command, the /usr/lib64/python2.5/site-packages path is missing.

Version-Release number of selected component (if applicable):
python-virtualenv-1.3-1.fc9.noarch

Steps to Reproduce:
1. Create a virtual environment like with 'virtualenv /tmp/env'
2. Compare paths outside and inside using
   "python -c 'import sys, pprint; pprint.pprint(sys.path);'" and 
   "/tmp/env/bin/python -c 'import sys, pprint; pprint.pprint(sys.path);'"
   respectively.
  
Actual results:
% python -c 'import sys, pprint; pprint.pprint(sys.path);' 
['',
 '/usr/lib64/python25.zip',
 '/usr/lib64/python2.5',
 '/usr/lib64/python2.5/plat-linux2',
 '/usr/lib64/python2.5/lib-tk',
 '/usr/lib64/python2.5/lib-dynload',
 '/usr/lib64/python2.5/site-packages',
 '/usr/lib64/python2.5/site-packages/Numeric',
 '/usr/lib64/python2.5/site-packages/PIL',
 '/usr/lib64/python2.5/site-packages/gst-0.10',
 '/usr/lib64/python2.5/site-packages/gtk-2.0',
 '/usr/lib/python2.5/site-packages']

% /tmp/env/bin/python -c 'import sys, pprint; pprint.pprint(sys.path);'
['',
 '/tmp/env/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg',
 '/tmp/env/lib64/python25.zip',
 '/tmp/env/lib64/python2.5',
 '/tmp/env/lib64/python2.5/plat-linux2',
 '/tmp/env/lib64/python2.5/lib-tk',
 '/tmp/env/lib64/python2.5/lib-dynload',
 '/usr/lib/python2.5',
 '/usr/lib64/python2.5',
 '/usr/lib64/python2.5/lib-tk',
 '/tmp/env/lib/python2.5/site-packages',
 '/usr/lib/python2.5/site-packages']

Expected results:
See how /usr/lib64/python2.5/site-packages (and thus all paths below, like Numeric, PIL, etc.) are missing in the second output.

Additional info:
A similar problem (on OpenSuSE) was described here https://bugs.launchpad.net/virtualenv/+bug/248558/comments/2 (note that that is different from the original bug report) and propagated to OpenSuSE's bugtracker here: https://bugzilla.novell.com/show_bug.cgi?id=430761. That might be related if Fedora is also patching site.py.
Comment 1 Steve Milner 2008-11-17 10:15:20 EST
Thanks for the report. I'll look into pushing up an upstream patch.
Comment 2 Steve Milner 2008-11-17 10:25:19 EST
I'll give the patch listed at https://bugs.launchpad.net/fedora/+source/python-virtualenv/+bug/2485 a run and see if that fixes the issue on a 64bit host. It was applied on 2008-09-29.
Comment 3 Fedora Update System 2008-11-28 14:15:07 EST
python-virtualenv-1.3.1-1.fc10 has been submitted as an update for Fedora 10.
http://admin.fedoraproject.org/updates/python-virtualenv-1.3.1-1.fc10
Comment 4 Thomas Moschny 2008-11-28 14:40:54 EST
(In reply to comment #3)
> python-virtualenv-1.3.1-1.fc10 has been submitted as an update for Fedora 10.

This doesn't fix the issue -- and wasn't supposed to, because upstream's 1.3.1 fixes the original problem reported in launchpad's bug 248558, not the problem described in comment 2 thereof. 

Here's the current output on fc10, with
python-2.5.2-1.fc10.x86_64
python-setuptools-0.6c8-1.fc10.noarch
python-virtualenv-1.3.1-1.fc10.noarch

% python -c 'import sys, pprint; pprint.pprint(sys.path);' 
['',
 '/usr/lib64/python25.zip',
 '/usr/lib64/python2.5',
 '/usr/lib64/python2.5/plat-linux2',
 '/usr/lib64/python2.5/lib-tk',
 '/usr/lib64/python2.5/lib-dynload',
 '/usr/lib64/python2.5/site-packages',
 '/usr/lib64/python2.5/site-packages/Numeric',
 '/usr/lib64/python2.5/site-packages/PIL',
 '/usr/lib64/python2.5/site-packages/gst-0.10',
 '/usr/lib64/python2.5/site-packages/gtk-2.0',
 '/usr/lib/python2.5/site-packages']

vs.

% /tmp/env/bin/python -c 'import sys, pprint; pprint.pprint(sys.path);' 
['',
 '/tmp/env/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg',
 '/tmp/env/lib64/python25.zip',
 '/tmp/env/lib64/python2.5',
 '/tmp/env/lib64/python2.5/plat-linux2',
 '/tmp/env/lib64/python2.5/lib-tk',
 '/tmp/env/lib64/python2.5/lib-dynload',
 '/usr/lib/python2.5',
 '/usr/lib64/python2.5',
 '/usr/lib64/python2.5/lib-tk',
 '/tmp/env/lib/python2.5/site-packages',
 '/usr/lib/python2.5/site-packages']

Still, in the second output, paths /usr/lib64/python2.5/site-packages and below are missing.

The problem is that fedora patches site.py, which virtualenv doesn't (can't?) detect.
Comment 5 Steve Milner 2008-12-01 18:58:54 EST
Ok, I see what you mean. virtualenv is writing out it's site.py via a zipped version it holds in virtualenv.py as base64 encoded zipped data. I'll take a look at the patches being applied in the python fedora package that might make sense to pull into virtualenv's copy.
Comment 6 Steve Milner 2008-12-01 20:47:07 EST
These all end up mapping to pth files and are owned by different packages ... here is an example from my F9 laptop ...

[steve@tachikoman site-packages]$ for x in `find . -name "*.pth"`; do
    echo -n "$x: ";
    echo `rpm -q --whatprovides /usr/lib/python2.5/site-packages/$x`;
done
./PasteDeploy-1.3.1-py2.5-nspkg.pth: python-paste-deploy-1.3.1-2.fc9.noarch
./Numeric.pth: python-numeric-24.2-11.fc9.i386
./Paste-1.6-py2.5-nspkg.pth: python-paste-1.6-1.fc9.noarch
./PIL.pth: python-imaging-1.1.6-9.fc9.i386
./pygtk.pth: pygobject2-2.14.2-1.fc9.i386
./easy-install.pth: file /usr/lib/python2.5/site-packages/easy-install.pth is not owned by any package
./pygst.pth: gstreamer-python-0.10.11-2.fc9.i386
./DecoratorTools-1.7-py2.5-nspkg.pth: python-decoratortools-1.7-1.fc9.noarch
./PasteScript-1.6.2-py2.5-nspkg.pth: python-paste-script-1.6.2-2.fc9.noarch
./wx.pth: wxPython-2.8.9.1-1.fc9.i386

or on my F10 desktop

[steve@powerhouse site-packages]$ for x in `find . -name "*.pth"`; do
>     echo -n "$x: ";
>     echo `rpm -q --whatprovides /usr/lib64/python2.5/site-packages/$x`;
> done
./pygtk.pth: pygobject2-2.15.4-3.fc10.x86_64
./Numeric.pth: python-numeric-24.2-11.fc9.x86_64
./PIL.pth: python-imaging-1.1.6-12.fc10.x86_64
./pygst.pth: gstreamer-python-0.10.12-1.fc10.x86_64
./scim.pth: scim-python-0.1.13rc1-1.fc10.x86_64

I looked quickly at the patching that is done on site.py in the python package and it's pretty minimal and it didn't look to be related.

I guess I could make a patch that generates full path pth's off of whats in the site-packages dir(s). I don't know if that really makes sense though since it's per package.

I could still be missing something though ... let me know if I am Thomas.
Comment 7 Thomas Moschny 2008-12-02 03:29:14 EST
(In reply to comment #6)

> I looked quickly at the patching that is done on site.py in the python package
> and it's pretty minimal and it didn't look to be related.

Well, it is surely related, because it (at least) changes one of the sitedirs and adds another one on a 64 bit system. Here's one of the most relevant chunks from python/F-10/python-2.5-lib64.patch (applied as patch No. 102):

--- Python-2.5b1/Lib/site.py.lib64	2006-06-12 04:23:02.000000000 -0400
+++ Python-2.5b1/Lib/site.py	2006-06-22 12:20:35.000000000 -0400
@@ -182,9 +182,14 @@
                 sitedirs = [os.path.join(prefix, "Lib", "site-packages")]
             elif os.sep == '/':
                 sitedirs = [os.path.join(prefix,
-                                         "lib",
+                                         "lib64",
                                          "python" + sys.version[:3],
                                          "site-packages"),
+                            os.path.join(prefix,
+                                         "lib",
+                                         "python" + sys.version[:3],
+                                         "site-packages"),                            
+                            os.path.join(prefix, "lib64", "site-python"),
                             os.path.join(prefix, "lib", "site-python")]
             else:
                 sitedirs = [prefix, os.path.join(prefix, "lib", "site-packages")]

Imho that's exactly the sort of patch that should either be removed from the python package, or added to the virtualenv package. And there might be some other relevant chunks in that patch.

Maybe we should get one of the main python package's maintainers on board here?


> I guess I could make a patch that generates full path pth's off of whats
> in the site-packages dir(s). I don't know if that really makes sense though
> since it's per package.

That doesn't sound like the right (and neither a feasible) solution to me.
Comment 8 Ignacio Vazquez-Abrams 2008-12-02 03:47:10 EST
When building, grab site.py from the python package and patch it as necessary.
Comment 9 Steve Milner 2008-12-02 13:58:07 EST
The problem with grabbing site.py from the python package is that it is vastly different than the site.py zipped and base encoded in virtualenv. I can modify what virtualenv holds though using the python package's site.py file.

In the actual unzipped virtualenv source they talk about extra paths using pth's and give an example of debain's lib locations which is what made me look that the pth's being placed by things like Numeric, PIL, gst and gtk.

If you guys really think using the python packages patched version will fix the issue I'm willing to give it a shot ... but at first run virtualenv failed for me when trying to use it like that.
Comment 10 Thomas Moschny 2008-12-02 15:13:50 EST
(In reply to comment #9)
> The problem with grabbing site.py from the python package is that it is vastly
> different than the site.py zipped and base encoded in virtualenv.

Most of the differences come from the fact though that virtualenv's site.py is from python 2.3. The differences can be seen using (zsh syntax):

diff -u /usr/lib64/python2.5/site.py <(python -c 'import virtualenv; print virtualenv.SITE_PY')
Comment 11 Ignacio Vazquez-Abrams 2008-12-02 17:11:53 EST
I have a confession.

This is not the first time I've seen this bug.

I thought about it LONG AND HARD for the better part of a month when I initially hit it, and this was the only sane method I could find for solving it. Unfortunately I never got around to actually implementing it.
Comment 12 Thomas Moschny 2008-12-02 18:54:01 EST
Here's a naive patch:

Index: python-virtualenv.spec
===================================================================
RCS file: /cvs/pkgs/rpms/python-virtualenv/devel/python-virtualenv.spec,v
retrieving revision 1.5
diff -r1.5 python-virtualenv.spec
12a13
> Patch0:         virtualenv-lib64.patch
28c29,33
< %setup -q -n virtualenv-%{version}
---
> %setup0 -q -n virtualenv-%{version}
> %if "%{_lib}" == "lib64"
> %patch0 -p1 -b .lib64
> %endif
> %{__python} rebuild-script.py

diff -up virtualenv-1.3.1/support-files/site.py.orig virtualenv-1.3.1/support-files/site.py
--- virtualenv-1.3.1/support-files/site.py.orig	2008-11-25 03:27:26.000000000 +0100
+++ virtualenv-1.3.1/support-files/site.py	2008-12-03 00:38:40.000000000 +0100
@@ -213,6 +213,10 @@ def addsitepackages(known_paths, sys_pre
 
             elif os.sep == '/':
                 sitedirs = [os.path.join(prefix,
+                                         "lib64",
+                                         "python" + sys.version[:3],
+                                         "site-packages"),
+                            os.path.join(prefix,
                                          "lib",
                                          "python" + sys.version[:3],
                                          "site-packages"),


This seems to work, but now there are too many dirs in the path:

% /tmp/env/bin/python -c 'import sys, pprint; pprint.pprint(sys.path);'
['',
 '/tmp/env/lib64/python2.5/site-packages/setuptools-0.6c9-py2.5.egg',
 '/tmp/env/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg',
 '/tmp/env/lib64/python25.zip',
 '/tmp/env/lib64/python2.5',
 '/tmp/env/lib64/python2.5/plat-linux2',
 '/tmp/env/lib64/python2.5/lib-tk',
 '/tmp/env/lib64/python2.5/lib-dynload',
 '/usr/lib/python2.5',
 '/usr/lib64/python2.5',
 '/usr/lib64/python2.5/lib-tk',
 '/tmp/env/lib64/python2.5/site-packages',
 '/tmp/env/lib/python2.5/site-packages',
 '/usr/lib64/python2.5/site-packages',
 '/usr/lib64/python2.5/site-packages/Numeric',
 '/usr/lib64/python2.5/site-packages/PIL',
 '/usr/lib64/python2.5/site-packages/gst-0.10',
 '/usr/lib64/python2.5/site-packages/gtk-2.0',
 '/usr/lib/python2.5/site-packages']

Note that simply copying the system site.py did not work for me neither. According to a message found on their mailinglist, the version in the virtualenv package is slightly modified. So maybe the right way is to use system's site.py, but apply virtualenv's modfications.
Comment 13 Steve Milner 2008-12-11 20:54:24 EST
Well, if that works and the extra paths are not harming anything, I'm fine with using that patch ... at least for now. I'm going to run some tests with a modified version of the patch (- original virtualenv site.py, + my changes to the original virtualenv setup.py)....


--- /tmp/site.py.virtualenv	2008-12-11 19:40:58.000000000 -0500
+++ lib/python2.5/site.py	2008-12-11 20:50:55.000000000 -0500
@@ -201,11 +201,27 @@
                                              "python" + sys.version[:3], "site-packages")]
 
             elif os.sep == '/':
-                sitedirs = [os.path.join(prefix,
-                                         "lib",
-                                         "python" + sys.version[:3],
-                                         "site-packages"),
-                            os.path.join(prefix, "lib", "site-python")]
+                # PATCH:BZ471617
+                # https://bugzilla.redhat.com/show_bug.cgi?id=471617
+                import platform
+                # Maps the processor type with the library path name
+                lib_map = {'lib': 'lib',
+                           'x86_64': 'lib64'}
+
+                def __easy_path(cpu):
+                    """
+                    Simply returns a python path based on cpu and lib_map.
+                    """
+                    if cpu not in lib_map.keys():
+                        return ''
+                    return os.path.join(prefix,
+                                        lib_map[cpu],
+                                        "python" + sys.version[:3],
+                                        "site-packages")
+
+                sitedirs = [__easy_path("lib")]
+                sitedirs.append(__easy_path(platform.processor()))
+                # END PATCH:BZ471617
                 try:
                     # sys.getobjects only available in --with-pydebug build
                     sys.getobjects
Comment 14 Thomas Moschny 2008-12-12 07:52:26 EST
(In reply to comment #13)
> Well, if that works and the extra paths are not harming anything, I'm fine with
> using that patch ... at least for now.

Did you talk to upstream about this problem already? Maybe they have some idea how to cleanly solve it.

> modified version of the patch (- original virtualenv site.py, + my changes to
> the original virtualenv setup.py)....

That code looks far more complicated to me than it needs to be, but if it works *shrug*.

Maybe Ignacio can comment?
Comment 15 Steve Milner 2008-12-25 23:47:12 EST
From 1.3.2 release

    * Added platform-specific paths, like /usr/lib/pythonX.Y/plat-linux2

I now get the paths added in this version:

<snip>
'/usr/lib/python2.5/site-packages',
'/usr/lib64/python2.5/site-packages',
'/usr/lib64/python2.5/site-packages/Numeric',
'/usr/lib64/python2.5/site-packages/PIL',
'/usr/lib64/python2.5/site-packages/gst-0.10',
'/usr/lib64/python2.5/site-packages/gtk-2.0',
'/usr/lib64/python2.5/site-packages/scim-0.1']

Just an FYI as I'll be pushing the new version up with a spec fix with this new version.
Comment 17 Steve Milner 2009-01-08 22:23:16 EST
The package has been pushed to stable.
Comment 18 Thomas Moschny 2012-12-21 16:26:57 EST
Seeing this again on F17:

# virtualenv /tmp/env
New python executable in /tmp/env/bin/python
Installing setuptools............done.
Installing pip...............done.

# python -c 'import sys; print "\n".join(sys.path)' | grep -v '\.local'

/usr/lib64/python27.zip 
/usr/lib64/python2.7
/usr/lib64/python2.7/plat-linux2
/usr/lib64/python2.7/lib-tk
/usr/lib64/python2.7/lib-old
/usr/lib64/python2.7/lib-dynload
/usr/lib64/python2.7/site-packages
/usr/lib64/python2.7/site-packages/PIL
/usr/lib64/python2.7/site-packages/gst-0.10
/usr/lib64/python2.7/site-packages/gtk-2.0
/usr/lib64/python2.7/site-packages/wx-2.8-gtk2-unicode
/usr/lib/python2.7/site-packages
/usr/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg-info

# /tmp/env/bin/python -c 'import sys; print "\n".join(sys.path)'                    

/tmp/env/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg
/tmp/env/lib/python2.7/site-packages/pip-1.1-py2.7.egg
/tmp/env/lib64/python27.zip
/tmp/env/lib64/python2.7
/tmp/env/lib64/python2.7/plat-linux2
/tmp/env/lib64/python2.7/lib-tk
/tmp/env/lib64/python2.7/lib-old
/tmp/env/lib64/python2.7/lib-dynload
/usr/lib/python2.7
/usr/lib64/python2.7
/usr/lib64/python2.7/lib-tk
/tmp/env/lib/python2.7/site-packages

This yields to import failures, e.g.

# python -c 'import yaml'  
# /tmp/env/bin/python -c 'import yaml'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named yaml
Comment 19 Thomas Moschny 2015-04-24 15:01:17 EDT
Works on F21 with virtualenv --system-site-packages.

Note You need to log in before you can comment on or make changes to this bug.