Bug 662034

Yeah, I've wondered about doing the dist-packages approach:
https://fedoraproject.org/wiki/DaveMalcolm/PythonIdeas#dist-packages_rather_than_site-packages_.3F

but it's a big change to how we package things: unfortunately, every python specfile has the macros referring to "python_sitelib" or "python_sitearch":
  http://fedoraproject.org/wiki/Packaging/Python#Macros
and would be confusing to have these refer to "dist" rather than "site" without also renaming the macros.

One issue is that Debian's approach is a downstream modification (albeit a minor one), so perhaps an upstream PEP might be a good idea, so agree on cross-distribution standardization on this approach.

I seem to remember some other criticisms of Debian's approach, but I can't remember what the exact concerns were.

Having made big changes in Fedora 13 for Python 3,  I'm nervous about making another big change to how we package Python: my feeling is that if we change it again, I'd prefer to make multiple fixes, so that the payoff from the effort is worth it.  In particular, I'd love to implement the "multiple parallel runtimes" idea from:
  http://lists.fedoraproject.org/pipermail/python-devel/2010-March/000213.html
but I don't know if our Fedora packaging community is ready to embrace it (and there are some warts in that proposal that would need fixing).

(since I wrote that email, PEP 384 and PEP 3147 are now done, along with PEP 3149; having all of these suggests other ways in which we could reorganize things)

I don't see any value in harmonizing with what Debian is doing WRT dist-packages.  It's a change from upstream that doesn't seem to serve any purpose (If you look at what they're doing, they want sysadmins installing packages from upstream to put them into /usr/local/dist-packages anyway... not into any directory named site-packages).  Adding a /usr/local/XXX to sys.path does seem useful though.

Created attachment 468667 [details]
Proposed patch

I have fixed this issue for myself by adding /usr/local to the search path and then found this bug report, so I'm attaching a patch here. Please consider it. From comments in the modified file it seems that this is the way preferred by upstream.

The problem I had is that when a Python package uses automake (something_PYTHON etc.), and if I install it using "./configure; make; sudo make install", the package is installed into /usr/local/lib/python2.7/site-packages and the interpreter don't see it. 

I was surprised this doesn't work. FHS says: The /usr/local hierarchy is for use by the system administrator when installing software locally.

In order to make easy_install (and pip) to install to /usr/local I had to (per http://packages.python.org/distribute/easy_install.html#administrator-installation):

1) create file /usr/lib64/python2.6/site-packages/altinstall.pth with this content:

import os, site; site.addsitedir('/usr/local/lib/python2.6/site-packages')

2) create file /usr/lib64/python2.6/distutils/distutils.cfg with this content

[install]
install_lib = /usr/local/lib/python2.6/site-packages

# This next line is optional but often quite useful; it directs EasyInstall
# and the distutils to install scripts in the user's "bin" directory.  For
# Mac OS X framework Python builds, you should use /usr/local/bin instead,
# because neither ~/bin nor the default script installation location are on
# the system PATH.
install_scripts = /usr/local/bin/

3) Add /usr/local/lib/python2.6/site-packages to users' $PYTHONPATH in their ~/.bashrc.

However, then I got broken building of python-* packages locally (see bug 749861).

*** Bug 850915 has been marked as a duplicate of this bug. ***

Has anything been done with this?

I'm having the same issue and filed another bug on it before I found this one.

Please note, that we are currently using similar approach in RubyGems installation:

1) $ gem install foo # user installs Gems from rubygems.org to home dir
2) # gem install foo # root installs Gems from rubygems.org under /usr/local
3) # yum install rubygem-foo # installs the RPM-packaged gems under /usr

Advantages of this approach:
- Each user can have own packages in home, that don't change anything for other users.
- If there is a specific version, that system administrator needs installed and it is not RPM-packaged, he installs that into /usr/local using "gem install".
- Packages under /usr are only touched by RPM and the "gem" command can't touch them by default. This prevents mixing of libraries installed via "gem" and "yum", which is a nice feature.

I think that Python could have at least 2) and 3), but having also 1) would be very nice.

(If you don't know anything about Ruby, then s/gem/easy_install/ and s/ruby/python/ :) ).

(In reply to comment #7)
> I think that Python could have at least 2) and 3), but having also 1) would
> be very nice.

Actually, Python currently has 1 and 3. But it (IMHO) desperately needs 2.

Currently 2 can be achieved by hand editing /usr/lib64/python$py_short_version/site.py and putting '/usr/local' first in the PREFIXES list. Also, you have to remember to give the right arguments when using pip-python or easy_install to install things. But all that's really ugly to have to do. It should be done when Python is packaged.

(In reply to comment #8)
> (In reply to comment #7)
> > I think that Python could have at least 2) and 3), but having also 1) would
> > be very nice.
> 
> Actually, Python currently has 1 and 3. But it (IMHO) desperately needs 2.
> 

Yes, but for 1), you need to specify --prefix parameter, AFAIK. In Ruby, it works out of box. Just run "gem install foo".
3) is also supported, yes, but the disadvantage is, that modules installed via "sudo easy_install foo" (as in 2) ) get mixed with the system ones, which is what Ruby manages to avoid by installing these under /usr/local automatically.

> Currently 2 can be achieved by hand editing
> /usr/lib64/python$py_short_version/site.py and putting '/usr/local' first in
> the PREFIXES list. Also, you have to remember to give the right arguments
> when using pip-python or easy_install to install things. But all that's
> really ugly to have to do. It should be done when Python is packaged.

(In reply to comment #9)
> 
> Yes, but for 1), you need to specify --prefix parameter, AFAIK. In Ruby, it
> works out of box. Just run "gem install foo".

The upstream python-setuptools and python-pip tools are designed to install into a sitewide directory by default.  They give you an explicit --user switch to properly install into the user's home directory.

(In reply to comment #10)
> (In reply to comment #9)
> > 
> > Yes, but for 1), you need to specify --prefix parameter, AFAIK. In Ruby, it
> > works out of box. Just run "gem install foo".
> 
> The upstream python-setuptools and python-pip tools are designed to install
> into a sitewide directory by default.  They give you an explicit --user
> switch to properly install into the user's home directory.

I think that the "--user" may be a good thing for root, who may want to specify whether to install into his home or "/usr/local". For normal non-privileged user, I think the best course is to install into his home by default.

(In reply to comment #11)

> I think that the "--user" may be a good thing for root, who may want to
> specify whether to install into his home or "/usr/local". For normal
> non-privileged user, I think the best course is to install into his home by
> default.

You should take that up with the upstreams then.

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

> You should take that up with the upstreams then.

What were upstreams reactions?

*** Bug 994307 has been marked as a duplicate of this bug. ***

*** Bug 1010005 has been marked as a duplicate of this bug. ***

Found out today that dstufft is working on --user by default upstream: https://github.com/pypa/pip/issues/1668 which should help out with the pip install portion of this bug.

I have created a thread at pypa-dev google group
https://groups.google.com/forum/#!topic/pypa-dev/r6qsAmJl9t0

I've been bitten today by accidentally upgrading packages installed by dnf with pip, and breaking dnf itself. I find having the default pip behaviour to overwrite system packages quite unsettling and user-unfriendly. It seems the upstream discussion has not gone anywhere, but I think the current situation is really bad as it is. pip should be changed to use /usr/local by default in my opinion: having pip be able to break the system package manager is like playing Russian roulette for no reason.

Would any patches in that direction be accepted?

I remember reading thread started and mentioned earlier by Robert and was very glad to see this initiative back at the time.
Robert, could you refer here what is the status of that discussion?
Shouldn't Nick Coghlan be added to CC?

(In reply to Daniel Miranda from comment #19)
> I've been bitten today by accidentally upgrading packages installed by dnf
> with pip, and breaking dnf itself. I find having the default pip behaviour
> to overwrite system packages quite unsettling and user-unfriendly. It seems
> the upstream discussion has not gone anywhere, but I think the current
> situation is really bad as it is. pip should be changed to use /usr/local by
> default in my opinion: having pip be able to break the system package
> manager is like playing Russian roulette for no reason.
> 
> Would any patches in that direction be accepted?

Introducing /usr/local/... directory for package installation by pip is something I've wanted for a long time, but this change really needs a proper upstream discussion and consensus. Robert, is there any chance you can get back to this?
I'm adding Nick Coghlan to CC, since he's doing tons of upstream stuff around packaging, etc and I'm sure he'll have something interesting to say here.

If/when upstream implements this change, it will only take us halfway through solving this issue, though. The other part is having a commandline switch for Python that would allow ignoring all paths except /usr/lib[64]/pythonX.Y - and this switch will need to be added to all system packaged scripts with python hashbangs.

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

I think we really critically need this.  Especially as we move towards a layered approach to the OS.  If the layering tools (pip) corrupts the lower layers, we're doomed.

Yup, this gets a +1 from me too!

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

+1, also got bitten by this.

Possibly also worth noting that `pip list --local` on Debian lists the non-distro python modules that are installed, whereas on Red Hat it gives the exact same output as `pip list`. Which is not what you want! It does this by using dist-packages for distro-installed python modules and site-packages for pip-installed python modules.

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

F27 will have this issue resolved.

More details at: https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe