Bug 662034 - Consider including a /usr/local directory for Python sysadmins to install their own packages
Summary: Consider including a /usr/local directory for Python sysadmins to install the...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: python3
Version: rawhide
Hardware: Unspecified
OS: Unspecified
low
medium
Target Milestone: ---
Assignee: Charalampos Stratakis
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 850915 994307 1010005 (view as bug list)
Depends On: 1421694
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-12-10 11:41 UTC by Richard W.M. Jones
Modified: 2018-04-11 06:57 UTC (History)
29 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-12-07 15:28:14 UTC
Type: ---


Attachments (Terms of Use)
Proposed patch (1004 bytes, patch)
2010-12-14 18:53 UTC, Karel Klíč
no flags Details | Diff

Description Richard W.M. Jones 2010-12-10 11:41:09 UTC
Description of problem:

Rather than describing it all again, it's best to read the
discussion on the packaging mailing list here:

http://lists.fedoraproject.org/pipermail/packaging/2010-December/007539.html
http://lists.fedoraproject.org/pipermail/packaging/2010-December/007541.html
http://lists.fedoraproject.org/pipermail/packaging/2010-December/007542.html
http://lists.fedoraproject.org/pipermail/packaging/2010-December/007543.html

Simplest solution would be to add a /usr/local directory
for Python sysadmins to install their own packages.
But also from the above discussion, should we harmonize the
directory structure with what Debian are doing?

Version-Release number of selected component (if applicable):

Python in Rawhide (Fedora 15).

Comment 1 Dave Malcolm 2010-12-10 17:02:17 UTC
Yeah, I've wondered about doing the dist-packages approach:
https://fedoraproject.org/wiki/DaveMalcolm/PythonIdeas#dist-packages_rather_than_site-packages_.3F

but it's a big change to how we package things: unfortunately, every python specfile has the macros referring to "python_sitelib" or "python_sitearch":
  http://fedoraproject.org/wiki/Packaging/Python#Macros
and would be confusing to have these refer to "dist" rather than "site" without also renaming the macros.

One issue is that Debian's approach is a downstream modification (albeit a minor one), so perhaps an upstream PEP might be a good idea, so agree on cross-distribution standardization on this approach.

I seem to remember some other criticisms of Debian's approach, but I can't remember what the exact concerns were.

Having made big changes in Fedora 13 for Python 3,  I'm nervous about making another big change to how we package Python: my feeling is that if we change it again, I'd prefer to make multiple fixes, so that the payoff from the effort is worth it.  In particular, I'd love to implement the "multiple parallel runtimes" idea from:
  http://lists.fedoraproject.org/pipermail/python-devel/2010-March/000213.html
but I don't know if our Fedora packaging community is ready to embrace it (and there are some warts in that proposal that would need fixing).

(since I wrote that email, PEP 384 and PEP 3147 are now done, along with PEP 3149; having all of these suggests other ways in which we could reorganize things)

Comment 2 Toshio Ernie Kuratomi 2010-12-10 19:43:33 UTC
I don't see any value in harmonizing with what Debian is doing WRT dist-packages.  It's a change from upstream that doesn't seem to serve any purpose (If you look at what they're doing, they want sysadmins installing packages from upstream to put them into /usr/local/dist-packages anyway... not into any directory named site-packages).  Adding a /usr/local/XXX to sys.path does seem useful though.

Comment 3 Karel Klíč 2010-12-14 18:53:25 UTC
Created attachment 468667 [details]
Proposed patch

I have fixed this issue for myself by adding /usr/local to the search path and then found this bug report, so I'm attaching a patch here. Please consider it. From comments in the modified file it seems that this is the way preferred by upstream.

The problem I had is that when a Python package uses automake (something_PYTHON etc.), and if I install it using "./configure; make; sudo make install", the package is installed into /usr/local/lib/python2.7/site-packages and the interpreter don't see it. 

I was surprised this doesn't work. FHS says: The /usr/local hierarchy is for use by the system administrator when installing software locally.

Comment 4 Matěj Cepl 2011-10-28 18:13:26 UTC
In order to make easy_install (and pip) to install to /usr/local I had to (per http://packages.python.org/distribute/easy_install.html#administrator-installation):

1) create file /usr/lib64/python2.6/site-packages/altinstall.pth with this content:

import os, site; site.addsitedir('/usr/local/lib/python2.6/site-packages')

2) create file /usr/lib64/python2.6/distutils/distutils.cfg with this content

[install]
install_lib = /usr/local/lib/python2.6/site-packages

# This next line is optional but often quite useful; it directs EasyInstall
# and the distutils to install scripts in the user's "bin" directory.  For
# Mac OS X framework Python builds, you should use /usr/local/bin instead,
# because neither ~/bin nor the default script installation location are on
# the system PATH.
install_scripts = /usr/local/bin/

3) Add /usr/local/lib/python2.6/site-packages to users' $PYTHONPATH in their ~/.bashrc.

However, then I got broken building of python-* packages locally (see bug 749861).

Comment 5 Eric Hopper 2012-08-24 00:44:54 UTC
*** Bug 850915 has been marked as a duplicate of this bug. ***

Comment 6 Eric Hopper 2012-08-24 00:46:55 UTC
Has anything been done with this?

I'm having the same issue and filed another bug on it before I found this one.

Comment 7 Bohuslav "Slavek" Kabrda 2012-08-24 06:06:51 UTC
Please note, that we are currently using similar approach in RubyGems installation:

1) $ gem install foo # user installs Gems from rubygems.org to home dir
2) # gem install foo # root installs Gems from rubygems.org under /usr/local
3) # yum install rubygem-foo # installs the RPM-packaged gems under /usr

Advantages of this approach:
- Each user can have own packages in home, that don't change anything for other users.
- If there is a specific version, that system administrator needs installed and it is not RPM-packaged, he installs that into /usr/local using "gem install".
- Packages under /usr are only touched by RPM and the "gem" command can't touch them by default. This prevents mixing of libraries installed via "gem" and "yum", which is a nice feature.

I think that Python could have at least 2) and 3), but having also 1) would be very nice.

(If you don't know anything about Ruby, then s/gem/easy_install/ and s/ruby/python/ :) ).

Comment 8 Eric Hopper 2012-08-24 06:51:15 UTC
(In reply to comment #7)
> I think that Python could have at least 2) and 3), but having also 1) would
> be very nice.

Actually, Python currently has 1 and 3. But it (IMHO) desperately needs 2.

Currently 2 can be achieved by hand editing /usr/lib64/python$py_short_version/site.py and putting '/usr/local' first in the PREFIXES list. Also, you have to remember to give the right arguments when using pip-python or easy_install to install things. But all that's really ugly to have to do. It should be done when Python is packaged.

Comment 9 Bohuslav "Slavek" Kabrda 2012-08-24 06:58:32 UTC
(In reply to comment #8)
> (In reply to comment #7)
> > I think that Python could have at least 2) and 3), but having also 1) would
> > be very nice.
> 
> Actually, Python currently has 1 and 3. But it (IMHO) desperately needs 2.
> 

Yes, but for 1), you need to specify --prefix parameter, AFAIK. In Ruby, it works out of box. Just run "gem install foo".
3) is also supported, yes, but the disadvantage is, that modules installed via "sudo easy_install foo" (as in 2) ) get mixed with the system ones, which is what Ruby manages to avoid by installing these under /usr/local automatically.

> Currently 2 can be achieved by hand editing
> /usr/lib64/python$py_short_version/site.py and putting '/usr/local' first in
> the PREFIXES list. Also, you have to remember to give the right arguments
> when using pip-python or easy_install to install things. But all that's
> really ugly to have to do. It should be done when Python is packaged.

Comment 10 Toshio Ernie Kuratomi 2012-08-24 16:54:28 UTC
(In reply to comment #9)
> 
> Yes, but for 1), you need to specify --prefix parameter, AFAIK. In Ruby, it
> works out of box. Just run "gem install foo".

The upstream python-setuptools and python-pip tools are designed to install into a sitewide directory by default.  They give you an explicit --user switch to properly install into the user's home directory.

Comment 11 Bohuslav "Slavek" Kabrda 2012-08-27 06:20:27 UTC
(In reply to comment #10)
> (In reply to comment #9)
> > 
> > Yes, but for 1), you need to specify --prefix parameter, AFAIK. In Ruby, it
> > works out of box. Just run "gem install foo".
> 
> The upstream python-setuptools and python-pip tools are designed to install
> into a sitewide directory by default.  They give you an explicit --user
> switch to properly install into the user's home directory.

I think that the "--user" may be a good thing for root, who may want to specify whether to install into his home or "/usr/local". For normal non-privileged user, I think the best course is to install into his home by default.

Comment 12 Toshio Ernie Kuratomi 2012-08-27 14:34:36 UTC
(In reply to comment #11)

> I think that the "--user" may be a good thing for root, who may want to
> specify whether to install into his home or "/usr/local". For normal
> non-privileged user, I think the best course is to install into his home by
> default.

You should take that up with the upstreams then.

Comment 13 Fedora Admin XMLRPC Client 2013-05-10 04:58:08 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 14 Alan Pevec 2013-08-05 12:54:01 UTC
> You should take that up with the upstreams then.

What were upstreams reactions?

Comment 15 Pádraig Brady 2013-08-07 15:30:01 UTC
*** Bug 994307 has been marked as a duplicate of this bug. ***

Comment 16 Tim Flink 2013-10-15 14:58:20 UTC
*** Bug 1010005 has been marked as a duplicate of this bug. ***

Comment 17 Toshio Ernie Kuratomi 2014-03-21 22:23:51 UTC
Found out today that dstufft is working on --user by default upstream: https://github.com/pypa/pip/issues/1668 which should help out with the pip install portion of this bug.

Comment 18 Robert Kuska 2014-04-01 11:51:36 UTC
I have created a thread at pypa-dev google group
https://groups.google.com/forum/#!topic/pypa-dev/r6qsAmJl9t0

Comment 19 Daniel Miranda 2015-02-27 09:04:25 UTC
I've been bitten today by accidentally upgrading packages installed by dnf with pip, and breaking dnf itself. I find having the default pip behaviour to overwrite system packages quite unsettling and user-unfriendly. It seems the upstream discussion has not gone anywhere, but I think the current situation is really bad as it is. pip should be changed to use /usr/local by default in my opinion: having pip be able to break the system package manager is like playing Russian roulette for no reason.

Would any patches in that direction be accepted?

Comment 20 Piotr Dobrogost 2015-02-27 11:02:12 UTC
I remember reading thread started and mentioned earlier by Robert and was very glad to see this initiative back at the time.
Robert, could you refer here what is the status of that discussion?
Shouldn't Nick Coghlan be added to CC?

Comment 21 Bohuslav "Slavek" Kabrda 2015-03-02 09:34:50 UTC
(In reply to Daniel Miranda from comment #19)
> I've been bitten today by accidentally upgrading packages installed by dnf
> with pip, and breaking dnf itself. I find having the default pip behaviour
> to overwrite system packages quite unsettling and user-unfriendly. It seems
> the upstream discussion has not gone anywhere, but I think the current
> situation is really bad as it is. pip should be changed to use /usr/local by
> default in my opinion: having pip be able to break the system package
> manager is like playing Russian roulette for no reason.
> 
> Would any patches in that direction be accepted?

Introducing /usr/local/... directory for package installation by pip is something I've wanted for a long time, but this change really needs a proper upstream discussion and consensus. Robert, is there any chance you can get back to this?
I'm adding Nick Coghlan to CC, since he's doing tons of upstream stuff around packaging, etc and I'm sure he'll have something interesting to say here.

If/when upstream implements this change, it will only take us halfway through solving this issue, though. The other part is having a commandline switch for Python that would allow ignoring all paths except /usr/lib[64]/pythonX.Y - and this switch will need to be added to all system packaged scripts with python hashbangs.

Comment 22 Fedora Admin XMLRPC Client 2015-05-12 12:01:40 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 23 Orion Poplawski 2015-09-12 14:07:56 UTC
I think we really critically need this.  Especially as we move towards a layered approach to the OS.  If the layering tools (pip) corrupts the lower layers, we're doomed.

Comment 24 Graham White 2015-11-18 13:56:24 UTC
Yup, this gets a +1 from me too!

Comment 25 Fedora Admin XMLRPC Client 2016-01-29 13:04:34 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 26 Marcus Furlong 2016-07-07 02:30:01 UTC
+1, also got bitten by this.

Possibly also worth noting that `pip list --local` on Debian lists the non-distro python modules that are installed, whereas on Red Hat it gives the exact same output as `pip list`. Which is not what you want! It does this by using dist-packages for distro-installed python modules and site-packages for pip-installed python modules.

Comment 27 Fedora Admin XMLRPC Client 2017-01-10 18:49:10 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 28 Charalampos Stratakis 2017-06-27 13:28:26 UTC
F27 will have this issue resolved.

More details at: https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe


Note You need to log in before you can comment on or make changes to this bug.