Bug 1937494

> sys.path does not contain /usr/local

Can you please list files of the installed package? In which directory was it installed? You can try "pip uninstall YOUR_PACKAGE" to list files (reply no to not uninstall it ;-)).

Python only adds /usr/local/(...) directories to sys.path if they exist.


On RHEL 8, "sudo python3 -m install (...)" installs the package in /usr/local by default. Example upgrading pip:

$ python3 -m site
sys.path = [
    (...)
    '/usr/lib64/python36.zip',
    '/usr/lib64/python3.6',
    '/usr/lib64/python3.6/lib-dynload',
    '/usr/lib64/python3.6/site-packages',
    '/usr/lib/python3.6/site-packages',
]
(...)

$ sudo python3 -m pip install --upgrade pip
(...)

$ python3 -m site
sys.path = [
    (...)
    '/usr/lib64/python36.zip',
    '/usr/lib64/python3.6',
    '/usr/lib64/python3.6/lib-dynload',
    '/usr/local/lib/python3.6/site-packages',   <=== NEW PATH
    '/usr/lib64/python3.6/site-packages',
    '/usr/lib/python3.6/site-packages',
]
(...)


pip is installed in /usr/local:

$ sudo python3 -m pip uninstall pip
Found existing installation: pip 21.0.1
Uninstalling pip-21.0.1:
  Would remove:
    /usr/bin/pip3
    /usr/local/bin/pip
    /usr/local/bin/pip3
    /usr/local/bin/pip3.6
    /usr/local/lib/python3.6/site-packages/pip-21.0.1.dist-info/*
    /usr/local/lib/python3.6/site-packages/pip/*
Proceed (y/n)?

bz#1937492 is the same issue for Python 3.8, I propose to discuss the issue here.

/usr/local is also excluded for scripts in isolated mode (python3 -I):

# python3.6 -m pip install --prefix=/usr/local/ defusedxml
WARNING: Running pip install with root privileges is generally not a good idea. Try `__main__.py install --user` instead.
Collecting defusedxml
  Downloading https://files.pythonhosted.org/packages/07/6c/aa3f2f849e01cb6a001cd8554a88d4c77c5c1a31c95bdf1cf9301e6d9ef4/defusedxml-0.7.1-py2.py3-none-any.whl
Installing collected packages: defusedxml
Successfully installed defusedxml-0.7.1

# python3
Python 3.6.8 (default, Aug 18 2020, 08:33:21) 
[GCC 8.3.1 20191121 (Red Hat 8.3.1-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import defusedxml
>>> defusedxml
<module 'defusedxml' from '/usr/local/lib/python3.6/site-packages/defusedxml/__init__.py'>

# python3 -I
Python 3.6.8 (default, Aug 18 2020, 08:33:21) 
[GCC 8.3.1 20191121 (Red Hat 8.3.1-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import defusedxml
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'defusedxml'

Note that /usr/local/... is not in sys.path when RPM_BUILD_ROOT is set. Any chance this happens to you when building RPM packages?

Indeed, I did initially encounter this when building an RPM and thought I'd correctly traced it back to python.  I'm bootstraping a bunch of python module RPMs that have some interdependence, I'll open a ticket focused on that and try to keep this one focused on python itself.


It seems a bit odd that python3 -I excludes /usr/local/, that should be a root writable only location.  Was excluding /usr/local/ an 'EL' choice or did that come from upstream?

> Was excluding /usr/local/ an 'EL' choice or did that come from upstream?

From Fedora. Upstream only ever includes /usr/local/ when Python itself is installed into /usr/local/.

I'm wondering if that exclude matches expectations...

> isolated mode. This also implies -E and -s.
> In isolated mode sys.path contains neither the script’s directory nor the user’s site-packages directory.
> All PYTHON* environment variables are ignored, too.
> Further restrictions may be imposed to prevent the user from injecting malicious code.

/usr/local/ isn't traditionally considered a place where a user can write files (and thereby inject code).

If I'm trying to build a python environment with some additional (not in an RPM) python modules and run the app under `-I`, the most obvious solution is to use pip to install my modules as root.  However, those modules will go into /usr/local/ and not be accessible to to the app under `-I`.

With the exclusion of /usr/local/ by `-I`, but pip install defaulting to /usr/local, this seems to transform `-I` into "only use modules from RPMs".

Indeed, -I will isolate the interpreter to use system packages (i.e. RPMs) only.

For motivation why it's set up that way, see the relevant Fedora Change: https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe

Interesting read!

Tracked down to https://src.fedoraproject.org/rpms/python3/c/332b947dfc8d0f0d3a4525864b121d0f239beb4d?branch=master thanks to that link!

I'll confess a bit of surprise to see /usr/local dropped with 'ENABLE_USER_SITE` as I don't typically think of that as a user area.

(In reply to Pat Riehecky from comment #10)
> Interesting read!
> 
> Tracked down to
> https://src.fedoraproject.org/rpms/python3/c/
> 332b947dfc8d0f0d3a4525864b121d0f239beb4d?branch=master thanks to that link!
> 
> I'll confess a bit of surprise to see /usr/local dropped with
> 'ENABLE_USER_SITE` as I don't typically think of that as a user area.

Indeed, it's not the first thing that comes to mind when talking of a user area. However, in our long experience of people breaking their DNF and other RPM-installed software by sudo pip installing software, we resorted to this solution to isolate such critical software. And looking back, this change is now going on 3 years old, and it has achieved the stated goal of making usage of sudo pip much safer for users and their systems.

*** Bug 1937765 has been marked as a duplicate of this bug. ***

*** Bug 1937492 has been marked as a duplicate of this bug. ***

I am closing this as NOTABUG based on this comment: https://bugzilla.redhat.com/show_bug.cgi?id=1937765#c3

If you want to follow up on the discussion, may I suggest to move the the Fedora's Python mailing list? https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/