Bug 1937494 - python3.6 sys.path doesn't include /usr/local/ under some conditions
Summary: python3.6 sys.path doesn't include /usr/local/ under some conditions
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: python36
Version: CentOS Stream
Hardware: All
OS: Linux
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Python Maintainers
QA Contact: RHEL CS Apps Subsystem QE
URL:
Whiteboard:
: 1937492 1937765 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-03-10 18:33 UTC by Pat Riehecky
Modified: 2021-03-16 09:03 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1937492
Environment:
Last Closed: 2021-03-16 09:03:28 UTC
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)

Description Pat Riehecky 2021-03-10 18:33:29 UTC
+++ This bug was initially created as a clone of Bug #1937492 +++

Description of problem:

When I python packages into `/usr/local`, it comes as a surprise that `/usr/local/` is not within the default python search path.



Version-Release number of selected component (if applicable):python36-3.6.8-2.module_el8.3.0+562+e162826a.x86_64


How reproducible: 100%


Steps to Reproduce:
1.python3.6 -m pip install --prefix=/usr/local/ SOMEPACKAGE_FROM_PYPI 
2.python3.6 -m SOMEPACKAGE_FROM_PYPI
3.

Actual results:

sys.path does not contain /usr/local

Expected results:

sys.path should contain /usr/local sections

Additional info:

Comment 1 Victor Stinner 2021-03-11 11:40:15 UTC
> sys.path does not contain /usr/local

Can you please list files of the installed package? In which directory was it installed? You can try "pip uninstall YOUR_PACKAGE" to list files (reply no to not uninstall it ;-)).

Python only adds /usr/local/(...) directories to sys.path if they exist.


On RHEL 8, "sudo python3 -m install (...)" installs the package in /usr/local by default. Example upgrading pip:

$ python3 -m site
sys.path = [
    (...)
    '/usr/lib64/python36.zip',
    '/usr/lib64/python3.6',
    '/usr/lib64/python3.6/lib-dynload',
    '/usr/lib64/python3.6/site-packages',
    '/usr/lib/python3.6/site-packages',
]
(...)

$ sudo python3 -m pip install --upgrade pip
(...)

$ python3 -m site
sys.path = [
    (...)
    '/usr/lib64/python36.zip',
    '/usr/lib64/python3.6',
    '/usr/lib64/python3.6/lib-dynload',
    '/usr/local/lib/python3.6/site-packages',   <=== NEW PATH
    '/usr/lib64/python3.6/site-packages',
    '/usr/lib/python3.6/site-packages',
]
(...)


pip is installed in /usr/local:

$ sudo python3 -m pip uninstall pip
Found existing installation: pip 21.0.1
Uninstalling pip-21.0.1:
  Would remove:
    /usr/bin/pip3
    /usr/local/bin/pip
    /usr/local/bin/pip3
    /usr/local/bin/pip3.6
    /usr/local/lib/python3.6/site-packages/pip-21.0.1.dist-info/*
    /usr/local/lib/python3.6/site-packages/pip/*
Proceed (y/n)?

Comment 2 Victor Stinner 2021-03-11 11:42:59 UTC
bz#1937492 is the same issue for Python 3.8, I propose to discuss the issue here.

Comment 3 Christian Heimes 2021-03-11 11:44:10 UTC
/usr/local is also excluded for scripts in isolated mode (python3 -I):

# python3.6 -m pip install --prefix=/usr/local/ defusedxml
WARNING: Running pip install with root privileges is generally not a good idea. Try `__main__.py install --user` instead.
Collecting defusedxml
  Downloading https://files.pythonhosted.org/packages/07/6c/aa3f2f849e01cb6a001cd8554a88d4c77c5c1a31c95bdf1cf9301e6d9ef4/defusedxml-0.7.1-py2.py3-none-any.whl
Installing collected packages: defusedxml
Successfully installed defusedxml-0.7.1

# python3
Python 3.6.8 (default, Aug 18 2020, 08:33:21) 
[GCC 8.3.1 20191121 (Red Hat 8.3.1-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import defusedxml
>>> defusedxml
<module 'defusedxml' from '/usr/local/lib/python3.6/site-packages/defusedxml/__init__.py'>

# python3 -I
Python 3.6.8 (default, Aug 18 2020, 08:33:21) 
[GCC 8.3.1 20191121 (Red Hat 8.3.1-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import defusedxml
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'defusedxml'

Comment 4 Miro Hrončok 2021-03-11 12:16:22 UTC
Note that /usr/local/... is not in sys.path when RPM_BUILD_ROOT is set. Any chance this happens to you when building RPM packages?

Comment 5 Pat Riehecky 2021-03-11 14:04:27 UTC
Indeed, I did initially encounter this when building an RPM and thought I'd correctly traced it back to python.  I'm bootstraping a bunch of python module RPMs that have some interdependence, I'll open a ticket focused on that and try to keep this one focused on python itself.


It seems a bit odd that python3 -I excludes /usr/local/, that should be a root writable only location.  Was excluding /usr/local/ an 'EL' choice or did that come from upstream?

Comment 6 Miro Hrončok 2021-03-11 14:28:47 UTC
> Was excluding /usr/local/ an 'EL' choice or did that come from upstream?

From Fedora. Upstream only ever includes /usr/local/ when Python itself is installed into /usr/local/.

Comment 7 Pat Riehecky 2021-03-11 14:45:33 UTC
I'm wondering if that exclude matches expectations...

> isolated mode. This also implies -E and -s.
> In isolated mode sys.path contains neither the script’s directory nor the user’s site-packages directory.
> All PYTHON* environment variables are ignored, too.
> Further restrictions may be imposed to prevent the user from injecting malicious code.

/usr/local/ isn't traditionally considered a place where a user can write files (and thereby inject code).

If I'm trying to build a python environment with some additional (not in an RPM) python modules and run the app under `-I`, the most obvious solution is to use pip to install my modules as root.  However, those modules will go into /usr/local/ and not be accessible to to the app under `-I`.

With the exclusion of /usr/local/ by `-I`, but pip install defaulting to /usr/local, this seems to transform `-I` into "only use modules from RPMs".

Comment 8 Petr Viktorin 2021-03-11 14:50:47 UTC
Indeed, -I will isolate the interpreter to use system packages (i.e. RPMs) only.

Comment 9 Tomas Orsava 2021-03-11 16:24:19 UTC
For motivation why it's set up that way, see the relevant Fedora Change: https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe

Comment 10 Pat Riehecky 2021-03-11 21:03:08 UTC
Interesting read!

Tracked down to https://src.fedoraproject.org/rpms/python3/c/332b947dfc8d0f0d3a4525864b121d0f239beb4d?branch=master thanks to that link!

I'll confess a bit of surprise to see /usr/local dropped with 'ENABLE_USER_SITE` as I don't typically think of that as a user area.

Comment 11 Tomas Orsava 2021-03-12 10:47:41 UTC
(In reply to Pat Riehecky from comment #10)
> Interesting read!
> 
> Tracked down to
> https://src.fedoraproject.org/rpms/python3/c/
> 332b947dfc8d0f0d3a4525864b121d0f239beb4d?branch=master thanks to that link!
> 
> I'll confess a bit of surprise to see /usr/local dropped with
> 'ENABLE_USER_SITE` as I don't typically think of that as a user area.

Indeed, it's not the first thing that comes to mind when talking of a user area. However, in our long experience of people breaking their DNF and other RPM-installed software by sudo pip installing software, we resorted to this solution to isolate such critical software. And looking back, this change is now going on 3 years old, and it has achieved the stated goal of making usage of sudo pip much safer for users and their systems.

Comment 12 Miro Hrončok 2021-03-16 09:00:46 UTC
*** Bug 1937765 has been marked as a duplicate of this bug. ***

Comment 13 Miro Hrončok 2021-03-16 09:00:56 UTC
*** Bug 1937492 has been marked as a duplicate of this bug. ***

Comment 14 Miro Hrončok 2021-03-16 09:03:28 UTC
I am closing this as NOTABUG based on this comment: https://bugzilla.redhat.com/show_bug.cgi?id=1937765#c3

If you want to follow up on the discussion, may I suggest to move the the Fedora's Python mailing list? https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/


Note You need to log in before you can comment on or make changes to this bug.