Bug 1532287 - uwsgi isn't aware of /usr/local on Fedora 27
Summary: uwsgi isn't aware of /usr/local on Fedora 27
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: python3
Version: 27
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Michal Cyprian
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-01-08 15:17 UTC by Victor Stinner
Modified: 2018-03-01 15:02 UTC (History)
12 users (show)

Fixed In Version: python3-3.6.4-8.fc27
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-02-20 17:14:28 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Victor Stinner 2018-01-08 15:17:17 UTC
Description of problem:

Fedora 27 was modified to install modules into /usr/local when using "pip install uswsgi":

* https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe
* https://src.fedoraproject.org/rpms/python3/blob/master/f/00251-change-user-install-location.patch

uwsgi embeds Python: it's dynamically linked to libpython to use the Python C API (ex: call Py_Initialize()).

Fedora's downstream patch checks the name of the Python binary to only add /usr/local prefix if the Python binary path starts with "/usr/bin/python".

Problem: uwsgi binary path doesn't start with "/usr/bin/python", but is "/usr/local/bin/uwsgi", /usr/local prefix isn't used, and so OpenStack services like Keystone cannot load their module (ex: "import keystone").


vstinner@devstack$ python3
Python 3.6.3 (default, Oct  9 2017, 12:07:10) 
>>> import sys, pprint
>>> pprint.pprint(sys.path)
['',
 (...)
 '/usr/local/lib64/python3.6/site-packages',
 '/usr/local/lib/python3.6/site-packages',
 (...)]
>>> import keystone
>>> keystone
<module 'keystone' from '/opt/stack/keystone/keystone/__init__.py'>


uwsgi behaves differently, it's unable to import the keystone module:

vstinner@devstack$ /usr/local/bin/uwsgi --procname-prefix keystone --ini /etc/keystone/keystone-uwsgi-public.ini
(...)
Traceback (most recent call last):
  File "/usr/local/bin/keystone-wsgi-public", line 6, in <module>
    from keystone.server.wsgi import initialize_public_application
ModuleNotFoundError: No module named 'keystone'
unable to load app 0 (mountpoint='') (callable not found or import error)
(...)


Version-Release number of selected component (if applicable):

* Fedora 27
* python3: 3.6.3


How reproducible:

Always


Steps to Reproduce:

1. install uwsgi using: "sudo pip3 install uwsgi", it's installed in /usr/local/bin/uwsgi
2. install a uwsgi application like keystone
3. run a uwsgi application like keystone: uwsgi runs the application which is unable to import its Python modules

Comment 1 Victor Stinner 2018-01-08 15:24:09 UTC
I understood that the intent of the Fedora change is to not load Python code from /usr/local when using system/platform python. Why using a whitelist approach? Why not doing the opposite, use a blacklist: don't add /usr/local if the binary is the system/platform python?


Workaround patch:

--- /usr/lib64/python3.6/site.py        2018-01-08 16:18:43.206511043 +0100
+++ /usr/lib64/python3.6/site.py        2018-01-08 16:19:10.374476306 +0100
@@ -332,7 +332,7 @@
     and RPM build is not detected to make sudo pip installed packages visible.
 
     """
-    if (ENABLE_USER_SITE and sys.executable.startswith("/usr/bin/python")
+    if (ENABLE_USER_SITE and sys.executable.startswith(("/usr/bin/python", "/usr/local/bin/uwsgi"))
         and 'RPM_BUILD_ROOT' not in os.environ):
         PREFIXES.insert(0, "/usr/local")
     for sitedir in getsitepackages(prefixes):

Comment 2 Petr Viktorin (pviktori) 2018-01-08 15:32:20 UTC
The blacklist might make sense.
Tomáš, you're probably most up to date now, what do you think?

Comment 3 Victor Stinner 2018-01-08 16:46:32 UTC
FYI it's not currently possible to run DevStack on Fedora 27, there are at least two other issues:

* https://bugs.launchpad.net/devstack/+bug/1741901
* https://bugs.launchpad.net/devstack/+bug/1740194

Comment 4 Miro Hrončok 2018-01-08 21:31:06 UTC
FYI I don't think there's a need for black/white list anymore. There's no more system python, or platform python either (there is actually platform python in Fedora 27, removed in Fedora 28, but nothing uses is).

Comment 5 Miro Hrončok 2018-01-09 16:38:31 UTC
There is a FTBFS due to https://fedoraproject.org/wiki/Changes/SunRPCRemovalhttps://bugs.python.org/issue32521 https://bugs.python.org/issue32007

So we cannot fix this right away.

Comment 7 Tomas Orsava 2018-01-10 14:26:15 UTC
I've created a new patch that does not use the whitelist and solely depends on detecting the rpm build environment.

I'm currently doing testing (F27 scratch build: https://koji.fedoraproject.org/koji/taskinfo?taskID=24098598).

Comment 8 Miro Hrončok 2018-01-10 14:31:08 UTC
It's your then ;)

Comment 9 Tomas Orsava 2018-01-18 11:38:37 UTC
There is a problem with the patch that removes the whitelist. While it fixes the reported issue, it causes venv to break down, and after some investigation I believe that's exactly why the whitelist was put there in the first place. I've created a different patch that broadens the whitelist to any executables in /usr/bin/ and /usr/local/bin, but it's not an ideal solution. 

I've asked mcyprian to look into it as he's the original author of the change.

Comment 10 Victor Stinner 2018-01-22 12:00:06 UTC
"... I believe that's exactly why the whitelist was put there in the first place. I've created a different patch that broadens the whitelist to any executables in /usr/bin/ and /usr/local/bin, but it's not an ideal solution."

Can't we special case site when running in a virtual environment instead? See venv() function of Lib/site.py.

Comment 11 Michal Cyprian 2018-01-23 15:18:37 UTC
That's right. It seemed to be elegant solution to exclude virtual environments and isolate system/platform python at the same time. As far as platform python won't work as planned, it might be sufficient for now to simply exclude venvs. I am working on the patch.

Comment 12 Michal Cyprian 2018-01-30 14:50:28 UTC
My approach isn't working, the virtualenvs are still broken. Can you please describe your latest idea of patching site.py in more details Victor?

Comment 13 Michal Cyprian 2018-02-02 11:44:38 UTC
Actually simple detection of virtual environments might work, I had a mistake in condition before. I wrote this patch [0] and here [1] is a copr with python3 containing the new patch. Can you please check whether this really resolves the issue?


[0] https://github.com/mcyprian/python3/blob/sudo-pip-patch/00251-change-user-install-location.patch
[1] https://copr.fedorainfracloud.org/coprs/mcyprian/sudo-pip-change-patch/builds/

Comment 14 Fedora Update System 2018-02-09 15:53:41 UTC
python3-3.6.4-8.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-37591178fc

Comment 15 Fedora Update System 2018-02-13 07:58:36 UTC
python3-3.6.4-8.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-37591178fc

Comment 16 Fedora Update System 2018-02-17 22:54:04 UTC
python37-3.7.0-0.9.b1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2018-c743211a24

Comment 17 Fedora Update System 2018-02-18 18:13:45 UTC
python37-3.7.0-0.9.b1.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-c743211a24

Comment 18 Fedora Update System 2018-02-20 17:14:28 UTC
python3-3.6.4-8.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

Comment 19 Fedora Update System 2018-02-27 16:55:12 UTC
python37-3.7.0-0.9.b1.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Comment 20 Victor Stinner 2018-03-01 15:02:30 UTC
Sorry, I couldn't test the fix, but it seems like a fix was released anyway: thanks! I will test it.


Note You need to log in before you can comment on or make changes to this bug.