Description of problem: Fedora 27 was modified to install modules into /usr/local when using "pip install uswsgi": * https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe * https://src.fedoraproject.org/rpms/python3/blob/master/f/00251-change-user-install-location.patch uwsgi embeds Python: it's dynamically linked to libpython to use the Python C API (ex: call Py_Initialize()). Fedora's downstream patch checks the name of the Python binary to only add /usr/local prefix if the Python binary path starts with "/usr/bin/python". Problem: uwsgi binary path doesn't start with "/usr/bin/python", but is "/usr/local/bin/uwsgi", /usr/local prefix isn't used, and so OpenStack services like Keystone cannot load their module (ex: "import keystone"). vstinner@devstack$ python3 Python 3.6.3 (default, Oct 9 2017, 12:07:10) >>> import sys, pprint >>> pprint.pprint(sys.path) ['', (...) '/usr/local/lib64/python3.6/site-packages', '/usr/local/lib/python3.6/site-packages', (...)] >>> import keystone >>> keystone <module 'keystone' from '/opt/stack/keystone/keystone/__init__.py'> uwsgi behaves differently, it's unable to import the keystone module: vstinner@devstack$ /usr/local/bin/uwsgi --procname-prefix keystone --ini /etc/keystone/keystone-uwsgi-public.ini (...) Traceback (most recent call last): File "/usr/local/bin/keystone-wsgi-public", line 6, in <module> from keystone.server.wsgi import initialize_public_application ModuleNotFoundError: No module named 'keystone' unable to load app 0 (mountpoint='') (callable not found or import error) (...) Version-Release number of selected component (if applicable): * Fedora 27 * python3: 3.6.3 How reproducible: Always Steps to Reproduce: 1. install uwsgi using: "sudo pip3 install uwsgi", it's installed in /usr/local/bin/uwsgi 2. install a uwsgi application like keystone 3. run a uwsgi application like keystone: uwsgi runs the application which is unable to import its Python modules
I understood that the intent of the Fedora change is to not load Python code from /usr/local when using system/platform python. Why using a whitelist approach? Why not doing the opposite, use a blacklist: don't add /usr/local if the binary is the system/platform python? Workaround patch: --- /usr/lib64/python3.6/site.py 2018-01-08 16:18:43.206511043 +0100 +++ /usr/lib64/python3.6/site.py 2018-01-08 16:19:10.374476306 +0100 @@ -332,7 +332,7 @@ and RPM build is not detected to make sudo pip installed packages visible. """ - if (ENABLE_USER_SITE and sys.executable.startswith("/usr/bin/python") + if (ENABLE_USER_SITE and sys.executable.startswith(("/usr/bin/python", "/usr/local/bin/uwsgi")) and 'RPM_BUILD_ROOT' not in os.environ): PREFIXES.insert(0, "/usr/local") for sitedir in getsitepackages(prefixes):
The blacklist might make sense. Tomáš, you're probably most up to date now, what do you think?
FYI it's not currently possible to run DevStack on Fedora 27, there are at least two other issues: * https://bugs.launchpad.net/devstack/+bug/1741901 * https://bugs.launchpad.net/devstack/+bug/1740194
FYI I don't think there's a need for black/white list anymore. There's no more system python, or platform python either (there is actually platform python in Fedora 27, removed in Fedora 28, but nothing uses is).
There is a FTBFS due to https://fedoraproject.org/wiki/Changes/SunRPCRemovalhttps://bugs.python.org/issue32521 https://bugs.python.org/issue32007 So we cannot fix this right away.
(In reply to Miro Hrončok from comment #5) > https://fedoraproject.org/wiki/Changes/SunRPCRemovalhttps://bugs.python.org/ > issue32521 https://fedoraproject.org/wiki/Changes/SunRPCRemoval https://bugs.python.org/issue32521
I've created a new patch that does not use the whitelist and solely depends on detecting the rpm build environment. I'm currently doing testing (F27 scratch build: https://koji.fedoraproject.org/koji/taskinfo?taskID=24098598).
It's your then ;)
There is a problem with the patch that removes the whitelist. While it fixes the reported issue, it causes venv to break down, and after some investigation I believe that's exactly why the whitelist was put there in the first place. I've created a different patch that broadens the whitelist to any executables in /usr/bin/ and /usr/local/bin, but it's not an ideal solution. I've asked mcyprian to look into it as he's the original author of the change.
"... I believe that's exactly why the whitelist was put there in the first place. I've created a different patch that broadens the whitelist to any executables in /usr/bin/ and /usr/local/bin, but it's not an ideal solution." Can't we special case site when running in a virtual environment instead? See venv() function of Lib/site.py.
That's right. It seemed to be elegant solution to exclude virtual environments and isolate system/platform python at the same time. As far as platform python won't work as planned, it might be sufficient for now to simply exclude venvs. I am working on the patch.
My approach isn't working, the virtualenvs are still broken. Can you please describe your latest idea of patching site.py in more details Victor?
Actually simple detection of virtual environments might work, I had a mistake in condition before. I wrote this patch [0] and here [1] is a copr with python3 containing the new patch. Can you please check whether this really resolves the issue? [0] https://github.com/mcyprian/python3/blob/sudo-pip-patch/00251-change-user-install-location.patch [1] https://copr.fedorainfracloud.org/coprs/mcyprian/sudo-pip-change-patch/builds/
python3-3.6.4-8.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-37591178fc
python3-3.6.4-8.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-37591178fc
python37-3.7.0-0.9.b1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2018-c743211a24
python37-3.7.0-0.9.b1.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-c743211a24
python3-3.6.4-8.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.
python37-3.7.0-0.9.b1.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.
Sorry, I couldn't test the fix, but it seems like a fix was released anyway: thanks! I will test it.