Description of problem: The dnf CLI uses a shebang with python -OO. (Which means, the second level of python optimization). This is a dangerous construct that shouldn't be used in system pacakges as it removes docstrings. If by packaging mistake or user interaction (like hotpatching the library modules on their system), .pyos for the .py files are not included on the user's system, then when dnf is run as root, dnf would write out .pyo files with no docstrings for those modules. At that point, code unrelated to dnf that relies on docstrings being present in those python modules could be broken. Commandline optimization switches do not provide much (if any) speedup for python code anyway. The -O and -OO commandline switches only remove asserts and remove docstrings respectively. Version-Release number of selected component (if applicable): dnf-0.6.4-5.fc21.noarch How reproducible: Everytime Steps to Reproduce: 1. head -1 /usr/bin/dnf 2. 3. Actual results: #!/usr/bin/python -OO Expected results: #!/usr/bin/python Additional info: Using -O will only remove asserts and is what we do when we pre-generate our .pyo files normally. So it is safe to use that in a shebang line. But neither optimization level provides much (if any) speedup so we traditionally use #!/usr/bin/python without either -O switch.
It actually seems that it really provides some speedup: https://github.com/rpm-software-management/dnf/commit/c0ab927c5e4f7c45d18a0776c6d5380539d711da Anyway, how can a code rely on docstrings and why it would do that? BTW, DNF's docstrings definitely shouldn't be used anywhere. They are not even useful in DNF itself since they are mostly outdated (inherited from YUM).
The commit comment says it saves less than 100 ms.... that may be some speedup but it's hardly a major one :-) Any code can rely on docstrings because python makes them available for code to use via __doc__. There are both valid uses and abuses of docstrings in the wild. IDEs use them for dynamically showing information on functions. Some places store data about an object in them and then use the information at runtime (a quick google search shows just how the spectrum varies from use to abuse... the ranger file manager displays online help stored in docstrings which is somewhat valid. Some business is storing their actual SQL queries in docstrings on objects. The objects execute the SQL in the docstrings at runtime... clearly an abuse but yet another runtime use in the wild.) DNF's own docstrings are only a small part of the problem here. If one of dnf's python dependencies doesn't have an up-to-date .pyo file (for instance, a local sysadmin applies a hotfix to pygpgme), then when dnf is run and imports that dependency, python will rewrite the .pyo file. Since dnf needs to be run as root (in its makecache mode or by the sysadmin to manage the packages on the system) dnf will have the necessary permissions to write the .pyo. Later, if someone runs a program that needs to use the docstrings from pygpgme's .pyo files, they will be missing. tracking down the source of the problem will prove elusive since the bug won't be in the program the user is using or in pygpgme (either the system packaged version from fedora or the hotfix that the user applied).
Well, if these examples are valid uses of docstrings, I think that the bug should be filled against Python. I believe that this feature is designed precisely for being enabled when the code is run in production since I cannot imagine any other usage. If it makes some programs unusable than the feature is useless/wrong. Or at least there should be a Fedora guideline that forbids -OO. I'm for closing this as NOTABUG. Maybe my colleagues have a different opinion...
The main practical benefit of -OO is actually the memory reduction - there's no point consuming memory for docstrings when you don't need them at the runtime. However, for Python 3.4 and earlier, using -OO in a shared environment with other code that uses -O instead *is* a bug, because the two optimisation levels conflict on the ".pyo" extension when caching bytecode files. Since IDEs (et al) need the docstrings, that means general purpose environments (like a Linux distro) need to disallow the use of -OO for execution of Python programs. PEP 488 fixes that collision for Python 3.5+ by including the optimisation level in the generated filenames used by the bytecode cache: https://www.python.org/dev/peps/pep-0488/ I'd suggest adjusting the Python packaging policy to state the following: * -OO should never be used in Python 2 shebang lines (or other Python invocations) * -OO should never be used in Python 3 shebang lines (or other Python invocations) for Python 3.4 and earlier In the context of current Fedora, that means this bug report is valid, and dnf should stop using -OO for the time being. That then raises the question of "When will Fedora switch to Python 3.5?". We haven't actually set a target release yet, but the two immediately relevant schedule docs are: Fedora 23: https://fedoraproject.org/wiki/Releases/23/Schedule Python 3.5: https://www.python.org/dev/peps/pep-0478/ The upstream 3.5rc1 release is due on August 9, the final release is due on September 13. To switch in F23 that would mean: * getting a system-wide change for a Python 3 upgrade approved by the F23 deadline on Jun 26 * getting a beta release incorporated by the testability deadline on July 28 (this would correspond to 3.5b3 upstream, which is due for release on July 5) * F23 Alpha would ship with a Python 3.5 beta release * F23 Beta would ship with a Python 3.5 release candidate * F23 final would ship with the Python 3.5.0 final release I'll start a thread on the Fedora python-devel list about the idea of pursuing that more aggressive Python 3.5 adoption cycle, rather than waiting until Fedora 24.
OK, makes sense. So yes, DNF should work around the Python issues for the time being. I've filed a request for enhancing the guidelines to have the workaround supported by Fedora: https://fedorahosted.org/fpc/ticket/542
This message is a reminder that Fedora 21 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 21. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '21'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 21 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 21 changed to end-of-life (EOL) status on 2015-12-01. Fedora 21 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.
dnf-plugins-core-0.1.15-1.fc23 dnf-1.1.5-1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-c72cd24fa0
dnf-plugins-core-0.1.15-1.fc22 dnf-1.1.5-1.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2015-1bb112ccc6
dnf-1.1.5-1.fc23, dnf-plugins-core-0.1.15-1.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report. If you want to test the update, you can install it with $ su -c 'dnf --enablerepo=updates-testing update dnf dnf-plugins-core' You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-c72cd24fa0
This bug is currently assigned to an unsupported release. If you think this bug is still valid and should remain open, please re-assign it to a supported release (F22, F23) or to rawhide. Bugs which will be assigned to an unsupported release are going to be closed as EOL (End Of Life) on January 26th, 2016.
dnf-1.1.5-1.fc22, dnf-plugins-core-0.1.15-1.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-1bb112ccc6
dnf-1.1.5-1.fc23, dnf-plugins-core-0.1.15-1.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.
dnf-1.1.5-1.fc22, dnf-plugins-core-0.1.15-1.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.