Bug 1230820

Summary: /usr/bin/dnf using -OO
Product: [Fedora] Fedora Reporter: Toshio Ernie Kuratomi <a.badger>
Component: dnfAssignee: Packaging Maintenance Team <packaging-team-maint>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 21CC: jsilhan, jzeleny, mluscon, ncoghlan, packaging-team-maint, pnemade, tim.lauridsen, vmukhame
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: dnf-1.1.5-1.fc23 dnf-1.1.5-1.fc22 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-12-22 21:59:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Toshio Ernie Kuratomi 2015-06-11 14:56:18 UTC
Description of problem:

The dnf CLI uses a shebang with python -OO.  (Which means, the second level of python optimization).  This is a dangerous construct that shouldn't be used in system pacakges as it removes docstrings.

If by packaging mistake or user interaction (like hotpatching the library modules on their system), .pyos for the .py files are not included on the user's system, then when dnf is run as root, dnf would write out .pyo files with no docstrings for those modules.  At that point, code unrelated to dnf that relies on docstrings being present in those python modules could be broken.

Commandline optimization switches do not provide much (if any) speedup for python code anyway.  The -O and -OO commandline switches only remove asserts and remove docstrings respectively.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. head -1 /usr/bin/dnf

Actual results:
#!/usr/bin/python -OO

Expected results:

Additional info:

Using -O will only remove asserts and is what we do when we pre-generate our .pyo files normally.  So it is safe to use that in a shebang line.  But neither optimization level provides much (if any) speedup so we traditionally use #!/usr/bin/python without either -O switch.

Comment 1 Radek Holy 2015-06-11 15:05:59 UTC
It actually seems that it really provides some speedup:


Anyway, how can a code rely on docstrings and why it would do that? BTW, DNF's docstrings definitely shouldn't be used anywhere. They are not even useful in DNF itself since they are mostly outdated (inherited from YUM).

Comment 2 Toshio Ernie Kuratomi 2015-06-12 16:24:51 UTC
The commit comment says it saves less than 100 ms.... that may be some speedup but it's hardly a major one :-)

Any code can rely on docstrings because python makes them available for code to use via __doc__.

There are both valid uses and abuses of docstrings in the wild.  IDEs use them for dynamically showing information on functions.  Some places store data about an object in them and then use the information at runtime (a quick google search shows just how the spectrum varies from use to abuse... the ranger file manager displays online help stored in docstrings which is somewhat valid.  Some business is storing their actual SQL queries in docstrings on objects.  The objects execute the SQL in the docstrings at runtime... clearly an abuse but yet another runtime use in the wild.)

DNF's own docstrings are only a small part of the problem here.  If one of dnf's python dependencies doesn't have an up-to-date .pyo file (for instance, a local sysadmin applies a hotfix to pygpgme), then when dnf is run and imports that dependency, python will rewrite the .pyo file.  Since dnf needs to be run as root (in its makecache mode or by the sysadmin to manage the packages on the system) dnf will have the necessary permissions to write the .pyo.  Later, if someone runs a program that needs to use the docstrings from pygpgme's .pyo files, they will be missing.  tracking down the source of the problem will prove elusive since the bug won't be in the program the user is using or in pygpgme (either the system packaged version from fedora or the hotfix that the user applied).

Comment 3 Radek Holy 2015-06-12 18:45:07 UTC
Well, if these examples are valid uses of docstrings, I think that the bug should be filled against Python. I believe that this feature is designed precisely for being enabled when the code is run in production since I cannot imagine any other usage. If it makes some programs unusable than the feature is useless/wrong. Or at least there should be a Fedora guideline that forbids -OO. I'm for closing this as NOTABUG. Maybe my colleagues have a different opinion...

Comment 4 Nick Coghlan 2015-06-15 00:26:12 UTC
The main practical benefit of -OO is actually the memory reduction - there's no point consuming memory for docstrings when you don't need them at the runtime.

However, for Python 3.4 and earlier, using -OO in a shared environment with other code that uses -O instead *is* a bug, because the two optimisation levels conflict on the ".pyo" extension when caching bytecode files. Since IDEs (et al) need the docstrings, that means general purpose environments (like a Linux distro) need to disallow the use of -OO for execution of Python programs.

PEP 488 fixes that collision for Python 3.5+ by including the optimisation level in the generated filenames used by the bytecode cache: https://www.python.org/dev/peps/pep-0488/

I'd suggest adjusting the Python packaging policy to state the following:

* -OO should never be used in Python 2 shebang lines (or other Python invocations)
* -OO should never be used in Python 3 shebang lines (or other Python invocations) for Python 3.4 and earlier

In the context of current Fedora, that means this bug report is valid, and dnf should stop using -OO for the time being.

That then raises the question of "When will Fedora switch to Python 3.5?". We haven't actually set a target release yet, but the two immediately relevant schedule docs are:

Fedora 23: https://fedoraproject.org/wiki/Releases/23/Schedule
Python 3.5: https://www.python.org/dev/peps/pep-0478/

The upstream 3.5rc1 release is due on August 9, the final release is due on September 13. To switch in F23 that would mean:

* getting a system-wide change for a Python 3 upgrade approved by the F23 deadline on Jun 26
* getting a beta release incorporated by the testability deadline on July 28 (this would correspond to 3.5b3 upstream, which is due for release on July 5)
* F23 Alpha would ship with a Python 3.5 beta release
* F23 Beta would ship with a Python 3.5 release candidate
* F23 final would ship with the Python 3.5.0 final release

I'll start a thread on the Fedora python-devel list about the idea of pursuing that more aggressive Python 3.5 adoption cycle, rather than waiting until Fedora 24.

Comment 5 Radek Holy 2015-06-15 08:04:03 UTC
OK, makes sense. So yes, DNF should work around the Python issues for the time being.

I've filed a request for enhancing the guidelines to have the workaround supported by Fedora: https://fedorahosted.org/fpc/ticket/542

Comment 6 Fedora End Of Life 2015-11-04 10:18:08 UTC
This message is a reminder that Fedora 21 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 21. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '21'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 21 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 7 Fedora End Of Life 2015-12-02 13:41:46 UTC
Fedora 21 changed to end-of-life (EOL) status on 2015-12-01. Fedora 21 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 8 Fedora Update System 2015-12-18 17:46:42 UTC
dnf-plugins-core-0.1.15-1.fc23 dnf-1.1.5-1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-c72cd24fa0

Comment 9 Fedora Update System 2015-12-18 17:48:29 UTC
dnf-plugins-core-0.1.15-1.fc22 dnf-1.1.5-1.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2015-1bb112ccc6

Comment 10 Fedora Update System 2015-12-19 19:59:23 UTC
dnf-1.1.5-1.fc23, dnf-plugins-core-0.1.15-1.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update dnf dnf-plugins-core'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-c72cd24fa0

Comment 11 Jan Kurik 2015-12-22 11:28:28 UTC
This bug is currently assigned to an unsupported release. If you think this bug is still valid and should remain open, please re-assign it to a supported release (F22, F23) or to rawhide.

Bugs which will be assigned to an unsupported release are going to be closed as EOL (End Of Life) on January 26th, 2016.

Comment 12 Fedora Update System 2015-12-22 15:35:05 UTC
dnf-1.1.5-1.fc22, dnf-plugins-core-0.1.15-1.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-1bb112ccc6

Comment 13 Fedora Update System 2015-12-22 21:58:39 UTC
dnf-1.1.5-1.fc23, dnf-plugins-core-0.1.15-1.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.

Comment 14 Fedora Update System 2016-01-04 19:51:57 UTC
dnf-1.1.5-1.fc22, dnf-plugins-core-0.1.15-1.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.