Bug 531117

Summary: PATCH: support multiple python implementations in brp-python-bytecompile
Product: [Fedora] Fedora Reporter: Dave Malcolm <dmalcolm>
Component: rpmAssignee: Panu Matilainen <pmatilai>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: low    
Version: rawhideCC: amcnabb, ffesti, jnovy, pmatilai
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 557610 (view as bug list) Environment:
Last Closed: 2009-10-28 03:56:21 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Bug Depends On:    
Bug Blocks: 530636    
Description Flags
Patch to support multiple python interpreters within one build
Updated version of patch, also handling /usr/lib64 none

Description Dave Malcolm 2009-10-26 19:13:21 EDT
Description of problem:
As part of work towards:
I want to support deploying multiple python versions via RPM on one
machine.  Hence I want to make sure that when we bytecompile a .py file,
we use the correct python interpreter.

Patch posted on upstream mailing list here: http://lists.rpm.org/pipermail/rpm-maint/2009-October/002578.html
am also attaching here for convenience.

Python's runtime compiles .py source files to bytecode "code objects",
and saves a cache of this data as .pyc and .pyo files on the filesystem.
It looks for .pyc files before parsing a .py file, using the .pyc file
if it can, to avoid having to parse the file.

rpm.org has a script: "rpm/scripts/brp-python-bytecompile".  We invoke
this during postprocessing of an rpm build in Fedora and derived
distributions, so that we can ship pre-parsed bytecode files for .py
files in our package payloads.

The python runtime requires that a 4-byte "magic" ABI value stored in
the header of the pyc file equals that of the python runtime.  (this is
implemented in Python/import.c:check_compiled_module; see also the
importlib/_bootstrap.py implementation within Python 3; similarly a
4-byte mtime value is embedded, which has to match the mtime of the .py

If there's an ABI mismatch, the .pyc file is treated as stale, and
the .py file is parsed.  If this happens for a packaged python module,
the .pyc files aren't writable by non-root, and so python will typically
fail to write out an updated cache (if running as non-root), and will
constantly have to reparse the code, without any visible indication
apart from a big loss of performance.  Additionally, the Python syntax
changed a lot between Python 2 and 3, and picking the wrong interpreter
will often lead to syntax errors when byte-compiling the file.

I want to support deploying multiple python versions via RPM on one
machine.  Hence I want to make sure that when we bytecompile a .py file,
we use the correct python interpreter.

Currently, brp-python-bytecompile can take a single optional argument: a
path to the python interpreter, using /usr/bin/python as the default.

My first thought was to fixup our macros to pass along a "__python"
variable as an argument to the invocation of brp-python-bytecompile

However doing so assumes that every .py file within an rpmbuild is to be
processed using the same python runtime.  I don't think this is
sophisticated enough.  I'd like to be able to support having an srpm in
which multiple subpackages are emitted, each for a different python
runtime, all within the same build.  To handle this, some .py files
within RPM_BUILD_ROOT are handled by one runtime, and some by another.

I'm attaching a patch that covers this case.

For every directory of the form /usr/lib/python$VERSION/, the .py files
below it are assumed to be associated with the python interpreter
at /usr/bin/python$VERSION.  For all other .py files, they are
associated with a default python runtime, which is either the argument
to the script, or defaulting to /usr/bin/python (similar to the existing
behavior of the script).

I've also submitted a possible rpmlint test for verifying this in built
RPMs here:

which I've filed in rhbz as bug 531102 (see also bug 531082 for "file" detection of newer python bytecode).

[1] In Fedora we do this using "__os_install_post", which is defined
in /usr/lib/rpm/redhat/macros (from the redhat-rpm-config package),
which has the invocation of /usr/lib/rpm/brp-python-bytecompile
Comment 1 Dave Malcolm 2009-10-26 19:14:17 EDT
Created attachment 366174 [details]
Patch to support multiple python interpreters within one build
Comment 2 Dave Malcolm 2009-10-27 09:01:06 EDT
Created attachment 366260 [details]
Updated version of patch, also handling /usr/lib64
Comment 3 Andrew McNabb 2009-10-28 00:05:05 EDT
I looked at the patch, and it seems simple and well-commented.  I especially like that it makes brp-python-bytecompile pick the right Python version for /usr/lib*/pythonX.Y/ directories.  This makes it so package maintainers won't have to worry about picking a specific Python version.  I'm not sure if my review counts for anything, but I think this patch looks great.
Comment 4 Panu Matilainen 2009-10-28 03:56:21 EDT
The updated version is now in rpm-4.7.1-8.fc13 / rawhide.