Bug 531117 - PATCH: support multiple python implementations in brp-python-bytecompile
Summary: PATCH: support multiple python implementations in brp-python-bytecompile
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: rpm
Version: rawhide
Hardware: All
OS: Linux
low
medium
Target Milestone: ---
Assignee: Panu Matilainen
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: Python3F13
TreeView+ depends on / blocked
 
Reported: 2009-10-26 23:13 UTC by Dave Malcolm
Modified: 2009-10-28 07:56 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 557610 (view as bug list)
Environment:
Last Closed: 2009-10-28 07:56:21 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
Patch to support multiple python interpreters within one build (2.71 KB, patch)
2009-10-26 23:14 UTC, Dave Malcolm
no flags Details | Diff
Updated version of patch, also handling /usr/lib64 (2.71 KB, patch)
2009-10-27 13:01 UTC, Dave Malcolm
no flags Details | Diff

Description Dave Malcolm 2009-10-26 23:13:21 UTC
Description of problem:
As part of work towards:
  https://fedoraproject.org/wiki/Features/Python3F13
I want to support deploying multiple python versions via RPM on one
machine.  Hence I want to make sure that when we bytecompile a .py file,
we use the correct python interpreter.

Patch posted on upstream mailing list here: http://lists.rpm.org/pipermail/rpm-maint/2009-October/002578.html
am also attaching here for convenience.

Python's runtime compiles .py source files to bytecode "code objects",
and saves a cache of this data as .pyc and .pyo files on the filesystem.
It looks for .pyc files before parsing a .py file, using the .pyc file
if it can, to avoid having to parse the file.

rpm.org has a script: "rpm/scripts/brp-python-bytecompile".  We invoke
this during postprocessing of an rpm build in Fedora and derived
distributions, so that we can ship pre-parsed bytecode files for .py
files in our package payloads.

The python runtime requires that a 4-byte "magic" ABI value stored in
the header of the pyc file equals that of the python runtime.  (this is
implemented in Python/import.c:check_compiled_module; see also the
importlib/_bootstrap.py implementation within Python 3; similarly a
4-byte mtime value is embedded, which has to match the mtime of the .py
file).

If there's an ABI mismatch, the .pyc file is treated as stale, and
the .py file is parsed.  If this happens for a packaged python module,
the .pyc files aren't writable by non-root, and so python will typically
fail to write out an updated cache (if running as non-root), and will
constantly have to reparse the code, without any visible indication
apart from a big loss of performance.  Additionally, the Python syntax
changed a lot between Python 2 and 3, and picking the wrong interpreter
will often lead to syntax errors when byte-compiling the file.

I want to support deploying multiple python versions via RPM on one
machine.  Hence I want to make sure that when we bytecompile a .py file,
we use the correct python interpreter.

Currently, brp-python-bytecompile can take a single optional argument: a
path to the python interpreter, using /usr/bin/python as the default.

My first thought was to fixup our macros to pass along a "__python"
variable as an argument to the invocation of brp-python-bytecompile
[1].  

However doing so assumes that every .py file within an rpmbuild is to be
processed using the same python runtime.  I don't think this is
sophisticated enough.  I'd like to be able to support having an srpm in
which multiple subpackages are emitted, each for a different python
runtime, all within the same build.  To handle this, some .py files
within RPM_BUILD_ROOT are handled by one runtime, and some by another.

I'm attaching a patch that covers this case.

For every directory of the form /usr/lib/python$VERSION/, the .py files
below it are assumed to be associated with the python interpreter
at /usr/bin/python$VERSION.  For all other .py files, they are
associated with a default python runtime, which is either the argument
to the script, or defaulting to /usr/bin/python (similar to the existing
behavior of the script).

I've also submitted a possible rpmlint test for verifying this in built
RPMs here:
https://www.zarb.org/pipermail/rpmlint-discuss/2009-October/000775.html

which I've filed in rhbz as bug 531102 (see also bug 531082 for "file" detection of newer python bytecode).

[1] In Fedora we do this using "__os_install_post", which is defined
in /usr/lib/rpm/redhat/macros (from the redhat-rpm-config package),
which has the invocation of /usr/lib/rpm/brp-python-bytecompile

Comment 1 Dave Malcolm 2009-10-26 23:14:17 UTC
Created attachment 366174 [details]
Patch to support multiple python interpreters within one build

Comment 2 Dave Malcolm 2009-10-27 13:01:06 UTC
Created attachment 366260 [details]
Updated version of patch, also handling /usr/lib64

Comment 3 Andrew McNabb 2009-10-28 04:05:05 UTC
I looked at the patch, and it seems simple and well-commented.  I especially like that it makes brp-python-bytecompile pick the right Python version for /usr/lib*/pythonX.Y/ directories.  This makes it so package maintainers won't have to worry about picking a specific Python version.  I'm not sure if my review counts for anything, but I think this patch looks great.

Comment 4 Panu Matilainen 2009-10-28 07:56:21 UTC
The updated version is now in rpm-4.7.1-8.fc13 / rawhide.


Note You need to log in before you can comment on or make changes to this bug.