Hide Forgot
Description of problem: As part of work towards: https://fedoraproject.org/wiki/Features/Python3F13 I want to support deploying multiple python versions via RPM on one machine. Hence I want to make sure that when we bytecompile a .py file, we use the correct python interpreter. Patch posted on upstream mailing list here: http://lists.rpm.org/pipermail/rpm-maint/2009-October/002578.html am also attaching here for convenience. Python's runtime compiles .py source files to bytecode "code objects", and saves a cache of this data as .pyc and .pyo files on the filesystem. It looks for .pyc files before parsing a .py file, using the .pyc file if it can, to avoid having to parse the file. rpm.org has a script: "rpm/scripts/brp-python-bytecompile". We invoke this during postprocessing of an rpm build in Fedora and derived distributions, so that we can ship pre-parsed bytecode files for .py files in our package payloads. The python runtime requires that a 4-byte "magic" ABI value stored in the header of the pyc file equals that of the python runtime. (this is implemented in Python/import.c:check_compiled_module; see also the importlib/_bootstrap.py implementation within Python 3; similarly a 4-byte mtime value is embedded, which has to match the mtime of the .py file). If there's an ABI mismatch, the .pyc file is treated as stale, and the .py file is parsed. If this happens for a packaged python module, the .pyc files aren't writable by non-root, and so python will typically fail to write out an updated cache (if running as non-root), and will constantly have to reparse the code, without any visible indication apart from a big loss of performance. Additionally, the Python syntax changed a lot between Python 2 and 3, and picking the wrong interpreter will often lead to syntax errors when byte-compiling the file. I want to support deploying multiple python versions via RPM on one machine. Hence I want to make sure that when we bytecompile a .py file, we use the correct python interpreter. Currently, brp-python-bytecompile can take a single optional argument: a path to the python interpreter, using /usr/bin/python as the default. My first thought was to fixup our macros to pass along a "__python" variable as an argument to the invocation of brp-python-bytecompile [1]. However doing so assumes that every .py file within an rpmbuild is to be processed using the same python runtime. I don't think this is sophisticated enough. I'd like to be able to support having an srpm in which multiple subpackages are emitted, each for a different python runtime, all within the same build. To handle this, some .py files within RPM_BUILD_ROOT are handled by one runtime, and some by another. I'm attaching a patch that covers this case. For every directory of the form /usr/lib/python$VERSION/, the .py files below it are assumed to be associated with the python interpreter at /usr/bin/python$VERSION. For all other .py files, they are associated with a default python runtime, which is either the argument to the script, or defaulting to /usr/bin/python (similar to the existing behavior of the script). I've also submitted a possible rpmlint test for verifying this in built RPMs here: https://www.zarb.org/pipermail/rpmlint-discuss/2009-October/000775.html which I've filed in rhbz as bug 531102 (see also bug 531082 for "file" detection of newer python bytecode). [1] In Fedora we do this using "__os_install_post", which is defined in /usr/lib/rpm/redhat/macros (from the redhat-rpm-config package), which has the invocation of /usr/lib/rpm/brp-python-bytecompile
Created attachment 366174 [details] Patch to support multiple python interpreters within one build
Created attachment 366260 [details] Updated version of patch, also handling /usr/lib64
I looked at the patch, and it seems simple and well-commented. I especially like that it makes brp-python-bytecompile pick the right Python version for /usr/lib*/pythonX.Y/ directories. This makes it so package maintainers won't have to worry about picking a specific Python version. I'm not sure if my review counts for anything, but I think this patch looks great.
The updated version is now in rpm-4.7.1-8.fc13 / rawhide.