Currently, we utilize byte compilation during rpmbuild in 2 ways: via /usr/lib/rpm/brp-python-bytecompile via %py_byte_compile macro defined in macros.pybytecompile3.X (and in a similar way in python3.spec) See https://github.com/rpm-software-management/rpm/blob/master/scripts/brp-python-bytecompile And https://src.fedoraproject.org/rpms/python3/blob/master/f/macros.pybytecompile3.6 And https://bugzilla.redhat.com/show_bug.cgi?id=1484993 Both ways do it a bit differently. The brp script invokes compileall for each file, the macro says the following in the comment: # Python's compile_all module only works on directories, and requires a max # recursion depth Ans uses the py_compile module on an unreadable oneliner. We should figure out what exactly is missing from the compileall module, so we might eventually just call: python -m compileall ... That would allow us to: drop complicated and error prone downstream only code that we drag since who knows what. (Note that rpm can benefit from this as well, as it maintains the brp-python-bytecompile script). Eventually we could also make this parallel, faster, whatever. First steps: Figure out how it's done in other RPM distros and DEB distros. Possible scenario if nothing useful is found in other distros (hopefully not likely): Try to fix the compileall module (either wrap it or fork it), release something on PyPI that does this and works with all Pythons we currently ship. Package it as RPM, depend on it from python2-devel, python3-devel. Switch the macro/script to use it. If possible, put this back to PyPy and CPython upstreams. (Note that we might be beyond a point where this would be accepted in Python 2, but this tool can be linked from the docs (as are requests).) This is a followup for https://src.fedoraproject.org/rpms/python37/pull-request/3#comment-3675 (Note that I'm assigning this to me, but i have very little time to start working on this, so this will probably be here for some time. Feel free to start digging.)
Note that any notion of parallelism only makes sense if and only if the particular outputs will be presented in an non-interleaved (hence serialized and eligible) form. See also discussion at https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/12
It seems that: - OpenSUSE uses brp_python_bytecompile (compileall) implementation from upstream RPM. - Ubuntu and Debian do byte-compilation in postinstall script (auto-generated) and they use py_compile in it. We discussed it with Petr and the plan is: 1. Fork upstream compileall module and: - remove recursion depth limit - add a feature to strip a buildroot from compiled files (rhbz#1647212) 2. Make this module available on PyPI and in Fedora 3. Try to use it everywhere and make it universal for all byte-compilation use cases 4. If it'll work, try to merge the changes back to Python upstream 5. profit :) We won't prepare the same solution for Python 2 because: - The upstream compileall module is not Python 2/3 compatible and if we want it to be merged back to upstream one day, we should focus on Python 3 - Python 2 is very close to EOL and might be removed from Fedora before we finish this so it's okay to use the workaround. Anything else we should consider?
Consider that %py_byte_compile and brp_python_bytecompile support all Python versions we ship ATM (not sure about 2.6). If we deliver a Python 3 only solution, I OK with it if there is a fallback for 2.7.
It would be nice to keep it 3.6+ for EPEL (go easy bleeding edge stuff :=) Is %py_byte_compile actually used for the older Pythons? AFAIK pythonXY do their own bytecompilation, and we don't do other packages for them. The obvious fallback would be using the current solution for py2.
> It would be nice to keep it 3.6+ for EPEL (go easy bleeding edge stuff :=) 3.4+ for EPEL? 3.5+ for PyPy? ... > Is %py_byte_compile actually used for the older Pythons? Not within Fedora official packages. > we don't do other packages for them. Yes, **we** don't.
I see. Fall back to the current method for all older Pythons, then.
New Python module is ready on [Github] and [PyPI]. Proof of concept of how this can work in %py_byte_compile macro is ready in [copr]. We are now working on how to make it available for all Pythons and use it in %py_byte_compile macro and brp-python-bytecompile script. [Github] https://github.com/frenzymadness/compileall2 [PyPI] https://pypi.org/project/compileall2/ [copr] https://copr.fedorainfracloud.org/coprs/lbalhar/compileall2/
An updated version of RPM macros with compileall2 is ready in COPR. https://copr.fedorainfracloud.org/coprs/lbalhar/compileall2/builds/ The next step is to switch brp-python-bytecompile to use compileall2 for Python 3 as well.
PR for %py_byte_compile macro: https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/25
That pull request was merged, and it's also used in redhat-rpm-config now -- https://src.fedoraproject.org/rpms/redhat-rpm-config/c/5437dfca813421605f635ff4475ca4a79217a6d6?branch=master The commit got in with less testing than we wanted. We decided not to revert it, but we'll be on the lookout check for mass rebuild failures.
The mass rebuild happened with compileall2, and there was no obvious breakage. The next goal is to get compileall2 merged back into upstream compileall.
compileall2 is merged upstream. Next steps: - merge changes from now-upstream compileall back to compileall2 - merge those changes into python-rpm-macros
(In reply to Petr Viktorin from comment #12) > compileall2 is merged upstream. > Next steps: > - merge changes from now-upstream compileall back to compileall2 Pull request: https://github.com/fedora-python/compileall2/pull/13 > - merge those changes into python-rpm-macros I'll release a new version and update the macros package.
Compileall2 version 0.6.0 with all enhancenments from Python stdlib is available. PR for python-rpm-macros: https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/36
Next steps: - Use stdlib compileall in python-rpm-macros for Python 3.9+ - Remove the bundled script from python-rpm-macros when Python 3.8 is dropped (far future)
- Make sure the python3 spec actually uses compileall.
(In reply to Petr Viktorin from comment #16) > - Make sure the python3 spec actually uses compileall. python3 package does not use compileall2. PR to fix this: https://src.fedoraproject.org/rpms/python3/pull-request/162
> Use stdlib compileall in python-rpm-macros for Python 3.9+ Note: We could use the stdlib compileall module on 3.9 instead of our own compileall2, but I'd like us to keep the one we have control over for now, in case we will have --hardlink-dupes sooner than upstream. Once --hardlink-dupes is merged or not accepted upstream, we can re-evaluate.
Let's test --hardlink-dupes in Fedora/compileall2 until f32 is released, then propose the change to upstream compileall.
Upstream discussion: https://discuss.python.org/t/compileall-option-to-hardlink-duplicate-optimization-levels-bytecode-cache-files/3014
Upstream bug: https://bugs.python.org/issue40495 Upstream PR: https://github.com/python/cpython/pull/19901
hardlink-dupes is part of Python 3.9, so we should switch to it for 3.9+