When debugging another problem, I noticed in a build log of python3.11-3.11.0-1.fc38 that we compile the PYC files again and again. (I've noticed this because I saw -j0 in the log and was looking for the source -- it is hardcoded in the Makefile.) First, the files are compiled without -O/with -O/with -OO by the debug build via make install (commoninstall -> libinstall): PYTHONPATH=/builddir/build/BUILDROOT/python3.11-3.11.0-1.fc38.x86_64/usr/lib64/python3.11 LD_LIBRARY_PATH=/builddir/build/BUILD/Python-3.11.0/build/debug \ ./python -E -Wi /builddir/build/BUILDROOT/python3.11-3.11.0-1.fc38.x86_64/usr/lib64/python3.11/compileall.py \ -j0 -d /usr/lib64/python3.11 -f \ -x 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' \ /builddir/build/BUILDROOT/python3.11-3.11.0-1.fc38.x86_64/usr/lib64/python3.11 ... PYTHONPATH=/builddir/build/BUILDROOT/python3.11-3.11.0-1.fc38.x86_64/usr/lib64/python3.11 LD_LIBRARY_PATH=/builddir/build/BUILD/Python-3.11.0/build/debug \ ./python -E -Wi -O /builddir/build/BUILDROOT/python3.11-3.11.0-1.fc38.x86_64/usr/lib64/python3.11/compileall.py \ -j0 -d /usr/lib64/python3.11 -f \ -x 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' \ /builddir/build/BUILDROOT/python3.11-3.11.0-1.fc38.x86_64/usr/lib64/python3.11 ... PYTHONPATH=/builddir/build/BUILDROOT/python3.11-3.11.0-1.fc38.x86_64/usr/lib64/python3.11 LD_LIBRARY_PATH=/builddir/build/BUILD/Python-3.11.0/build/debug \ ./python -E -Wi -OO /builddir/build/BUILDROOT/python3.11-3.11.0-1.fc38.x86_64/usr/lib64/python3.11/compileall.py \ -j0 -d /usr/lib64/python3.11 -f \ -x 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' \ /builddir/build/BUILDROOT/python3.11-3.11.0-1.fc38.x86_64/usr/lib64/python3.11 (Then then same repeats for empty site-packages, so nothing really happens.) The whole thing repeats with the optimized Python, which should produce the same result, so it is useless and replaces the result of the debug build compilation: PYTHONPATH=/builddir/build/BUILDROOT/python3.11-3.11.0-1.fc38.x86_64/usr/lib64/python3.11 LD_LIBRARY_PATH=/builddir/build/BUILD/Python-3.11.0/build/optimized \ ./python -E -Wi /builddir/build/BUILDROOT/python3.11-3.11.0-1.fc38.x86_64/usr/lib64/python3.11/compileall.py \ -j0 -d /usr/lib64/python3.11 -f \ -x 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' \ /builddir/build/BUILDROOT/python3.11-3.11.0-1.fc38.x86_64/usr/lib64/python3.11 ... PYTHONPATH=/builddir/build/BUILDROOT/python3.11-3.11.0-1.fc38.x86_64/usr/lib64/python3.11 LD_LIBRARY_PATH=/builddir/build/BUILD/Python-3.11.0/build/optimized \ ./python -E -Wi -O /builddir/build/BUILDROOT/python3.11-3.11.0-1.fc38.x86_64/usr/lib64/python3.11/compileall.py \ -j0 -d /usr/lib64/python3.11 -f \ -x 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' \ /builddir/build/BUILDROOT/python3.11-3.11.0-1.fc38.x86_64/usr/lib64/python3.11 ... PYTHONPATH=/builddir/build/BUILDROOT/python3.11-3.11.0-1.fc38.x86_64/usr/lib64/python3.11 LD_LIBRARY_PATH=/builddir/build/BUILD/Python-3.11.0/build/optimized \ ./python -E -Wi -OO /builddir/build/BUILDROOT/python3.11-3.11.0-1.fc38.x86_64/usr/lib64/python3.11/compileall.py \ -j0 -d /usr/lib64/python3.11 -f \ -x 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' \ /builddir/build/BUILDROOT/python3.11-3.11.0-1.fc38.x86_64/usr/lib64/python3.11 (+ empty site-packages) In the end we call the bytecompilation from our spec manually (all three optimization levels at once): + LD_LIBRARY_PATH=/builddir/build/BUILDROOT/python3.11-3.11.0-1.fc38.x86_64/usr/lib64/python3.11/lib-dynload/:/builddir/build/BUILDROOT/python3.11-3.11.0-1.fc38.x86_64/usr/lib64 + /builddir/build/BUILDROOT/python3.11-3.11.0-1.fc38.x86_64/usr/bin/python3.11 -s -B -m compileall -f -j6 -o 0 -o 1 -o 2 -s /builddir/build/BUILDROOT/python3.11-3.11.0-1.fc38.x86_64 -p / /builddir/build/BUILDROOT/python3.11-3.11.0-1.fc38.x86_64 --hardlink-dupes All pyc files are overridden once again. I have not measured what amount of resources this takes, but it feels wasteful. (We should probably also contribute upstream not to call this three times, but to pass -o 0 -o 1 -o 2 as we do.) Considering we pass custom options (notably -s %{buildroot} -p / --hardlink-dupes) to the final bytecompilation, we cannot simply remove it from the spec. We either need to: - find a way to pass custom options for compileall to make install (and pass them, so we can skip our explicit bytecompilation, reducing the number of bytecompilations from 3 to 2) - or convince make install to not do this (and keep our custom bytecompilation only) - or both (and only let the optimized build bytecompile, with our custom options).
I proposed a simple change upstream allowing to replace "-j0" option with "--hardlink-dupes -j4" options. My change also merges the 3 compileall commands into a single command using "-o 0 -o 1 -o 2". PR: https://github.com/python/cpython/pull/99291 It should allow to avoid the additional compileall command in the specfile just to pass --hardlink-dupes option, since we can directly pass --hardlink-dupes in proposed COMPILEALL_OPTS variable in Makefile.
> It should allow to avoid the additional compileall command in the specfile just to pass --hardlink-dupes option, since we can directly pass --hardlink-dupes in proposed COMPILEALL_OPTS variable in Makefile. There ar more options we pass.
I see now that the COMPILEALL_OPTS would allow to pass more options. Cool. However, we still run make install twice, so having a way to disable compiling pycs entirely sounds better to me.
This bug appears to have been reported against 'rawhide' during the Fedora Linux 38 development cycle. Changing version to 38.
This message is a reminder that Fedora Linux 38 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora Linux 38 on 2024-05-21. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a 'version' of '38'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, change the 'version' to a later Fedora Linux version. Note that the version field may be hidden. Click the "Show advanced fields" button if you do not see it. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora Linux 38 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora Linux, you are encouraged to change the 'version' to a later version prior to this bug being closed.