Description of problem: The Python standard library distutils module will be removed from Python 3.12+ https://peps.python.org/pep-0632/ As preparatory work, we build all python packages in a Copr repository with Python 3.11 sans distutils. https://copr.fedorainfracloud.org/coprs/g/python/python-without-distutils/ firefox fails to build: + ./mach build -v created virtual environment CPython3.11.0.candidate.2-64 in 43ms creator CPython3Posix(dest=/builddir/.mozbuild/srcdirs/firefox-105.0.2-589d8dcd2428/_virtualenvs/mach, clear=False, no_vcs_ignore=False, global=False) activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/__main__.py", line 29, in <module> from pip._internal.cli.main import main as _main File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/main.py", line 9, in <module> from pip._internal.cli.autocompletion import autocomplete File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/autocompletion.py", line 10, in <module> from pip._internal.cli.main_parser import create_main_parser File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/main_parser.py", line 8, in <module> from pip._internal.cli import cmdoptions File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/cmdoptions.py", line 23, in <module> from pip._internal.cli.parser import ConfigOptionParser File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/parser.py", line 12, in <module> from pip._internal.configuration import Configuration, ConfigurationError File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/configuration.py", line 27, in <module> from pip._internal.utils.misc import ensure_dir, enum File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/utils/misc.py", line 42, in <module> from pip._internal.locations import get_major_minor_version, site_packages, user_site File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/locations/__init__.py", line 14, in <module> from . import _distutils, _sysconfig File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/locations/_distutils.py", line 9, in <module> from distutils.cmd import Command as DistutilsCommand ModuleNotFoundError: No module named 'distutils' Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/__main__.py", line 29, in <module> from pip._internal.cli.main import main as _main File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/main.py", line 9, in <module> from pip._internal.cli.autocompletion import autocomplete File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/autocompletion.py", line 10, in <module> from pip._internal.cli.main_parser import create_main_parser File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/main_parser.py", line 8, in <module> from pip._internal.cli import cmdoptions File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/cmdoptions.py", line 23, in <module> from pip._internal.cli.parser import ConfigOptionParser File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/parser.py", line 12, in <module> from pip._internal.configuration import Configuration, ConfigurationError File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/configuration.py", line 27, in <module> from pip._internal.utils.misc import ensure_dir, enum File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/utils/misc.py", line 42, in <module> from pip._internal.locations import get_major_minor_version, site_packages, user_site File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/locations/__init__.py", line 14, in <module> from . import _distutils, _sysconfig File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/locations/_distutils.py", line 9, in <module> from distutils.cmd import Command as DistutilsCommand ModuleNotFoundError: No module named 'distutils' Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/__main__.py", line 29, in <module> from pip._internal.cli.main import main as _main File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/main.py", line 9, in <module> from pip._internal.cli.autocompletion import autocomplete File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/autocompletion.py", line 10, in <module> from pip._internal.cli.main_parser import create_main_parser File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/main_parser.py", line 8, in <module> from pip._internal.cli import cmdoptions File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/cmdoptions.py", line 23, in <module> from pip._internal.cli.parser import ConfigOptionParser File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/parser.py", line 12, in <module> from pip._internal.configuration import Configuration, ConfigurationError File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/configuration.py", line 27, in <module> from pip._internal.utils.misc import ensure_dir, enum File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/utils/misc.py", line 42, in <module> from pip._internal.locations import get_major_minor_version, site_packages, user_site File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/locations/__init__.py", line 14, in <module> from . import _distutils, _sysconfig File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/locations/_distutils.py", line 9, in <module> from distutils.cmd import Command as DistutilsCommand ModuleNotFoundError: No module named 'distutils' Version-Release number of selected component: 105.0.2-1 How reproducible: Build firefox without distutils Steps to Reproduce: 1. fedpkg clone firefox && cd firefox 2. mock -r fedora-rawhide-x86_64 init 3. mock -r fedora-rawhide-x86_64 install python3-devel 4. sudo rm -rf /var/lib/mock/fedora-rawhide-x86_64/root/usr/lib64/python3.11/distutils/ 4. fedpkg mockbuild -N Actual results: fails to build Expected results: builds fine
Can you guide me how to fix that please? Do we need to require it explicitly or so? Thanks.
There is nothing to require explicitly. The module is gone. Some packages might get away by using setuptools instead, but even when setuptools is BuildRequired, firefox does not build. I could try to help you replace distutils usage in firefox with sysconfig or setuptools, but it appears from the tracebacks that this problem comes from an old version of bundled pip. I've inspected the source and it seems firefox bundles pip 21.2.4. My guide to fixing this is: don't bundle pip, use python3-pip. If it is not possible, upgrade the bundled pip to 22.2+ or backport: https://github.com/pypa/pip/commit/0e0645253041bd3b953fd33b6f393a2e383ebb5a https://github.com/pypa/pip/commit/db4751595867db1d938df7183a60dfb15fa0d708 (this might not be needed, but it won't hurt)
Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.12: https://copr.fedorainfracloud.org/coprs/python3.12/ Please use this copr for testing rather than https://copr.fedorainfracloud.org/coprs/g/python/python-without-distutils/ If your package is listed in [0], you may workaround this issue by BuildRequiring python-setuptools, note that adding such BuildRequires might however hide some transitive dependency problem, if the distutils import comes from a dependency. Cooperation with upstream is recommended. Additional context [1]. [0] https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/message/6BHNAWHE7M5VY3YQVJLOYHLY4M7KIFFN/ [1] https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/thread/N6ITYHLRWIDNYNXGPYG2ZHF3ZLQWZN7L/ https://docs.python.org/3.12/whatsnew/3.12.html For all our attempts to build firefox with Python 3.12, see: https://copr.fedorainfracloud.org/coprs/python3.12/package/firefox/ Let us know here if you have any questions. Python 3.12 is planned to be included in Fedora 39. To make that update smoother, we're building Fedora packages with all pre-releases of Python 3.12. A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon. We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.
This bug appears to have been reported against 'rawhide' during the Fedora Linux 38 development cycle. Changing version to 38.
https://koschei.fedoraproject.org/package/firefox The current issue is: + mkdir -p objdir/_virtualenvs/init_py3 + cat ++ pwd + tar xf /builddir/build/SOURCES/mochitest-python.tar.gz + xvfb-run ./mach build -v + cat - ERROR: Exception: Traceback (most recent call last): File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper status = run_func(*args) ^^^^^^^^^^^^^^^ File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/commands/list.py", line 169, in run packages: "_ProcessedDists" = [ ^ File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/base.py", line 664, in <genexpr> return (d for d in it if d.canonical_name not in skip) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/base.py", line 612, in iter_all_distributions for dist in self._iter_distributions(): File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 176, in _iter_distributions for dist in finder.find_eggs(location): File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 144, in find_eggs yield from self._find_eggs_in_dir(location) File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 111, in _find_eggs_in_dir from pip._vendor.pkg_resources import find_distributions File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_vendor/pkg_resources/__init__.py", line 2164, in <module> register_finder(pkgutil.ImpImporter, find_on_path) ^^^^^^^^^^^^^^^^^^^ AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'? ERROR: Exception: Traceback (most recent call last): File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper status = run_func(*args) ^^^^^^^^^^^^^^^ File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/commands/list.py", line 169, in run packages: "_ProcessedDists" = [ ^ File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/base.py", line 664, in <genexpr> return (d for d in it if d.canonical_name not in skip) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/base.py", line 612, in iter_all_distributions for dist in self._iter_distributions(): File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 176, in _iter_distributions for dist in finder.find_eggs(location): File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 144, in find_eggs yield from self._find_eggs_in_dir(location) File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 111, in _find_eggs_in_dir from pip._vendor.pkg_resources import find_distributions File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_vendor/pkg_resources/__init__.py", line 2164, in <module> register_finder(pkgutil.ImpImporter, find_on_path) ^^^^^^^^^^^^^^^^^^^ AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'? ERROR: Exception: Traceback (most recent call last): File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper status = run_func(*args) ^^^^^^^^^^^^^^^ File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/commands/list.py", line 169, in run packages: "_ProcessedDists" = [ ^ File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/base.py", line 664, in <genexpr> return (d for d in it if d.canonical_name not in skip) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/base.py", line 612, in iter_all_distributions for dist in self._iter_distributions(): File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 176, in _iter_distributions for dist in finder.find_eggs(location): File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 144, in find_eggs yield from self._find_eggs_in_dir(location) File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 111, in _find_eggs_in_dir from pip._vendor.pkg_resources import find_distributions File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_vendor/pkg_resources/__init__.py", line 2164, in <module> register_finder(pkgutil.ImpImporter, find_on_path) ^^^^^^^^^^^^^^^^^^^ AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'? Mach and the build system store shared state in a common directory on the filesystem. The following directory will be created: /builddir/.mozbuild If you would like to use a different directory, rename or move it to your desired location, and set the MOZBUILD_STATE_PATH environment variable accordingly. Creating default state directory: /builddir/.mozbuild Creating local state directory: /builddir/.mozbuild/srcdirs/firefox-115.0-0be14ac973c8 Could not install glean-sdk, so telemetry will not be collected. Continuing. Could not install psutil, so telemetry will be missing some data. Continuing. Could not install zstandard, so zstd archives will not be possible to extract. Continuing. Traceback (most recent call last): File "/builddir/build/BUILD/firefox-115.0/./mach", line 125, in <module> main(sys.argv[1:]) File "/builddir/build/BUILD/firefox-115.0/./mach", line 117, in main mach = check_and_get_mach(os.path.dirname(os.path.realpath(__file__))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/builddir/build/BUILD/firefox-115.0/./mach", line 37, in check_and_get_mach return load_mach(dir_path, mach_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/builddir/build/BUILD/firefox-115.0/./mach", line 23, in load_mach return mach_initialize.initialize(dir_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/builddir/build/BUILD/firefox-115.0/build/mach_initialize.py", line 153, in initialize import mach.main File "/builddir/build/BUILD/firefox-115.0/python/mach/mach/main.py", line 11, in <module> import imp ModuleNotFoundError: No module named 'imp' The first issue can be solved by unbundling pip (or upgrading it). For firefox-115.0/python/mach/mach/main.py and ModuleNotFoundError: No module named 'imp', you could BuildRequire python3-zombie-imp as a stopgap measure, but you should really post to importlib.
This bug appears to have been reported against 'rawhide' during the Fedora Linux 39 development cycle. Changing version to 39.