Bug 2135421 - firefox fails to build with Python 3.12: Bundles an outdated version of pip
Summary: firefox fails to build with Python 3.12: Bundles an outdated version of pip
Keywords:
Status: NEW
Alias: None
Product: Fedora
Classification: Fedora
Component: firefox
Version: 39
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Gecko Maintainer
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: PYTHON3.12 F39FTBFS F40FTBFS, RAWHIDEFTBFS
TreeView+ depends on / blocked
 
Reported: 2022-10-17 15:28 UTC by Miro Hrončok
Modified: 2023-08-16 07:06 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Mozilla Foundation 1743785 0 P2 NEW [meta] distutils is deprecated, refactor usages 2023-06-08 12:50:47 UTC

Description Miro Hrončok 2022-10-17 15:28:29 UTC
Description of problem:
The Python standard library distutils module will be removed from Python 3.12+

https://peps.python.org/pep-0632/

As preparatory work, we build all python packages in a Copr repository with Python 3.11 sans distutils.

https://copr.fedorainfracloud.org/coprs/g/python/python-without-distutils/

firefox fails to build:

+ ./mach build -v
created virtual environment CPython3.11.0.candidate.2-64 in 43ms
  creator CPython3Posix(dest=/builddir/.mozbuild/srcdirs/firefox-105.0.2-589d8dcd2428/_virtualenvs/mach, clear=False, no_vcs_ignore=False, global=False)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/__main__.py", line 29, in <module>
    from pip._internal.cli.main import main as _main
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/main.py", line 9, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/autocompletion.py", line 10, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/main_parser.py", line 8, in <module>
    from pip._internal.cli import cmdoptions
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/cmdoptions.py", line 23, in <module>
    from pip._internal.cli.parser import ConfigOptionParser
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/parser.py", line 12, in <module>
    from pip._internal.configuration import Configuration, ConfigurationError
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/configuration.py", line 27, in <module>
    from pip._internal.utils.misc import ensure_dir, enum
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/utils/misc.py", line 42, in <module>
    from pip._internal.locations import get_major_minor_version, site_packages, user_site
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/locations/__init__.py", line 14, in <module>
    from . import _distutils, _sysconfig
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/locations/_distutils.py", line 9, in <module>
    from distutils.cmd import Command as DistutilsCommand
ModuleNotFoundError: No module named 'distutils'
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/__main__.py", line 29, in <module>
    from pip._internal.cli.main import main as _main
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/main.py", line 9, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/autocompletion.py", line 10, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/main_parser.py", line 8, in <module>
    from pip._internal.cli import cmdoptions
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/cmdoptions.py", line 23, in <module>
    from pip._internal.cli.parser import ConfigOptionParser
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/parser.py", line 12, in <module>
    from pip._internal.configuration import Configuration, ConfigurationError
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/configuration.py", line 27, in <module>
    from pip._internal.utils.misc import ensure_dir, enum
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/utils/misc.py", line 42, in <module>
    from pip._internal.locations import get_major_minor_version, site_packages, user_site
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/locations/__init__.py", line 14, in <module>
    from . import _distutils, _sysconfig
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/locations/_distutils.py", line 9, in <module>
    from distutils.cmd import Command as DistutilsCommand
ModuleNotFoundError: No module named 'distutils'
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/__main__.py", line 29, in <module>
    from pip._internal.cli.main import main as _main
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/main.py", line 9, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/autocompletion.py", line 10, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/main_parser.py", line 8, in <module>
    from pip._internal.cli import cmdoptions
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/cmdoptions.py", line 23, in <module>
    from pip._internal.cli.parser import ConfigOptionParser
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/parser.py", line 12, in <module>
    from pip._internal.configuration import Configuration, ConfigurationError
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/configuration.py", line 27, in <module>
    from pip._internal.utils.misc import ensure_dir, enum
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/utils/misc.py", line 42, in <module>
    from pip._internal.locations import get_major_minor_version, site_packages, user_site
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/locations/__init__.py", line 14, in <module>
    from . import _distutils, _sysconfig
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/locations/_distutils.py", line 9, in <module>
    from distutils.cmd import Command as DistutilsCommand
ModuleNotFoundError: No module named 'distutils'


Version-Release number of selected component: 105.0.2-1


How reproducible: Build firefox without distutils


Steps to Reproduce:
1. fedpkg clone firefox && cd firefox
2. mock -r fedora-rawhide-x86_64 init
3. mock -r fedora-rawhide-x86_64 install python3-devel
4. sudo rm -rf /var/lib/mock/fedora-rawhide-x86_64/root/usr/lib64/python3.11/distutils/
4. fedpkg mockbuild -N

Actual results: fails to build
Expected results: builds fine

Comment 1 Martin Stransky 2022-10-25 20:44:07 UTC
Can you guide me how to fix that please? Do we need to require it explicitly or so?
Thanks.

Comment 2 Miro Hrončok 2022-10-25 21:56:09 UTC
There is nothing to require explicitly. The module is gone.

Some packages might get away by using setuptools instead, but even when setuptools is BuildRequired, firefox does not build.

I could try to help you replace distutils usage in firefox with sysconfig or setuptools, but it appears from the tracebacks that this problem comes from an old version of bundled pip. I've inspected the source and it seems firefox bundles pip 21.2.4.

My guide to fixing this is: don't bundle pip, use python3-pip. If it is not possible, upgrade the bundled pip to 22.2+ or backport:

https://github.com/pypa/pip/commit/0e0645253041bd3b953fd33b6f393a2e383ebb5a
https://github.com/pypa/pip/commit/db4751595867db1d938df7183a60dfb15fa0d708 (this might not be needed, but it won't hurt)

Comment 3 Tomáš Hrnčiar 2022-12-20 10:53:25 UTC
Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.12:
https://copr.fedorainfracloud.org/coprs/python3.12/

Please use this copr for testing rather than https://copr.fedorainfracloud.org/coprs/g/python/python-without-distutils/

If your package is listed in [0], you may workaround this issue by BuildRequiring python-setuptools, note that adding such BuildRequires might however hide some transitive dependency problem, if the distutils import comes from a dependency.
Cooperation with upstream is recommended. Additional context [1].

[0] https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/message/6BHNAWHE7M5VY3YQVJLOYHLY4M7KIFFN/

[1] https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/thread/N6ITYHLRWIDNYNXGPYG2ZHF3ZLQWZN7L/

https://docs.python.org/3.12/whatsnew/3.12.html

For all our attempts to build firefox with Python 3.12, see:
https://copr.fedorainfracloud.org/coprs/python3.12/package/firefox/

Let us know here if you have any questions.

Python 3.12 is planned to be included in Fedora 39. To make that update smoother, we're building Fedora packages with all pre-releases of Python 3.12.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.

Comment 4 Ben Cotton 2023-02-07 14:57:26 UTC
This bug appears to have been reported against 'rawhide' during the Fedora Linux 38 development cycle.
Changing version to 38.

Comment 5 Miro Hrončok 2023-07-08 10:10:21 UTC
https://koschei.fedoraproject.org/package/firefox

The current issue is:

+ mkdir -p objdir/_virtualenvs/init_py3
+ cat
++ pwd
+ tar xf /builddir/build/SOURCES/mochitest-python.tar.gz
+ xvfb-run ./mach build -v
+ cat -
ERROR: Exception:
Traceback (most recent call last):
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/commands/list.py", line 169, in run
    packages: "_ProcessedDists" = [
                                  ^
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/base.py", line 664, in <genexpr>
    return (d for d in it if d.canonical_name not in skip)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/base.py", line 612, in iter_all_distributions
    for dist in self._iter_distributions():
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 176, in _iter_distributions
    for dist in finder.find_eggs(location):
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 144, in find_eggs
    yield from self._find_eggs_in_dir(location)
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 111, in _find_eggs_in_dir
    from pip._vendor.pkg_resources import find_distributions
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_vendor/pkg_resources/__init__.py", line 2164, in <module>
    register_finder(pkgutil.ImpImporter, find_on_path)
                    ^^^^^^^^^^^^^^^^^^^
AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
ERROR: Exception:
Traceback (most recent call last):
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/commands/list.py", line 169, in run
    packages: "_ProcessedDists" = [
                                  ^
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/base.py", line 664, in <genexpr>
    return (d for d in it if d.canonical_name not in skip)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/base.py", line 612, in iter_all_distributions
    for dist in self._iter_distributions():
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 176, in _iter_distributions
    for dist in finder.find_eggs(location):
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 144, in find_eggs
    yield from self._find_eggs_in_dir(location)
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 111, in _find_eggs_in_dir
    from pip._vendor.pkg_resources import find_distributions
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_vendor/pkg_resources/__init__.py", line 2164, in <module>
    register_finder(pkgutil.ImpImporter, find_on_path)
                    ^^^^^^^^^^^^^^^^^^^
AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
ERROR: Exception:
Traceback (most recent call last):
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/commands/list.py", line 169, in run
    packages: "_ProcessedDists" = [
                                  ^
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/base.py", line 664, in <genexpr>
    return (d for d in it if d.canonical_name not in skip)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/base.py", line 612, in iter_all_distributions
    for dist in self._iter_distributions():
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 176, in _iter_distributions
    for dist in finder.find_eggs(location):
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 144, in find_eggs
    yield from self._find_eggs_in_dir(location)
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 111, in _find_eggs_in_dir
    from pip._vendor.pkg_resources import find_distributions
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_vendor/pkg_resources/__init__.py", line 2164, in <module>
    register_finder(pkgutil.ImpImporter, find_on_path)
                    ^^^^^^^^^^^^^^^^^^^
AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
Mach and the build system store shared state in a common directory
on the filesystem. The following directory will be created:
  /builddir/.mozbuild
If you would like to use a different directory, rename or move it to your
desired location, and set the MOZBUILD_STATE_PATH environment variable
accordingly.
Creating default state directory: /builddir/.mozbuild
Creating local state directory: /builddir/.mozbuild/srcdirs/firefox-115.0-0be14ac973c8
Could not install glean-sdk, so telemetry will not be collected. Continuing.
Could not install psutil, so telemetry will be missing some data. Continuing.
Could not install zstandard, so zstd archives will not be possible to extract. Continuing.
Traceback (most recent call last):
  File "/builddir/build/BUILD/firefox-115.0/./mach", line 125, in <module>
    main(sys.argv[1:])
  File "/builddir/build/BUILD/firefox-115.0/./mach", line 117, in main
    mach = check_and_get_mach(os.path.dirname(os.path.realpath(__file__)))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/firefox-115.0/./mach", line 37, in check_and_get_mach
    return load_mach(dir_path, mach_path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/firefox-115.0/./mach", line 23, in load_mach
    return mach_initialize.initialize(dir_path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/firefox-115.0/build/mach_initialize.py", line 153, in initialize
    import mach.main
  File "/builddir/build/BUILD/firefox-115.0/python/mach/mach/main.py", line 11, in <module>
    import imp
ModuleNotFoundError: No module named 'imp'



The first issue can be solved by unbundling pip (or upgrading it). For firefox-115.0/python/mach/mach/main.py and ModuleNotFoundError: No module named 'imp', you could BuildRequire python3-zombie-imp as a stopgap measure, but you should really post to importlib.

Comment 6 Fedora Release Engineering 2023-08-16 07:06:08 UTC
This bug appears to have been reported against 'rawhide' during the Fedora Linux 39 development cycle.
Changing version to 39.


Note You need to log in before you can comment on or make changes to this bug.