Bug 2135421 - firefox fails to build with Python 3.12: Bundles an outdated version of pip
Summary: firefox fails to build with Python 3.12: Bundles an outdated version of pip
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: firefox
Version: 39
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Gecko Maintainer
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: PYTHON3.12
TreeView+ depends on / blocked
 
Reported: 2022-10-17 15:28 UTC by Miro Hrončok
Modified: 2024-11-27 21:03 UTC (History)
13 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2024-11-27 21:03:00 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Mozilla Foundation 1743785 0 P2 RESOLVED [meta] distutils is deprecated, refactor usages 2024-12-01 17:48:48 UTC
Mozilla Foundation 1857515 0 -- NEW [meta] Build failures on Python 3.12 2024-12-01 17:48:47 UTC

Description Miro Hrončok 2022-10-17 15:28:29 UTC
Description of problem:
The Python standard library distutils module will be removed from Python 3.12+

https://peps.python.org/pep-0632/

As preparatory work, we build all python packages in a Copr repository with Python 3.11 sans distutils.

https://copr.fedorainfracloud.org/coprs/g/python/python-without-distutils/

firefox fails to build:

+ ./mach build -v
created virtual environment CPython3.11.0.candidate.2-64 in 43ms
  creator CPython3Posix(dest=/builddir/.mozbuild/srcdirs/firefox-105.0.2-589d8dcd2428/_virtualenvs/mach, clear=False, no_vcs_ignore=False, global=False)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/__main__.py", line 29, in <module>
    from pip._internal.cli.main import main as _main
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/main.py", line 9, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/autocompletion.py", line 10, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/main_parser.py", line 8, in <module>
    from pip._internal.cli import cmdoptions
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/cmdoptions.py", line 23, in <module>
    from pip._internal.cli.parser import ConfigOptionParser
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/parser.py", line 12, in <module>
    from pip._internal.configuration import Configuration, ConfigurationError
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/configuration.py", line 27, in <module>
    from pip._internal.utils.misc import ensure_dir, enum
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/utils/misc.py", line 42, in <module>
    from pip._internal.locations import get_major_minor_version, site_packages, user_site
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/locations/__init__.py", line 14, in <module>
    from . import _distutils, _sysconfig
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/locations/_distutils.py", line 9, in <module>
    from distutils.cmd import Command as DistutilsCommand
ModuleNotFoundError: No module named 'distutils'
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/__main__.py", line 29, in <module>
    from pip._internal.cli.main import main as _main
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/main.py", line 9, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/autocompletion.py", line 10, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/main_parser.py", line 8, in <module>
    from pip._internal.cli import cmdoptions
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/cmdoptions.py", line 23, in <module>
    from pip._internal.cli.parser import ConfigOptionParser
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/parser.py", line 12, in <module>
    from pip._internal.configuration import Configuration, ConfigurationError
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/configuration.py", line 27, in <module>
    from pip._internal.utils.misc import ensure_dir, enum
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/utils/misc.py", line 42, in <module>
    from pip._internal.locations import get_major_minor_version, site_packages, user_site
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/locations/__init__.py", line 14, in <module>
    from . import _distutils, _sysconfig
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/locations/_distutils.py", line 9, in <module>
    from distutils.cmd import Command as DistutilsCommand
ModuleNotFoundError: No module named 'distutils'
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/__main__.py", line 29, in <module>
    from pip._internal.cli.main import main as _main
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/main.py", line 9, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/autocompletion.py", line 10, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/main_parser.py", line 8, in <module>
    from pip._internal.cli import cmdoptions
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/cmdoptions.py", line 23, in <module>
    from pip._internal.cli.parser import ConfigOptionParser
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/cli/parser.py", line 12, in <module>
    from pip._internal.configuration import Configuration, ConfigurationError
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/configuration.py", line 27, in <module>
    from pip._internal.utils.misc import ensure_dir, enum
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/utils/misc.py", line 42, in <module>
    from pip._internal.locations import get_major_minor_version, site_packages, user_site
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/locations/__init__.py", line 14, in <module>
    from . import _distutils, _sysconfig
  File "/builddir/build/BUILD/firefox-105.0.2/third_party/python/pip/pip/_internal/locations/_distutils.py", line 9, in <module>
    from distutils.cmd import Command as DistutilsCommand
ModuleNotFoundError: No module named 'distutils'


Version-Release number of selected component: 105.0.2-1


How reproducible: Build firefox without distutils


Steps to Reproduce:
1. fedpkg clone firefox && cd firefox
2. mock -r fedora-rawhide-x86_64 init
3. mock -r fedora-rawhide-x86_64 install python3-devel
4. sudo rm -rf /var/lib/mock/fedora-rawhide-x86_64/root/usr/lib64/python3.11/distutils/
4. fedpkg mockbuild -N

Actual results: fails to build
Expected results: builds fine

Comment 1 Martin Stransky 2022-10-25 20:44:07 UTC
Can you guide me how to fix that please? Do we need to require it explicitly or so?
Thanks.

Comment 2 Miro Hrončok 2022-10-25 21:56:09 UTC
There is nothing to require explicitly. The module is gone.

Some packages might get away by using setuptools instead, but even when setuptools is BuildRequired, firefox does not build.

I could try to help you replace distutils usage in firefox with sysconfig or setuptools, but it appears from the tracebacks that this problem comes from an old version of bundled pip. I've inspected the source and it seems firefox bundles pip 21.2.4.

My guide to fixing this is: don't bundle pip, use python3-pip. If it is not possible, upgrade the bundled pip to 22.2+ or backport:

https://github.com/pypa/pip/commit/0e0645253041bd3b953fd33b6f393a2e383ebb5a
https://github.com/pypa/pip/commit/db4751595867db1d938df7183a60dfb15fa0d708 (this might not be needed, but it won't hurt)

Comment 3 Tomáš Hrnčiar 2022-12-20 10:53:25 UTC
Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.12:
https://copr.fedorainfracloud.org/coprs/python3.12/

Please use this copr for testing rather than https://copr.fedorainfracloud.org/coprs/g/python/python-without-distutils/

If your package is listed in [0], you may workaround this issue by BuildRequiring python-setuptools, note that adding such BuildRequires might however hide some transitive dependency problem, if the distutils import comes from a dependency.
Cooperation with upstream is recommended. Additional context [1].

[0] https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/message/6BHNAWHE7M5VY3YQVJLOYHLY4M7KIFFN/

[1] https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/thread/N6ITYHLRWIDNYNXGPYG2ZHF3ZLQWZN7L/

https://docs.python.org/3.12/whatsnew/3.12.html

For all our attempts to build firefox with Python 3.12, see:
https://copr.fedorainfracloud.org/coprs/python3.12/package/firefox/

Let us know here if you have any questions.

Python 3.12 is planned to be included in Fedora 39. To make that update smoother, we're building Fedora packages with all pre-releases of Python 3.12.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.

Comment 4 Ben Cotton 2023-02-07 14:57:26 UTC
This bug appears to have been reported against 'rawhide' during the Fedora Linux 38 development cycle.
Changing version to 38.

Comment 5 Miro Hrončok 2023-07-08 10:10:21 UTC
https://koschei.fedoraproject.org/package/firefox

The current issue is:

+ mkdir -p objdir/_virtualenvs/init_py3
+ cat
++ pwd
+ tar xf /builddir/build/SOURCES/mochitest-python.tar.gz
+ xvfb-run ./mach build -v
+ cat -
ERROR: Exception:
Traceback (most recent call last):
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/commands/list.py", line 169, in run
    packages: "_ProcessedDists" = [
                                  ^
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/base.py", line 664, in <genexpr>
    return (d for d in it if d.canonical_name not in skip)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/base.py", line 612, in iter_all_distributions
    for dist in self._iter_distributions():
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 176, in _iter_distributions
    for dist in finder.find_eggs(location):
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 144, in find_eggs
    yield from self._find_eggs_in_dir(location)
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 111, in _find_eggs_in_dir
    from pip._vendor.pkg_resources import find_distributions
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_vendor/pkg_resources/__init__.py", line 2164, in <module>
    register_finder(pkgutil.ImpImporter, find_on_path)
                    ^^^^^^^^^^^^^^^^^^^
AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
ERROR: Exception:
Traceback (most recent call last):
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/commands/list.py", line 169, in run
    packages: "_ProcessedDists" = [
                                  ^
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/base.py", line 664, in <genexpr>
    return (d for d in it if d.canonical_name not in skip)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/base.py", line 612, in iter_all_distributions
    for dist in self._iter_distributions():
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 176, in _iter_distributions
    for dist in finder.find_eggs(location):
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 144, in find_eggs
    yield from self._find_eggs_in_dir(location)
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 111, in _find_eggs_in_dir
    from pip._vendor.pkg_resources import find_distributions
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_vendor/pkg_resources/__init__.py", line 2164, in <module>
    register_finder(pkgutil.ImpImporter, find_on_path)
                    ^^^^^^^^^^^^^^^^^^^
AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
ERROR: Exception:
Traceback (most recent call last):
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/commands/list.py", line 169, in run
    packages: "_ProcessedDists" = [
                                  ^
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/base.py", line 664, in <genexpr>
    return (d for d in it if d.canonical_name not in skip)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/base.py", line 612, in iter_all_distributions
    for dist in self._iter_distributions():
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 176, in _iter_distributions
    for dist in finder.find_eggs(location):
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 144, in find_eggs
    yield from self._find_eggs_in_dir(location)
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 111, in _find_eggs_in_dir
    from pip._vendor.pkg_resources import find_distributions
  File "/builddir/build/BUILD/firefox-115.0/third_party/python/pip/pip/_vendor/pkg_resources/__init__.py", line 2164, in <module>
    register_finder(pkgutil.ImpImporter, find_on_path)
                    ^^^^^^^^^^^^^^^^^^^
AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
Mach and the build system store shared state in a common directory
on the filesystem. The following directory will be created:
  /builddir/.mozbuild
If you would like to use a different directory, rename or move it to your
desired location, and set the MOZBUILD_STATE_PATH environment variable
accordingly.
Creating default state directory: /builddir/.mozbuild
Creating local state directory: /builddir/.mozbuild/srcdirs/firefox-115.0-0be14ac973c8
Could not install glean-sdk, so telemetry will not be collected. Continuing.
Could not install psutil, so telemetry will be missing some data. Continuing.
Could not install zstandard, so zstd archives will not be possible to extract. Continuing.
Traceback (most recent call last):
  File "/builddir/build/BUILD/firefox-115.0/./mach", line 125, in <module>
    main(sys.argv[1:])
  File "/builddir/build/BUILD/firefox-115.0/./mach", line 117, in main
    mach = check_and_get_mach(os.path.dirname(os.path.realpath(__file__)))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/firefox-115.0/./mach", line 37, in check_and_get_mach
    return load_mach(dir_path, mach_path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/firefox-115.0/./mach", line 23, in load_mach
    return mach_initialize.initialize(dir_path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/firefox-115.0/build/mach_initialize.py", line 153, in initialize
    import mach.main
  File "/builddir/build/BUILD/firefox-115.0/python/mach/mach/main.py", line 11, in <module>
    import imp
ModuleNotFoundError: No module named 'imp'



The first issue can be solved by unbundling pip (or upgrading it). For firefox-115.0/python/mach/mach/main.py and ModuleNotFoundError: No module named 'imp', you could BuildRequire python3-zombie-imp as a stopgap measure, but you should really post to importlib.

Comment 6 Fedora Release Engineering 2023-08-16 07:06:08 UTC
This bug appears to have been reported against 'rawhide' during the Fedora Linux 39 development cycle.
Changing version to 39.

Comment 7 Miro Hrončok 2023-08-21 18:18:35 UTC
As of https://src.fedoraproject.org/rpms/firefox/c/bd405c0fbd1e7c3f444f20bd10b0db8b89fc16e9?branch=rawhide firefox uses python3.11 to build.

Comment 8 Tomas Popela 2023-10-23 09:49:29 UTC
All the Python 3.12 fixes linked from https://bugzilla.mozilla.org/show_bug.cgi?id=1857515 are targeting Firefox 120 that will be released on November 21 - https://whattrainisitnow.com/release/?version=120.

Comment 9 Aoife Moloney 2024-11-08 10:44:51 UTC
This message is a reminder that Fedora Linux 39 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 39 on 2024-11-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '39'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version. Note that the version field may be hidden.
Click the "Show advanced fields" button if you do not see it.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 39 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Comment 10 Aoife Moloney 2024-11-27 21:03:00 UTC
Fedora Linux 39 entered end-of-life (EOL) status on 2024-11-26.

Fedora Linux 39 is no longer maintained, which means that it
will not receive any further security or bug fix updates. As a result we
are closing this bug.

If you can reproduce this bug against a currently maintained version of Fedora Linux
please feel free to reopen this bug against that version. Note that the version
field may be hidden. Click the "Show advanced fields" button if you do not see
the version field.

If you are unable to reopen this bug, please file a new report against an
active release.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.