Bug 1989081

Summary:	Encourage upstreams to release Python 3.10 manylinux wheels before Fedora 35 final release
Product:	[Fedora] Fedora	Reporter:	Miro Hrončok <mhroncok>
Component:	python3.10	Assignee:	Tomáš Hrnčiar <thrnciar>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	35	CC:	cstratak, mhroncok, pviktori, python-maint, python-sig, quantum.analyst, thrnciar, torsava, vstinner
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-11-16 10:35:08 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1890881

Description Miro Hrončok 2021-08-02 11:10:24 UTC

This is a tracking bugzilla inspired by bz1875767.

Python 3.10.0rc1 should be released today. Once we reach the RC, the ABI is frozen and cannot change even for bug fixes.

We should encourage upstreams to release manylinux wheels as soon as the first rc is out. users of Fedora 35 will likely use Python 3.10 and when they `pip install numpy` etc., it should work flawlessly -- we should encourage upstreams to have the manylinux wheels ready, so pip does not attempt to compile stuff which is slow and often breaks for missing development dependencies.

Ideally, we want the wheels to be ready before the final release, but if we don't make it, we still want them to be released ASAP.

Here is the list of packages we used the last time:


---------


NumPy


For matplotlib:

Pillow
kiwisolver
matplotlib


For Jupyter Notebook:

pyzmq
MarkupSafe
cffi


Others:

pandas
scipy


---------

Note that cython has universal wheel available, so I've moved it from the list.


Verification:

$ python3.10 -m venv __venv__
$ . __venv__/bin/activate
(__venv__) $ pip install numpy matplotlib notebook pandas scipy

No packages (with extension modules) should be compiled locally, they should be installed from wheels.

Comment 1 Tomáš Hrnčiar 2021-08-09 10:43:40 UTC

NumPy - https://github.com/numpy/numpy/issues/19630

For matplotlib:
Pillow - https://github.com/python-pillow/Pillow/issues/5662
kiwisolver - https://github.com/nucleic/kiwi/issues/114
matplotlib - https://github.com/matplotlib/matplotlib/issues/20811

For Jupyter Notebook:
pyzmq - https://github.com/zeromq/pyzmq/issues/1583
MarkupSafe - https://github.com/pallets/markupsafe/issues/237
cffi - https://bugs.launchpad.net/cffi/+bug/1939288

Others:
pandas - https://github.com/pandas-dev/pandas/issues/42946
scipy - https://github.com/scipy/scipy/issues/14567

Comment 2 Tomáš Hrnčiar 2021-08-10 07:19:57 UTC

(In reply to Tomáš Hrnčiar from comment #1)
> For Jupyter Notebook:
> pyzmq - https://github.com/zeromq/pyzmq/issues/1583
done

Comment 3 Ben Cotton 2021-08-10 13:33:48 UTC

This bug appears to have been reported against 'rawhide' during the Fedora 35 development cycle.
Changing version to 35.

Comment 4 Tomáš Hrnčiar 2021-08-12 06:49:50 UTC

(In reply to Tomáš Hrnčiar from comment #1)
> MarkupSafe - https://github.com/pallets/markupsafe/issues/237
done

Comment 5 Tomáš Hrnčiar 2021-08-16 08:29:02 UTC

(In reply to Tomáš Hrnčiar from comment #1)
> NumPy - https://github.com/numpy/numpy/issues/19630
done

Comment 6 Miro Hrončok 2021-08-30 09:42:34 UTC

Also needed for notebook: tornado and cffi.

Comment 7 Miro Hrončok 2021-09-03 09:09:11 UTC

> Pillow - https://github.com/python-pillow/Pillow/issues/5662

Done.

Comment 8 Miro Hrončok 2021-09-03 09:13:38 UTC

(In reply to Miro Hrončok from comment #6)
> Also needed for notebook: tornado and cffi.

https://github.com/tornadoweb/tornado/issues/3057
https://foss.heptapod.net/pypy/cffi/-/issues/508

Comment 10 Miro Hrončok 2021-09-03 09:42:18 UTC

Status update, we are half way there:

DONE:

NumPy
Pillow
kiwisolver
pyzmq
MarkupSafe

TODO:

pandas       https://github.com/pandas-dev/pandas/issues/42946
scipy        https://github.com/scipy/scipy/issues/14567
matplotlib   https://github.com/matplotlib/matplotlib/issues/20811
torando      https://github.com/tornadoweb/tornado/issues/3057
cffi         https://foss.heptapod.net/pypy/cffi/-/issues/508

Comment 11 Miro Hrončok 2021-09-13 08:12:50 UTC

pandas https://github.com/pandas-dev/pandas/issues/42946 is done:


$ pip install pandas
Collecting pandas
  Downloading pandas-1.3.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.5 MB)
...

Comment 12 Tomáš Hrnčiar 2021-10-07 07:48:07 UTC

cffi https://foss.heptapod.net/pypy/cffi/-/issues/508 is done

I've opened PR to bring it into Fedora. https://src.fedoraproject.org/rpms/python-cffi/pull-request/11

Comment 13 Miro Hrončok 2021-10-07 08:34:30 UTC

cffi https://foss.heptapod.net/pypy/cffi/-/issues/508 is only released as a pre-release, so `pip install cffi` will still grab the latest stable version with sdists



> I've opened PR to bring it into Fedora. https://src.fedoraproject.org/rpms/python-cffi/pull-request/11

Thanks, but for the sake of this bugzilla, we need the stable version on pypi, the packaged version in Fedora is not really relevant.

Comment 14 Miro Hrončok 2021-10-13 16:43:31 UTC

cffi is done:


$ pip install cffi
Collecting cffi
  Downloading cffi-1.15.0-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (446 kB)
     |████████████████████████████████| 446 kB 2.7 MB/s 
Collecting pycparser
  Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Installing collected packages: pycparser, cffi
Successfully installed cffi-1.15.0 pycparser-2.20

Comment 15 Petr Viktorin (pviktori) 2021-10-26 09:10:38 UTC

DONE:

NumPy
Pillow
kiwisolver
pyzmq
MarkupSafe
pandas
cffi

TODO:

scipy        https://github.com/scipy/scipy/issues/14567
matplotlib   https://github.com/matplotlib/matplotlib/issues/20811
torando      https://github.com/tornadoweb/tornado/issues/3057

Comment 16 Miro Hrončok 2021-10-26 16:55:08 UTC

I've tested a Fedora 35 container and pip install.

- torando installs from sdist without issues, it just skips the speedups

- matplotlib fails with: GNU make (>= 3.80) or makepp (>= 1.19) is required to build FreeType2.

- scipy fails with: numpy.distutils.system_info.NotFoundError: No BLAS/LAPACK libraries found. / RuntimeError: Broken toolchain: cannot link a simple C program


pip will keep trying to download lower and lower versions of scipy and it keeps installing build dependencies over and over until it gives up after OSError: [Errno 28] No space left on device -- that is indeed a very bad user experience.

Comment 17 Tomáš Hrnčiar 2021-11-08 09:13:00 UTC

scipy wheels were released.

❯ pip install scipy
Defaulting to user installation because normal site-packages is not writeable
Collecting scipy
  Downloading scipy-1.7.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (39.9 MB)
     |████████████████████████████████| 39.9 MB 22.1 MB/s
Collecting numpy<1.23.0,>=1.16.5
  Downloading numpy-1.21.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (15.9 MB)
     |████████████████████████████████| 15.9 MB 3.4 MB/s
Installing collected packages: numpy, scipy
Successfully installed numpy-1.21.4 scipy-1.7.2

Comment 18 Miro Hrončok 2021-11-10 17:12:54 UTC

Elliott, could you please help us get matplotlib (not a pre-release) with Python 3.10 manylinux wheels released on PyPI? Our issue was closed in September, but when Fedora 35 users run `pip install matplotlib`, it will still build from source. We can help if only we would know how.

Thanks a lot!

Comment 19 Elliott Sales de Andrade 2021-11-16 09:56:59 UTC

That is now done.

Comment 20 Miro Hrončok 2021-11-16 10:35:08 UTC

(venv) [root@28675041da1f /]# python -m pip install NumPy matplotlib notebook pandas scipy 
Collecting NumPy
  Downloading numpy-1.21.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (15.9 MB)
     |████████████████████████████████| 15.9 MB 4.1 MB/s 
Collecting matplotlib
  Downloading matplotlib-3.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.9 MB)
     |████████████████████████████████| 11.9 MB 3.2 MB/s 
Collecting notebook
  Downloading notebook-6.4.5-py3-none-any.whl (9.9 MB)
     |████████████████████████████████| 9.9 MB 4.1 MB/s 
Collecting pandas
  Downloading pandas-1.3.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.5 MB)
     |████████████████████████████████| 11.5 MB 3.6 MB/s 
Collecting scipy
  Downloading scipy-1.7.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (39.9 MB)
     |████████████████████████████████| 39.9 MB 3.1 MB/s 
Collecting pillow>=6.2.0
  Downloading Pillow-8.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)
     |████████████████████████████████| 3.1 MB 3.4 MB/s 
Collecting packaging>=20.0
  Downloading packaging-21.2-py3-none-any.whl (40 kB)
     |████████████████████████████████| 40 kB 2.5 MB/s 
Collecting cycler>=0.10
  Downloading cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting fonttools>=4.22.0
  Downloading fonttools-4.28.1-py3-none-any.whl (873 kB)
     |████████████████████████████████| 873 kB 4.0 MB/s 
Collecting pyparsing>=2.2.1
  Downloading pyparsing-3.0.6-py3-none-any.whl (97 kB)
     |████████████████████████████████| 97 kB 2.6 MB/s 
Collecting python-dateutil>=2.7
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
     |████████████████████████████████| 247 kB 3.5 MB/s 
Collecting setuptools-scm>=4
  Downloading setuptools_scm-6.3.2-py3-none-any.whl (33 kB)
Collecting kiwisolver>=1.0.1
  Downloading kiwisolver-1.3.2-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB)
     |████████████████████████████████| 1.6 MB 4.0 MB/s 
Collecting Send2Trash>=1.5.0
  Downloading Send2Trash-1.8.0-py3-none-any.whl (18 kB)
Collecting jupyter-client>=5.3.4
  Downloading jupyter_client-7.0.6-py3-none-any.whl (125 kB)
     |████████████████████████████████| 125 kB 3.2 MB/s 
Collecting traitlets>=4.2.1
  Downloading traitlets-5.1.1-py3-none-any.whl (102 kB)
     |████████████████████████████████| 102 kB 3.4 MB/s 
Collecting nbformat
  Downloading nbformat-5.1.3-py3-none-any.whl (178 kB)
     |████████████████████████████████| 178 kB 3.2 MB/s 
Collecting tornado>=6.1
  Downloading tornado-6.1.tar.gz (497 kB)
     |████████████████████████████████| 497 kB 4.1 MB/s 
Collecting ipython-genutils
  Downloading ipython_genutils-0.2.0-py2.py3-none-any.whl (26 kB)
Collecting nbconvert
  Downloading nbconvert-6.3.0-py3-none-any.whl (556 kB)
     |████████████████████████████████| 556 kB 4.8 MB/s 
Collecting pyzmq>=17
  Downloading pyzmq-22.3.0-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1 MB)
     |████████████████████████████████| 1.1 MB 3.9 MB/s 
Collecting terminado>=0.8.3
  Downloading terminado-0.12.1-py3-none-any.whl (15 kB)
Collecting ipykernel
  Downloading ipykernel-6.5.0-py3-none-any.whl (125 kB)
     |████████████████████████████████| 125 kB 3.5 MB/s 
Collecting jupyter-core>=4.6.1
  Downloading jupyter_core-4.9.1-py3-none-any.whl (86 kB)
     |████████████████████████████████| 86 kB 4.4 MB/s 
Collecting prometheus-client
  Downloading prometheus_client-0.12.0-py2.py3-none-any.whl (57 kB)
     |████████████████████████████████| 57 kB 2.2 MB/s 
Collecting argon2-cffi
  Downloading argon2_cffi-21.1.0-cp35-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.whl (96 kB)
     |████████████████████████████████| 96 kB 3.9 MB/s 
Collecting jinja2
  Downloading Jinja2-3.0.3-py3-none-any.whl (133 kB)
     |████████████████████████████████| 133 kB 3.9 MB/s 
Collecting pytz>=2017.3
  Downloading pytz-2021.3-py2.py3-none-any.whl (503 kB)
     |████████████████████████████████| 503 kB 4.2 MB/s 
Collecting entrypoints
  Downloading entrypoints-0.3-py2.py3-none-any.whl (11 kB)
Collecting nest-asyncio>=1.5
  Downloading nest_asyncio-1.5.1-py3-none-any.whl (5.0 kB)
Collecting pyparsing>=2.2.1
  Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
     |████████████████████████████████| 67 kB 4.1 MB/s 
Collecting six>=1.5
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting tomli>=1.0.0
  Downloading tomli-1.2.2-py3-none-any.whl (12 kB)
Requirement already satisfied: setuptools in /venv/lib/python3.10/site-packages (from setuptools-scm>=4->matplotlib) (57.4.0)
Collecting ptyprocess
  Downloading ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB)
Collecting cffi>=1.0.0
  Downloading cffi-1.15.0-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (446 kB)
     |████████████████████████████████| 446 kB 4.0 MB/s 
Collecting pycparser
  Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
     |████████████████████████████████| 118 kB 3.0 MB/s 
Collecting ipython<8.0,>=7.23.1
  Downloading ipython-7.29.0-py3-none-any.whl (790 kB)
     |████████████████████████████████| 790 kB 4.2 MB/s 
Collecting matplotlib-inline<0.2.0,>=0.1.0
  Downloading matplotlib_inline-0.1.3-py3-none-any.whl (8.2 kB)
Collecting debugpy<2.0,>=1.0.0
  Downloading debugpy-1.5.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.9 MB)
     |████████████████████████████████| 1.9 MB 4.0 MB/s 
Collecting pexpect>4.3
  Downloading pexpect-4.8.0-py2.py3-none-any.whl (59 kB)
     |████████████████████████████████| 59 kB 4.1 MB/s 
Collecting pygments
  Downloading Pygments-2.10.0-py3-none-any.whl (1.0 MB)
     |████████████████████████████████| 1.0 MB 1.7 MB/s 
Collecting prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0
  Downloading prompt_toolkit-3.0.22-py3-none-any.whl (374 kB)
     |████████████████████████████████| 374 kB 3.7 MB/s 
Collecting pickleshare
  Downloading pickleshare-0.7.5-py2.py3-none-any.whl (6.9 kB)
Collecting decorator
  Downloading decorator-5.1.0-py3-none-any.whl (9.1 kB)
Collecting backcall
  Downloading backcall-0.2.0-py2.py3-none-any.whl (11 kB)
Collecting jedi>=0.16
  Downloading jedi-0.18.0-py2.py3-none-any.whl (1.4 MB)
     |████████████████████████████████| 1.4 MB 3.2 MB/s 
Collecting parso<0.9.0,>=0.8.0
  Downloading parso-0.8.2-py2.py3-none-any.whl (94 kB)
     |████████████████████████████████| 94 kB 2.8 MB/s 
Collecting wcwidth
  Downloading wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting MarkupSafe>=2.0
  Downloading MarkupSafe-2.0.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (30 kB)
Collecting pandocfilters>=1.4.1
  Downloading pandocfilters-1.5.0-py2.py3-none-any.whl (8.7 kB)
Collecting jupyterlab-pygments
  Downloading jupyterlab_pygments-0.1.2-py2.py3-none-any.whl (4.6 kB)
Collecting bleach
  Downloading bleach-4.1.0-py2.py3-none-any.whl (157 kB)
     |████████████████████████████████| 157 kB 4.2 MB/s 
Collecting nbclient<0.6.0,>=0.5.0
  Downloading nbclient-0.5.8-py3-none-any.whl (70 kB)
     |████████████████████████████████| 70 kB 2.6 MB/s 
Collecting testpath
  Downloading testpath-0.5.0-py3-none-any.whl (84 kB)
     |████████████████████████████████| 84 kB 3.3 MB/s 
Collecting defusedxml
  Downloading defusedxml-0.7.1-py2.py3-none-any.whl (25 kB)
Collecting mistune<2,>=0.8.1
  Downloading mistune-0.8.4-py2.py3-none-any.whl (16 kB)
Collecting jsonschema!=2.5.0,>=2.4
  Downloading jsonschema-4.2.1-py3-none-any.whl (69 kB)
     |████████████████████████████████| 69 kB 4.2 MB/s 
Collecting pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0
  Downloading pyrsistent-0.18.0.tar.gz (104 kB)
     |████████████████████████████████| 104 kB 5.1 MB/s 
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting attrs>=17.4.0
  Downloading attrs-21.2.0-py2.py3-none-any.whl (53 kB)
     |████████████████████████████████| 53 kB 3.1 MB/s 
Collecting webencodings
  Downloading webencodings-0.5.1-py2.py3-none-any.whl (11 kB)
Using legacy 'setup.py install' for tornado, since package 'wheel' is not installed.
Building wheels for collected packages: pyrsistent
  Building wheel for pyrsistent (PEP 517) ... done
  Created wheel for pyrsistent: filename=pyrsistent-0.18.0-cp310-cp310-linux_x86_64.whl size=56359 sha256=b1d8e4cf7258826c6e3eba938f5db534ee23e08b6184e9a14caaa861335697b8
  Stored in directory: /root/.cache/pip/wheels/bf/ee/17/b548a960bb9e20daf7987f316c326e8a368603809ace3b2374
Successfully built pyrsistent
Installing collected packages: traitlets, six, pyrsistent, attrs, wcwidth, tornado, pyzmq, python-dateutil, pyparsing, ptyprocess, parso, nest-asyncio, jupyter-core, jsonschema, ipython-genutils, entrypoints, webencodings, pygments, pycparser, prompt-toolkit, pickleshare, pexpect, packaging, nbformat, matplotlib-inline, MarkupSafe, jupyter-client, jedi, decorator, backcall, tomli, testpath, pandocfilters, nbclient, mistune, jupyterlab-pygments, jinja2, ipython, defusedxml, debugpy, cffi, bleach, terminado, setuptools-scm, Send2Trash, pytz, prometheus-client, pillow, NumPy, nbconvert, kiwisolver, ipykernel, fonttools, cycler, argon2-cffi, scipy, pandas, notebook, matplotlib
    Running setup.py install for tornado ... done
Successfully installed MarkupSafe-2.0.1 NumPy-1.21.4 Send2Trash-1.8.0 argon2-cffi-21.1.0 attrs-21.2.0 backcall-0.2.0 bleach-4.1.0 cffi-1.15.0 cycler-0.11.0 debugpy-1.5.1 decorator-5.1.0 defusedxml-0.7.1 entrypoints-0.3 fonttools-4.28.1 ipykernel-6.5.0 ipython-7.29.0 ipython-genutils-0.2.0 jedi-0.18.0 jinja2-3.0.3 jsonschema-4.2.1 jupyter-client-7.0.6 jupyter-core-4.9.1 jupyterlab-pygments-0.1.2 kiwisolver-1.3.2 matplotlib-3.5.0 matplotlib-inline-0.1.3 mistune-0.8.4 nbclient-0.5.8 nbconvert-6.3.0 nbformat-5.1.3 nest-asyncio-1.5.1 notebook-6.4.5 packaging-21.2 pandas-1.3.4 pandocfilters-1.5.0 parso-0.8.2 pexpect-4.8.0 pickleshare-0.7.5 pillow-8.4.0 prometheus-client-0.12.0 prompt-toolkit-3.0.22 ptyprocess-0.7.0 pycparser-2.21 pygments-2.10.0 pyparsing-2.4.7 pyrsistent-0.18.0 python-dateutil-2.8.2 pytz-2021.3 pyzmq-22.3.0 scipy-1.7.2 setuptools-scm-6.3.2 six-1.16.0 terminado-0.12.1 testpath-0.5.0 tomli-1.2.2 tornado-6.1 traitlets-5.1.1 wcwidth-0.2.5 webencodings-0.5.1
WARNING: You are using pip version 21.2.3; however, version 21.3.1 is available.
You should consider upgrading via the '/venv/bin/python -m pip install --upgrade pip' command.




This is without python3-devel and without gcc.

Thnak you all!