Bug 1758141 - Python automatic requires do not handle x.y.* versions or ~= properly
Summary: Python automatic requires do not handle x.y.* versions or ~= properly
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: python-rpm-generators
Version: 32
Hardware: Unspecified
OS: Linux
unspecified
unspecified
Target Milestone: ---
Assignee: Tomas Orsava
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: PYTHON39
TreeView+ depends on / blocked
 
Reported: 2019-10-03 11:54 UTC by Jiri Konecny
Modified: 2020-04-21 13:20 UTC (History)
16 users (show)

Fixed In Version: python-rpm-generators-10-1.fc32
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-04-21 13:20:08 UTC
Type: Bug


Attachments (Terms of Use)

Description Jiri Konecny 2019-10-03 11:54:47 UTC
Description of problem:
The python3-astroid package is missing dependencies which were auto-generated properly till next release. Precisely our tests are failing thanks to the missing wrapt package which is a more than a year old dependency.


Version-Release number of selected component (if applicable):
python-astroid-2.3.1-1.gitbff51e9.fc32

How reproducible:
Always


Steps to Reproduce:
1. mock -r fedora-rawhide-x86_64 --init
2. mock -r fedora-rawhide-x86_64 -i dnf
3. mock -r fedora-rawhide-x86_64 --shell
4. dnf repoquery --requires python3-astroid
or
4. dnf repoquery install python3-astroid

Actual results:
Only python3-six is an install dependency.


Expected results:
Should be python3-six, python3-wrapt and python3-lazy-object-proxy

Additional info:
The auto-generation was enabled here:
https://src.fedoraproject.org/rpms/python-astroid/c/23d91a8a7d057169f11921afab51c7e561910d2d?branch=master

Comment 1 Miro Hrončok 2019-10-03 12:12:07 UTC
What bug are you reporting? Is it that the automatic dependencies generator is broken (python-rpm-generators) or that astroid is missing dependencies?

Let me check what happens here.

Comment 2 Petr Viktorin 2019-10-03 12:16:41 UTC
The requires.txt went from:

lazy_object_proxy
six
wrapt

to:

lazy_object_proxy==1.4.*
six==1.12
wrapt==1.11.*


The asterisks are valid according to PEP 440, but easy to overlook -- I wouldn't be surprised if python-rpm-generators had a bug in handling them:
https://www.python.org/dev/peps/pep-0440/#version-matching

Comment 3 Miro Hrončok 2019-10-03 12:17:03 UTC
install_requires = [
    "lazy_object_proxy==1.4.*",
    "six==1.12",
    "wrapt==1.11.*",
    'typed-ast>=1.4.0,<1.5;implementation_name== "cpython" and python_version<"3.8"',
]

It only requires:

python3.8dist(six) = 1.12




I think I know what the problem is. The 1.11.* versions with * is not handled properly.

Comment 4 Miro Hrončok 2019-10-03 12:27:33 UTC
Note that in the first place, we should probably fallback to generating at least unversioned dependency when we cannot parse/convert the version.

Comment 5 Miro Hrončok 2019-10-03 12:28:10 UTC
RPMbuild says:

error: Illegal char '*' (0x2a) in: 1.4.*
error: Illegal char '*' (0x2a) in: 1.11.*


And ignores the dependency.

Comment 6 Miro Hrončok 2019-10-03 12:37:13 UTC
IMHO such error should stop the build, I've opened: https://github.com/rpm-software-management/rpm/issues/881

Comment 7 Petr Viktorin 2019-10-15 13:53:14 UTC
As an immediate workaround, either disable the dependency generator or patch the version in setup.py metadata.
We will fix this in python-rpm-generators and pyproject-rpm-macros, but we're low on resources.

Comment 8 Gordon Messmer 2019-11-25 01:47:25 UTC
I've submitted a PR to rpm to address this.  It'll probably need to go through a few iterations of review.

https://github.com/rpm-software-management/rpm/pull/951

Comment 10 Miro Hrončok 2020-01-20 18:09:41 UTC
.* is still broken with !=:


python-kubernetes:

python-10.0.1/requirements.txt
websocket-client>=0.32.0,!=0.40.0,!=0.41.*,!=0.42.* 

error: Illegal char '*' (0x2a) in: 0.42.*


.* is still broken with >=:

python-fsleyes:

fsleyes-0.32.0/fsleyes.egg-info/requires.txt
nibabel>=2.3.*

error: Illegal char '*' (0x2a) in: 2.3.*


The code explicitly only work with == ...* but not other specs:

    elif spec[0] == '==' and spec[1].endswith('.*'):

Comment 11 Miro Hrončok 2020-01-20 18:18:15 UTC
Since rpm now properly fails in those cases, this blocks the Python 3.9 rebuilds of the packages. I will come back with more as they present themselves, and we can figure out if we need to patch them as a workaround, or fix this.

Comment 12 Miro Hrončok 2020-01-20 20:31:57 UTC
Slightly unrelated, qtile has: error: Illegal char ']' (0x5d) in: 0.9[xcb]

Comment 13 Gordon Messmer 2020-01-20 21:31:19 UTC
Do you have a pointer to the specification or handling of "0.9[xcb]" as a version?

Comment 14 Miro Hrončok 2020-01-20 23:48:41 UTC
The input is: 'cairocffi>=0.9[xcb]'

I think it means cairocffi[xcb], with cairocffi>=0.9. Maybe it's just a quirk, https://www.python.org/dev/peps/pep-0508/ doesn't seem to mention it.

pip understands it:

$ pip install 'cairocffi>=0.9[xcb]'
Collecting cairocffi>=0.9[xcb]
  Downloading https://files.pythonhosted.org/packages/f7/99/b3a2c6393563ccbe081ffcceb359ec27a6227792c5169604c1bd8128031a/cairocffi-1.1.0.tar.gz (68kB)
     |████████████████████████████████| 71kB 1.6MB/s 
Collecting cffi>=1.1.0 (from cairocffi>=0.9[xcb])
  Downloading https://files.pythonhosted.org/packages/05/7d/723ceca757d8016850d1110df44b91121695f22f782f876b483ecb3928a9/cffi-1.13.2-cp38-cp38-manylinux1_x86_64.whl (439kB)
     |████████████████████████████████| 440kB 3.4MB/s 
Requirement already satisfied: setuptools>=39.2.0 in ./__venv__/lib/python3.8/site-packages (from cairocffi>=0.9[xcb]) (41.2.0)
Collecting xcffib>=0.3.2 (from cairocffi>=0.9[xcb])
  Downloading https://files.pythonhosted.org/packages/fc/35/101babf6c90faf3e785e3b0fa4f5010a0fd6d20ec8f5d9395f0560263452/xcffib-0.9.0.tar.gz (83kB)
     |████████████████████████████████| 92kB 7.8MB/s 
Collecting pycparser (from cffi>=1.1.0->cairocffi>=0.9[xcb])
Collecting six (from xcffib>=0.3.2->cairocffi>=0.9[xcb])
  Using cached https://files.pythonhosted.org/packages/65/eb/1f97cb97bfc2390a276969c6fae16075da282f5058082d4cb10c6c5c1dba/six-1.14.0-py2.py3-none-any.whl
Installing collected packages: pycparser, cffi, six, xcffib, cairocffi
  Running setup.py install for xcffib ... done
  Running setup.py install for cairocffi ... done
Successfully installed cairocffi-1.1.0 cffi-1.13.2 pycparser-2.19 six-1.14.0 xcffib-0.9.0

Comment 15 Gordon Messmer 2020-01-21 00:37:15 UTC
"cairocffi with the optional xcb component" makes sense, though I'm surprised that it can be specified in that order.

The rpm packages for python3-cairocffi.noarch don't seem to express that "xcb" is included, so I'm not sure if there's any options for handling extras other than to ignore them entirely.

Comment 16 Gordon Messmer 2020-01-21 00:52:39 UTC
I've also created: https://github.com/gordonmessmer/pyreq2rpm/issues/5

Comment 17 Gordon Messmer 2020-01-21 06:16:46 UTC
I've put some debugging statements in pip, and I'm *almost* sure that pip doesn't actually parse the "extras" part of that string:

DEBUG: req: cairocffi
DEBUG: req spec: >=0.9[xcb]
DEBUG: req extra: set()

This seems more like a bug in qtile than anything else.

Comment 18 Miro Hrončok 2020-01-21 09:33:58 UTC
Fixed in https://github.com/qtile/qtile/commit/749dfc9bc615b1a3c0cc15e7958d958711b0c2d6

I'll open a qtile bug.

Comment 19 Neal Gompa 2020-01-21 09:37:33 UTC
The dependency generator doesn't handle extras right now because nobody agrees on how we would handle them. It doesn't help that this particular part of Python module packaging does not appear to be defined well, and I'm pretty sure the behavior handling this is specific to the dependency manager (pip vs pipenv vs poetry, etc).

Personally, we should probably just filter those out.

Comment 20 Miro Hrončok 2020-01-21 09:41:55 UTC
I think "cairocffi[xcb] >= 0.9.0" should be trated as "cairocffi >= 0.9.0" at least. However the qtile case, "cairocffi>=0.9[xcb]", that is a qtile bug: https://bugzilla.redhat.com/show_bug.cgi?id=1793402

Comment 21 Gordon Messmer 2020-01-21 15:51:45 UTC
OK, let's discuss how to handle extras: https://github.com/rpm-software-management/rpm/pull/1014

Comment 22 Miro Hrončok 2020-01-21 17:01:13 UTC
Making sure we are on the same page:

Handling extras is nice to have. Handling issues from comment #10 (!=x.y.* and >=x.y.*) seems like the real problem.

Comment 24 Ankur Sinha (FranciscoD) 2020-01-30 15:31:11 UTC
Hello,

I'm seeing this in F31 too: https://koji.fedoraproject.org/koji/packageinfo?packageID=27750

It didn't fail in F30 or in F32 though. Have they received fixes already, or is this a regression in F31?

Cheers,
Ankur

Comment 25 Miro Hrončok 2020-01-30 15:43:47 UTC
> It didn't fail in F30 or in F32 though

F30 has RPM version that doesn't fail, but the error is in the log:

error: Illegal char '*' (0x2a) in: 1.*
error: Illegal char '*' (0x2a) in: 1.*
error: Illegal char '*' (0x2a) in: 4.*


F31 and F32 have RPM version that fails.

F31 has not yet received the fix for the == x.y.* case.

Comment 26 Ankur Sinha (FranciscoD) 2020-01-30 16:01:00 UTC
Ah, no worries. I'll tweak the requirements file to use the ">=x.y, <x+1" type specs for the time being.

Comment 27 Miro Hrončok 2020-01-30 16:19:32 UTC
Thanks.

Comment 28 Ben Cotton 2020-02-11 17:22:18 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 32 development cycle.
Changing version to 32.

Comment 29 Miro Hrončok 2020-03-02 18:02:24 UTC
Chances are * in != still blows up:

astropy>=2.0.12,!=3.0.*,!=3.1,!=3.1.1

Comment 30 Tomas Orsava 2020-04-15 11:58:22 UTC
Fixed upstream. I'll backport it a bit later with some other changes (unless somebody else does it).


Note You need to log in before you can comment on or make changes to this bug.