Bug 2012590 - Missing dependency on greenlet on several architectures
Summary: Missing dependency on greenlet on several architectures
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Fedora
Classification: Fedora
Component: python-sqlalchemy
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Nils Philippsen
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-10-10 13:57 UTC by Ben Beasley
Modified: 2021-10-20 19:47 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-10-20 19:47:14 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Ben Beasley 2021-10-10 13:57:45 UTC
Description of problem:

The python3-sqlalchemy package fails to depend on python3dist(greenlet) on i686, armv7hl, and s390x.

Version-Release number of selected component (if applicable): 1.4.25-1

I haven’t checked how many older versions, and Fedora releases, are affected.


How reproducible:


Steps to Reproduce:

1. sudo dnf install qemu-user-static
2. mock -r fedora-rawhide-i386 --dnf-cmd -- repoquery --requires python3-sqlalchemy
3. Observe “(python3.10dist(greenlet) < 0.4.17 or python3.10dist(greenlet) > 0.4.17)” does not appear in the output.
4. Compare to fedora-rawhide-i386 (also missing) and to the other architectures (present).

Actual results:

No dependency on greenlet.

Expected results:

Same dependency on greenlet as the 64-bit architectures.

Additional info:

You can see the problem by:

> $ mock -r fedora-rawhide-i386 --clean
> $ mock -r fedora-rawhide-i386 -i python3-sqlalchemy
> $ mock -r fedora-rawhide-i386 --shell
> # python3 -c 'import sqlalchemy.dialects.mysql'
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File "/usr/lib/python3.10/site-packages/sqlalchemy/dialects/mysql/__init__.py", line 57, in <module>
>     from . import asyncmy  # noqa
>   File "/usr/lib/python3.10/site-packages/sqlalchemy/dialects/mysql/asyncmy.py", line 175, in <module>
>     class AsyncAdapt_asyncmy_connection(AdaptedConnection):
>   File "/usr/lib/python3.10/site-packages/sqlalchemy/dialects/mysql/asyncmy.py", line 185, in AsyncAdapt_asyncmy_connection
>     async def _mutex_and_adapt_errors(self):
>   File "/usr/lib/python3.10/site-packages/sqlalchemy/util/concurrency.py", line 63, in asynccontextmanager
>     _not_implemented()
>   File "/usr/lib/python3.10/site-packages/sqlalchemy/util/concurrency.py", line 37, in _not_implemented
>     raise ValueError(
> ValueError: the greenlet library is required to use this function.

The root cause is in setup.cfg, which has:

> [options]
> …
> install_requires = 
>         importlib-metadata;python_version<"3.8"
>         greenlet != 0.4.17;python_version>='3' and (platform_machine=='aarch64' or (platform_machine=='ppc64le' or (platform_machine=='x86_64' or (platform_machine=='amd64' or (platform_machine=='AMD64' or (platform_machine=='win32' or platform_machine=='WIN32'))))))

Upstream has apparently attempted to list all of the architectures where they think greenlet will be installable from PyPI, but this list does not cover every architecture where Fedora provides a greenlet package. See, for history:

sqlalchemy 1.4.x: re-optionalize 'greenlet' dependency
https://github.com/sqlalchemy/sqlalchemy/issues/6136

Since we provide a python3-greenlet package for all architectures in Fedora, I think the dependency should be present on all architectures, too. A simple solution would be to patch out the platform conditionals in setup.cfg downstream:

> [options]
> …
> install_requires = 
>         importlib-metadata;python_version<"3.8"
>         greenlet != 0.4.17;python_version>='3'

I happened on this bug because python-databases, which I maintain, uses python3-sqlalchemy in its tests, including the async parts that use greenlet. It is noarch and was failing to build only when it happened to land on a koji builder of one of the mentioned architectures—which is very surprising behavior.

Comment 1 Ben Beasley 2021-10-10 14:03:11 UTC
> Same dependency on greenlet as the 64-bit architectures.

Sorry, the s390x architecture is affected and is 64-bit. This should have read “Same dependency on greenlet as the other architectures.”—that is, as x86_64, ppc64le, and aarch64.

Comment 2 Michael Bayer 2021-10-10 19:23:25 UTC
agree, since this is based on packaging and you have greenlet for all platforms, SQLAlchemy's limiting req, which is based on what people can reasonably install with pip, should be skipped.

I would note you can explicitly install all asyncio dependencies using the `sqlalchemy[asyncio]` pip target.

Comment 3 Kevin Fenzi 2021-10-19 23:18:55 UTC
We can of course patch this downstream, but that seems ugly. ;( 

Here's a scratch build with it commented: https://koji.fedoraproject.org/koji/taskinfo?taskID=77537004

I wish there was a better upstream solution here... Adding mhroncok in case he or the rest of the python team has some better solution here. :)

Comment 4 Miro Hrončok 2021-10-20 17:45:02 UTC
A better solution for what problem? If the package fails to function without greenlet, it should be required on all arches. Why does upstream do this thing? If they attempt not to depend on greenlet on architecture where greenlet is not available as a wheel on PyPI, does the package function without it?

Comment 5 Michael Bayer 2021-10-20 18:09:32 UTC
(In reply to Miro Hrončok from comment #4)
> A better solution for what problem? If the package fails to function without
> greenlet, it should be required on all arches. Why does upstream do this
> thing? If they attempt not to depend on greenlet on architecture where
> greenlet is not available as a wheel on PyPI, does the package function
> without it?

yes, the package functions without it.  the greenlet dependency for SQLAlchemy is used only for the "asyncio" extension which is optional.  if you dont use the "asyncio" extension, you don't need greenlet.    this is why we had a lot of user complaints about the greenlet dependency, because the vast majority of users didn't need it, and those users who were running on platforms that don't have build tools installed and are on an architecture for which greenlet did not publish a wheel, they couldn't use the package.

optional dependencies are a completely normal Python feature as stated in pep508 https://www.python.org/dev/peps/pep-0508/#extras.  SQLAlchemy supplies an install target for this optional dependency using the "sqlalchemy[asyncio]" extra.

Overall im not too clear on what the downstream problem is here, you can add "greenlet" as a dependency to the spec file, or simply use the "sqlalchemy[asyncio]" target.  why do the setup requirements in the library itself need to be patched?

Comment 6 Michael Bayer 2021-10-20 18:11:30 UTC
oh,  one more important thing.    Looking at the stack trace at the top:

# python3 -c 'import sqlalchemy.dialects.mysql'
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File "/usr/lib/python3.10/site-packages/sqlalchemy/dialects/mysql/__init__.py", line 57, in <module>
>     from . import asyncmy  # noqa
>   File "/usr/lib/python3.10/site-packages/sqlalchemy/dialects/mysql/asyncmy.py", line 175, in <module>
>     class AsyncAdapt_asyncmy_connection(AdaptedConnection):
>   File "/usr/lib/python3.10/site-packages/sqlalchemy/dialects/mysql/asyncmy.py", line 185, in AsyncAdapt_asyncmy_connection
>     async def _mutex_and_adapt_errors(self):
>   File "/usr/lib/python3.10/site-packages/sqlalchemy/util/concurrency.py", line 63, in asynccontextmanager
>     _not_implemented()
>   File "/usr/lib/python3.10/site-packages/sqlalchemy/util/concurrency.py", line 37, in _not_implemented
>     raise ValueError(
> ValueError: the greenlet library is required to use this function.

OK that's our bug and that's fixed in 1.4.26: https://docs.sqlalchemy.org/en/14/changelog/changelog_14.html#change-ba4c0d15b376580f1123f7ac4374cbe9  https://github.com/sqlalchemy/sqlalchemy/issues/7204

so the specific problem here is also fixed in upstream.  sorry i missed that in the description.

Comment 7 Ben Beasley 2021-10-20 18:42:56 UTC
> OK that's our bug and that's fixed in 1.4.26: https://docs.sqlalchemy.org/en/14/changelog/changelog_14.html#change-ba4c0d15b376580f1123f7ac4374cbe9  https://github.com/sqlalchemy/sqlalchemy/issues/7204

That’s what actually prompted me to file this issue. I’m happy to see a fix—thanks for pointing it out.

> Overall im not too clear on what the downstream problem is here, you can add "greenlet" as a dependency to the spec file, or simply use the "sqlalchemy[asyncio]" target.  why do the setup requirements in the library itself need to be patched?

Given that the failure to import sqlalchemy.dialects.mysql without greenlet is a separate issue, I agree that asking any projects that actually need the async functionality to require sqlalchemy[asyncio] is correct.

I still think that, as a Fedora user, the arch-dependent greenlet dependency is very surprising here. I expect that a distribution package will have different dependencies or different default-enabled functionality on different architectures only in cases where there is no other choice—due to an architecture build failure or a dependency that is inherently arch-specific.

However, I would still understand if the answer was, “it’s not actually broken, and we want to do exactly what upstream does.”

Comment 8 Miro Hrončok 2021-10-20 19:15:59 UTC
Since the package is arched, I'd do exactly what upstream does. If sqlalchemy was noarch, we would have a problem, but luckily it is not.


In upstream, I would not require greenlet on some arches, I would not require it at all -- I would leave that to the sqlalchemy[asyncio] extra. But that's not my battle to fight.

Comment 9 Ben Beasley 2021-10-20 19:47:14 UTC
Ok! Sounds like there’s a consensus—I will go ahead and close this as NOTABUG. I appreciate everyone’s input.


Note You need to log in before you can comment on or make changes to this bug.