Technically, setuptools is merely a build dependency of pyftpdlib. If pip required setuptools because of pyftpdlib, should it also require all the other build dependencies of all the existing Python packages? Should it require flit, poetry, numpy, cython, in case some setup.py script out there in the wild imports from them?


OTOH Since the wast majority of existing Python packages import setuptools from setup.py, we decided to recommend setuptools to support the most common use case by default. Users who explicitly opt-out of setuptools obviously know that they don't need them.


In my opinion, a missing optional dependency might cause hard failures under special circumstances. E.g. if the user wants to use an optional feature, it should fail if the optional dependency required for that feature is not installed, no?



About the pyftpdlib use case: You can explicitly pass the --use-pep517 option to install that package or you can submit a pull request to upstream that adds a pyproject.toml file with setuptools build backend:

[build-system]
# Minimum requirements for the build system to execute.
requires = ["setuptools", "wheel"]  # PEP 508 specifications.
build-backend = "setuptools.build_meta"


That way, pip will actually know it should install setuptools to the build environment.

See also https://github.com/pypa/pip/issues/9175

To clarify things, only some packages fail to install with `pip install` if setuptools is not installed on the system. All of the following conditions must be met to get the failure:

 - the package must not have a suitable wheel on PyPI, only sdist¹
 - the sdist must not contain pyproject.toml²
 - the sdist must contain setup.py which does an unconditional import of setuptools


¹ Or the user must opt into using sdist-only explicitly, which is not the default.
² I am not currently sure if the pyproject.toml must be entirely missing, or if it might exist without build-system.build-backend set.

> Technically, setuptools is merely a build dependency of pyftpdlib. If pip required setuptools because of pyftpdlib, should it also require all the other build dependencies of all the existing Python packages? Should it require flit, poetry, numpy, cython, in case some setup.py script out there in the wild imports from them?

No idea, seems like pip should pull those build deps if needed, but I know very little about how this works and I don't want to dig into it.

> In my opinion, a missing optional dependency might cause hard failures under special circumstances. E.g. if the user wants to use an optional feature, it should fail if the optional dependency required for that feature is not installed, no?

The thing here is I'm a mere clueless user of pip and I'm not able to distinguish the cases for which I need setuptools and for which I don't. And I don't think you can ask such users to dig into implementation details to find out whether they need this weak dep (and having to trial and error is ultimately just as bad).

> OTOH Since the wast majority of existing Python packages import setuptools from setup.py, we decided to recommend setuptools to support the most common use case by default.

If setuptools is actually required for the majority of packages, then it's not the special case. The special case is then those packages that don't require it...

And we're coming back to the semantics of weak deps, which seem to lump a lot of varying situations into a few categories which sometimes fit and sometimes not so well.

FWIW my use case is our CI, where we disable weak deps altogether, as we don't need the bells and whistles of the packages, just the core functionality... which fails for this case. Now what should we do, enable all weak deps and get a buch of extra deps pulled in? Or special-case this dep even though it's arbitrary from our point of view, we don't care about it, we just want a working pip?

I'll leave the resolution to you, but again from my (dumb user) point of view pip is meant to install packages and if it tracebacks in majority of cases without setuptools, that it seems like a strong dependency, not a weak one.

Also from a clean user experience perspective, if it (by decision of its developers/maintainers) is a weak dependency, I would expect it to not traceback, but print a message telling me I need to install X and Y weak deps to be able to install this package. That's obviously asking a bit too much as I can imagine that wouldn't be too pretty, but the traceback makes the matters worse.

(In reply to Lukáš Hrázký from comment #4)
> > Technically, setuptools is merely a build dependency of pyftpdlib. If pip required setuptools because of pyftpdlib, should it also require all the other build dependencies of all the existing Python packages? Should it require flit, poetry, numpy, cython, in case some setup.py script out there in the wild imports from them?
> 
> No idea, seems like pip should pull those build deps if needed, but I know
> very little about how this works and I don't want to dig into it.

It actually does that, but only for packages that clearly communicate they need some build deps: pyftpdlib doesn't.


> > In my opinion, a missing optional dependency might cause hard failures under special circumstances. E.g. if the user wants to use an optional feature, it should fail if the optional dependency required for that feature is not installed, no?
> 
> The thing here is I'm a mere clueless user of pip and I'm not able to
> distinguish the cases for which I need setuptools and for which I don't. And
> I don't think you can ask such users to dig into implementation details to
> find out whether they need this weak dep (and having to trial and error is
> ultimately just as bad).

I can relate to this opinion as I like things to "just work" as well, however, this really depends on what you are installing -- I don't see an easy way out. This is the first user report we got and setuptools has been only-recommended since Fedora 33. That makes me think that most of our users are not impacted by this decision.


> > OTOH Since the wast majority of existing Python packages import setuptools from setup.py, we decided to recommend setuptools to support the most common use case by default.
> 
> If setuptools is actually required for the majority of packages, then it's
> not the special case. The special case is then those packages that don't
> require it...

No, the special case is when they require it, but they don't declare it. There are plenty of packages like pyftpdlib (most of the old packages are like that I'd say), but the most popular packages on PyPI have wheels and/or they specify their build dependencies and users won't generally see this problem.


> And we're coming back to the semantics of weak deps, which seem to lump a
> lot of varying situations into a few categories which sometimes fit and
> sometimes not so well.
> 
> FWIW my use case is our CI, where we disable weak deps altogether, as we
> don't need the bells and whistles of the packages, just the core
> functionality... which fails for this case. Now what should we do, enable
> all weak deps and get a buch of extra deps pulled in? Or special-case this
> dep even though it's arbitrary from our point of view, we don't care about
> it, we just want a working pip?

I'd say go and fix pyftpdlib, so you don't have this problem. Or special case setuptools if installing packages like pyftpdlib is required for your CI to work.


> I'll leave the resolution to you, but again from my (dumb user) point of
> view pip is meant to install packages and if it tracebacks in majority of
> cases without setuptools, that it seems like a strong dependency, not a weak
> one.

I wouldn't say it tracebacks in the majority of cases. But that's just an opinion, unfortunately, we don't have any data.


> Also from a clean user experience perspective, if it (by decision of its
> developers/maintainers) is a weak dependency, I would expect it to not
> traceback, but print a message telling me I need to install X and Y weak
> deps to be able to install this package. That's obviously asking a bit too
> much as I can imagine that wouldn't be too pretty, but the traceback makes
> the matters worse.

That's virtually impossible. With pyftpdlib, you get "ModuleNotFoundError: No module named 'setuptools'" but with a different package, you might get "ModuleNotFoundError: No module named 'cython'" or "error: [Errno 2] No such file or directory: 'gcc'". We would need to catch those errors and figure out how to translate them to actionable error messages.

Note that `pip install <something>` basically means "go download a tarball from the internet and run a script from it" (at least when wheel or pyproject.toml is not available). The script can fail to execute for thousands of reasons.

> This is the first user report we got and setuptools has been only-recommended since Fedora 33.

We've only hit this on Fedora 35. I won't go looking, I assume setuptools was pulled in by something else on earlier Fedoras...

> No, the special case is when they require it, but they don't declare it. There are plenty of packages like pyftpdlib (most of the old packages are like that I'd say), but the most popular packages on PyPI have wheels and/or they specify their build dependencies and users won't generally see this problem.

I see, then it's more of a packaging issue of pyftpdlib and a lot of others.

> I'd say go and fix pyftpdlib, so you don't have this problem.

I likely won't just because it's not an efficient time investment :( but if you'd have pointers at hand and/or it was simple enough...

I'm confused by this in the output (if you don't want to spend time on this it's absolutely fine):
command: /usr/bin/python3 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-8wqv8hnm/pyftpdlib_13b96f7bef754e36b0e8130ebf8a9e07/setup.py'"'"'; __file__='"'"'/tmp/pip-install-8wqv8hnm/pyftpdlib_13b96f7bef754e36b0e8130ebf8a9e07/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-xvold042

First, this is ugly, second, it directly imports the setuptools there. I just checked pyftpdlib code and I don't see this, is it coming from pip itself?

pyftpdlib imports setuptools conditionally in setup.py, seems it's not a hard req. It's also present in some Makefile targets which I'm not sure are related...

This is indeed ugly. I wonder what does pip actually tries to accomplish here, considering it import setuptools even when setup.py does not :/ Most likely it assumes that setup.py is present => setuptools will be used. 

https://github.com/pypa/pip/blob/main/src/pip/_internal/utils/setuptools_build.py

If pip uses setuptools like this, maybe the requirement is in order after all (or maybe it only sues it when certain conditions ar met, I'll need to check).


I won't be able to look into that soon, but feel free to keep this open and I'll eventually get to it.


> I likely won't just because it's not an efficient time investment :( but if you'd have pointers at hand and/or it was simple enough...

As said, including a file called pyproject.toml with this content in the release tarball uploaded to pypi should do:

[build-system]
# Minimum requirements for the build system to execute.
requires = ["setuptools", "wheel"]  # PEP 508 specifications.
build-backend = "setuptools.build_meta"

OK, I've checked and even with setup.py like this:

from distutils.core import setup
setup()

pip fails with:

  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  ModuleNotFoundError: No module named 'setuptools'


This makes me wonder whether pip should indeed produce a nicer error message in this case, recommending installing setuptools or using --use-pep517.

I'll bring this up on https://discuss.python.org/c/packaging/14 next week.

> This makes me wonder whether pip should indeed produce a nicer error message in this case

Or add an implicit dependency on setuptools and install it automatically, if it knows it's required?

Thanks for looking into this.

That is a possibility, but I'd like to avoid that for practical reasons (explicitly removing setuptools should still remain possible).

(In reply to Miro Hrončok from comment #8)
> I'll bring this up on https://discuss.python.org/c/packaging/14 next week.

https://discuss.python.org/t/pip-without-setuptools-could-the-experience-be-improved/11810

https://github.com/pypa/pip/pull/10717 was approved upstream. Wehn merged, we might want to backport it.

PR was finally merged. I've opened PR to backport it.

https://src.fedoraproject.org/rpms/python-pip/pull-request/104

FEDORA-2022-d91de515c5 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2022-d91de515c5

FEDORA-2022-d91de515c5 has been pushed to the Fedora 37 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2023-446b0b621c has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-446b0b621c