Description of problem: I thought I would try out the pyproject macros, including dynamic BuildRequires. My first attempt was with python-manuel. The build ended with an error, however: No matching package to install: 'python3dist(zope-testing)' Package python3-zope-testing-4.6.1-11.fc32.noarch is already installed. Error: Some packages could not be found. The package in question actually provides python3dist(zope.testing). I don't see "zope-testing" anywhere in the manuel sources. A dot is used in every case. Therefore I suspect that something in pyproject incorrectly changed the dot to a hyphen. If that is not the case, then I need help understanding how to write the spec file to avoid this error. Adding BuildRequires: python3dist(zope.testing) to the spec file doesn't help. The build still fails the same way. Version-Release number of selected component (if applicable): pyproject-rpm-macros-0-10.fc32.noarch How reproducible: Always Steps to Reproduce: 1. Change python-manuel.spec to BR only pyproject-rpm-macros 2. Also add a %generate_buildrequires section with %pyproject_buildrequires -t 3. Attempt to build in mock Actual results: The error noted above. Expected results: The right BR should be generated. Additional info:
Investigation: 1. PyPI redirects zope-testing to zope.testing 2. %{py_dist_name zope.testing} **does not** convert a dot to a hyphen 3. python3-zope-testing provides python3dist(zope.testing) because (2) 4. https://packaging.pypa.io/en/latest/utils/#packaging.utils.canonicalize_name **does** convert a dot to a hyphen 5. pyproject macros use (4) and generate requirement on python3dist(zope-testing) The inconsistency is where the problem is from. Let me find if we should or should not, convert a dot to hyphen. Googling the PEP that defines the "canonical package name" term is harder than expected.
https://www.python.org/dev/peps/pep-0503/#normalized-names > re.sub(r"[-_.]+", "-", name).lower() py_dist_name is wrong here -> let's fix it ASAP before the mass rebuild (however, we do need to provide both names and require the old name for a while to mitigate the transition period), sigh.
Unfortunately, this was once converted to contradict the standard: https://src.fedoraproject.org/rpms/python-rpm-macros/c/6cc8000e1491ed61cdaedbb9762ba225111c6000 https://bugzilla.redhat.com/show_bug.cgi?id=1564095 It seems that the RPM dep generator does this the other way around and we have only changed the macro to make it behave the same. Reading the code, the dependency generator uses Distribution.key: a short for dist.project_name.lower(), the project_name argument is passed through the safe_name() utility function to filter out any unacceptable characters. https://setuptools.readthedocs.io/en/latest/pkg_resources.html#distribution-attributes safe_name(): Return a “safe” form of a project’s name, suitable for use in a Requirement string, as a distribution name, or a PyPI project name. All non-alphanumeric runs are condensed to single “-” characters, such that a name like “The $$$ Tree” becomes “The-Tree”. Note that if you are generating a filename from this value you should combine it with a call to to_filename() so all dashes (“-“) are replaced by underscores (“_”). See to_filename(). https://setuptools.readthedocs.io/en/latest/pkg_resources.html#parsing-utilities It doesn't say whether it converts dot, but it doesn't. There is a test in setuptools: assert safe_name("peak.web") != "peak-web" Since 2006 https://github.com/pypa/setuptools/commit/51d68aa576cd63dab44ed6f9578211a0e90def9a We have once changed this in setuptools: https://github.com/pypa/setuptools/pull/1324 But it was reverted: https://github.com/pypa/setuptools/commit/cdcc7e5f1ecb6e485a71676a071d1b37adac3d51 And here it goes, blocked on... wait for it, us: https://github.com/pypa/setuptools/pull/1492 Either way, we should really make up our mind about whether dots are OK or not, but in the meantime, I am afraid we really do need to provide both :(
OTOH pyproject macros is the only thing that blows up by following the docs, so we can change pyproject macros and the documentation instead. However, if setuptools will eventually change, everything will blow up once again.
At least as the upstream author of the generator, from my perspective there's nothing to fix, because I'm using pkg_resources to give me canonicalized names. And I'm very glad I did, because I don't want to think of all these cases. :/
Technically, you are using safe_name, which turns out not to be the canonicalized name, but something a bit different. If communicated/documented properly, this is fine.
(In reply to Miro Hrončok from comment #6) > Technically, you are using safe_name, which turns out not to be the > canonicalized name, but something a bit different. If > communicated/documented properly, this is fine. That's true, but I didn't know that based on documentation. :) It's also pretty cool that this is the first time this has come up in almost a decade of this...
Second time: https://bugzilla.redhat.com/show_bug.cgi?id=1564095
Our documentation is not accurate: https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_requires_and_buildrequires_with_standardized_names %{py_dist_name 0-._.-._.-._.-._.-._.-._.-0} # => 0-0 $ rpm --eval '%{py_dist_name 0-._.-._.-._.-._.-._.-._.-0}' 0-.-.-.-.-.-.-.-.-.-.-.-.-0 We should really doctest it :(
One problem with the current behavior is that if upstream specifies the dependency as zope-testing, it works with pip but breaks with RPM.
Jerry, thanks for the report! Not an easy fix. You'll probably want to avoid using pyproject macros for Zope-related things for now. IMO we really need to follow the standard (PEP 503). Do not assume setuptools follows/implements standards. It doesn't -- the standards were written to fix the issues after it became clear that setuptools is unmanageable. The proper function here would be packaging.utils.canonicalize_name ( https://packaging.pypa.io/en/latest/utils/#packaging.utils.canonicalize_name ). If we don't want quick fix we'll regret later, this'll probably need a large change :( Something like: - Change the Provides generator to give *both* names (everywhere) - Change the Requires generator to give PEP 503 normalized names *in Python 3.9+ distros* (where we're doing a full rebuild) - Change the Provides generator to give normalized names (in Python 3.9+ distros)
We should probably change the Provides generator to give *both* names before the F32 mass rebuild and hence before DevConf CZ.
For future reference, 23 packages have this in specs: $ rg --sort-files 'python\S+dist\(\S+\.\S+\)' rpm-specs/ rpm-specs/ansible-lint.spec 27:BuildRequires: python3dist(ruamel.yaml) 34:Requires: python3dist(ruamel.yaml) rpm-specs/buildbot.spec 35:BuildRequires: python3dist(zope.interface) >= 4.1.1 rpm-specs/ocrmypdf.spec 27:BuildRequires: python3dist(pdfminer.six) = 20181108 rpm-specs/pipenv.spec 190:Provides: bundled(python3dist(delegator.py)) == 0.1.1 213:Provides: bundled(python3dist(backports.functools_lru_cache)) == 1.5 214:Provides: bundled(python3dist(backports.shutil_get_terminal_size)) == 1.0.0 215:Provides: bundled(python3dist(backports.shutil_get_terminal_size)) == 1.0.0 rpm-specs/python-BTrees.spec 26:BuildRequires: python3dist(repoze.sphinx.autointerface) 31:BuildRequires: python3dist(zope.interface) 32:BuildRequires: python3dist(zope.testrunner) rpm-specs/python-ZEO.spec 26:BuildRequires: python3dist(zc.lockfile) 30:BuildRequires: python3dist(zope.interface) 31:BuildRequires: python3dist(zope.testing) 32:BuildRequires: python3dist(zope.testrunner) rpm-specs/python-ZODB.spec 17:BuildRequires: python3dist(j1m.sphinxautointerface) 18:BuildRequires: python3dist(j1m.sphinxautozconfig) 23:BuildRequires: python3dist(zc.lockfile) 26:BuildRequires: python3dist(zope.testing) 27:BuildRequires: python3dist(zope.testrunner) rpm-specs/python-cheroot.spec 27:Requires: python3dist(jaraco.functools) 34:BuildRequires: python3dist(jaraco.functools) rpm-specs/python-cherrypy.spec 35:BuildRequires: python3dist(path.py) rpm-specs/python-jaraco-classes.spec 43:BuildRequires: python3dist(jaraco.packaging) >= 3.2 44:BuildRequires: python3dist(rst.linker) >= 1.9 rpm-specs/python-jaraco-functools.spec 41:BuildRequires: python3dist(jaraco.packaging) >= 3.2 42:BuildRequires: python3dist(rst.linker) >= 1.9 rpm-specs/python-jaraco-packaging.spec 35:Requires: python3dist(rst.linker) 63:BuildRequires: python3dist(rst.linker) rpm-specs/python-launchpadlib.spec 28:BuildRequires: python3dist(lazr.restfulclient) >= 0.9.19 29:BuildRequires: python3dist(lazr.uri) rpm-specs/python-manuel.spec 17:BuildRequires: python3dist(zope.testing) rpm-specs/python-persistent.spec 17:BuildRequires: python3dist(repoze.sphinx.autointerface) 20:BuildRequires: python3dist(zope.interface) 21:BuildRequires: python3dist(zope.testrunner) rpm-specs/python-portend.spec 24:BuildRequires: python3dist(jaraco.functools) 42:BuildRequires: python3dist(jaraco.packaging) >= 3.2 43:BuildRequires: python3dist(rst.linker) >= 1.9 rpm-specs/python-tempora.spec 23:BuildRequires: python3dist(jaraco.functools) >= 1.20 42:BuildRequires: python3dist(jaraco.packaging) >= 3.2 rpm-specs/python-trimesh.spec 26:BuildRequires: python3dist(svg.path) 78:Requires: python3dist(svg.path) rpm-specs/python-twisted.spec 46:BuildRequires: python3dist(zope.interface) >= 4.4.2 rpm-specs/python-txtorcon.spec 42:BuildRequires: python3dist(repoze.sphinx.autointerface) >= 0.4 rpm-specs/python-wadllib.spec 23:BuildRequires: python3dist(lazr.uri) rpm-specs/python-zope-testrunner.spec 28:BuildRequires: python3dist(zope.exceptions) 29:BuildRequires: python3dist(zope.interface) 30:BuildRequires: python3dist(zope.testing) rpm-specs/sagemath.spec 342:BuildRequires: python3dist(path.py) 481:Requires: python3dist(path.py)
Step 1: https://src.fedoraproject.org/rpms/python-rpm-generators/pull-request/6 I've also added this to the @python/python3.9 copr.
This bug appears to have been reported against 'rawhide' during the Fedora 32 development cycle. Changing version to 32.
PR ready and tested, waiting for some other changes before merging.
This still waits for https://src.fedoraproject.org/rpms/python-rpm-generators/pull-request/17
This is part of the Python 3.9 side tag now.