Bug 2029651 - Review Request: python-frozenlist - List-like structure which can be made immutable
Summary: Review Request: python-frozenlist - List-like structure which can be made imm...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: Package Review
Version: rawhide
Hardware: All
OS: Linux
medium
medium
Target Milestone: ---
Assignee: Miro Hrončok
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: 2026630 2029682
TreeView+ depends on / blocked
 
Reported: 2021-12-07 01:11 UTC by Ben Beasley
Modified: 2021-12-22 02:06 UTC (History)
2 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2021-12-13 17:37:56 UTC
Type: ---
Embargoed:
mhroncok: fedora-review+


Attachments (Terms of Use)

Description Ben Beasley 2021-12-07 01:11:35 UTC
Spec URL: https://music.fedorapeople.org/python-frozenlist.spec
SRPM URL: https://music.fedorapeople.org/python-frozenlist-1.2.0-1.fc35.src.rpm

Description:

FrozenList is a list-like structure which implements
collections.abc.MutableSequence, and which can be made immutable.

Fedora Account System Username: music

Koji scratch builds:
F36: https://koji.fedoraproject.org/koji/taskinfo?taskID=79664588
F35: https://koji.fedoraproject.org/koji/taskinfo?taskID=79664683
F34: https://koji.fedoraproject.org/koji/taskinfo?taskID=79664685

Comment 1 Ben Beasley 2021-12-07 01:18:21 UTC
I have fixed a typo in the -doc subpackage’s summary. The spec and SRPM URLs are unchanged.

Comment 2 Miro Hrončok 2021-12-07 10:02:06 UTC
Upstream has no mechanism to regenerate sources with Cython during the build? Should we contribute that?

Comment 3 Miro Hrončok 2021-12-07 10:06:53 UTC
I am not particularly keen on the  "Temporary local installation" -- is setting PYTHONPATH to the built directory (depending on pip version, it might be in _pyproject_buildir or PWD) not enough?

Comment 4 Miro Hrončok 2021-12-07 10:20:58 UTC
OTOH If you actually need it, would %pyproject_install -T <TARGET> help?

Comment 5 Ben Beasley 2021-12-07 13:38:12 UTC
(In reply to Miro Hrončok from comment #2)
> Upstream has no mechanism to regenerate sources with Cython during the
> build? Should we contribute that?

There is a Makefile (https://github.com/aio-libs/frozenlist/blob/v1.2.0/Makefile) with a “cythonize” target. The interesting part looks like:

> .install-cython:
> 	pip install -r requirements/cython.txt
> 	touch .install-cython
> 
> frozenlist/%.c: frozenlist/%.pyx
> 	cython -3 -o $@ $< -I frozenlist
> 
> cythonize: .install-cython $(PYXS:.pyx=.c)

To use it, I would need to either switch to the GitHub tarball or include it as an additional source, then trick it into not pip-installing Cython:

> touch .install_cython
> %make_build cythonize

That works, and it’s not too hard to do, but it didn’t seem worth switching sources for given the actual Cython invocation is so simple.

Comment 6 Ben Beasley 2021-12-07 13:45:45 UTC
(In reply to Miro Hrončok from comment #3)
> I am not particularly keen on the  "Temporary local installation" -- is
> setting PYTHONPATH to the built directory (depending on pip version, it
> might be in _pyproject_buildir or PWD) not enough?

I’ve used this method a few times, and I like the reliability but not the ugliness—especially copying the pip invocation from pyproject-rpm-macros. It’s well worth revisiting.

Setting PYTHONPATH=$PWD, PYTHONPATH=$PWD/src, or similar is typically fine for normal pure-Python packages, but it definitely doesn’t work for projects with compiled extensions or generated code under the current combination of pip and pyproject-rpm-macros.

The only free-standing copy of the built extension is at a path like:

> ./build/lib.linux-x86_64-3.10/frozenlist/_frozenlist.cpython-310-x86_64-linux-gnu.so

but I believe that path is chosen by the Python build system. I guess I could imitate it with

> PYTHONPATH='build/lib.linux-%{_arch}-%{python3_version}'

and hope it remains stable.

I remember this path changing with different versions of setuptools in the past, prior to the introduction of pyproject-rpm-macros, but maybe it’s done changing for the time being, at least with the library versions in stable Fedora releases. Since packages using pyproject-rpm-macros require a lot of other changes to backport to EPEL7-8 anyway, it doesn’t matter so much if it’s different there.

I just tested on F34–F36, and this pattern works for all primary architectures, so this is a viable approach.

Beyond that, the only other copy of the compiled extension is inside the wheel. In particular, %{_pyproject_builddir} doesn’t have anything interesting after the build:

> + find /builddir/build/BUILD/frozenlist-1.2.0/.pyproject-builddir
> /builddir/build/BUILD/frozenlist-1.2.0/.pyproject-builddir
> /builddir/build/BUILD/frozenlist-1.2.0/.pyproject-builddir/pip-ephem-wheel-cache-1h_4pyl1
> /builddir/build/BUILD/frozenlist-1.2.0/.pyproject-builddir/pip-wheel-qqkfb8ts

> OTOH If you actually need it, would %pyproject_install -T <TARGET> help?

I haven’t tested it, but it seems like this would successfully eliminate the long, messy installation command and improve it a great deal. Either I didn’t find that option in the past, or I tried it but using %pyproject_install twice broke %pyproject_save_files/%pyproject_files.

In the end, I’m happy enough with

> PYTHONPATH='build/lib.linux-%{_arch}-%{python3_version}'

that I’ll do that instead, although I’d be even happier if I could definitively trust this path not to change over time.

Comment 7 Ben Beasley 2021-12-07 13:53:29 UTC
- Added a mention of the GitHub Makefile in a spec file comment next to the cython invocation

- Simplified documentation build, eliminating the temporary install

New spec URL: https://music.fedorapeople.org/20211207/python-frozenlist.spec
New SRPM URL: https://music.fedorapeople.org/20211207/python-frozenlist-1.2.0-1.fc35.src.rpm

Comment 8 Ben Beasley 2021-12-07 14:55:48 UTC
It turns out that this package doesn’t use autodoc or anything similar, so it doesn’t even import itself during the documentation build at all. I guess I forgot to check that. So setting PYTHONPATH to build the documentation can be dropped entirely, and I’ll update this package again to do that.

-----

For the general case, I tried

> PYTHONPATH='build/lib.linux-%{_arch}-%{python3_version}'

on another package awaiting review, https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=2020886., and found that it *almost* worked, so long as I formed an absolute path instead:

> PYTHONPATH="${PWD}:build/lib.linux-%{_arch}-%{python3_version}"

but it doesn’t quite work, because %{_arch} is i386 on x86, but the actual path is build/lib.linux-i686-3.10, and similarly %{_arch} is arm on armv7hl, but the actual path is build/lib.linux-armv7l-3.10. I now remember trying this and running into the same problem in the past.

A heuristic approach, using a pattern to find the absolute path to the build directory, does work on all architectures:

> PYTHONPATH="$(
>   find "${PWD}/build" -type d -name 'lib.linux-*-%{python3_version}' -print -quit
> )" %make_build -C docs latex SPHINXOPTS='%{?_smp_mflags}'

and maybe that’s the way to go. I don’t really like it, but I can live with it, and it’s a bit simpler than the temporary install.

Comment 9 Ben Beasley 2021-12-07 14:56:38 UTC
- No longer set PYTHONPATH for documentation build

New spec URL: https://music.fedorapeople.org/20211207-2/python-frozenlist.spec
New SRPM URL: https://music.fedorapeople.org/20211207-2/python-frozenlist-1.2.0-1.fc35.src.rpm

Comment 10 Miro Hrončok 2021-12-07 15:08:22 UTC
> Added a mention of the GitHub Makefile in a spec file comment next to the cython invocation

That settles it at least for me.

--------------


About the PYTHONPATH thing:


PYTHONPATH=${PWD}/build/lib.%{python3_platform}-%{python3_version} is a more reliable value (it works on all architectures), but it requires pip with in-tree builds, which is only happening on Fedora 36+ (pip 21.3+).

We could enable in-tree-build on Fedora 34/35 but I am afraid that it is not very backward compatible. We might add an opt-in mechanism for packagers who want to use this, though -- would that be helpful?

%{_pyproject_builddir} is mostly useless with in-tree build and once F35 goes EOL, it will be deprecated but I'd keep it defined for EL9-compatibility.


(It worked on F34-F35 only because it was not required in the first place.)


As a side note, we might replace pip with build/install projects in the macro implementations and this might all change a bit, but so would the direct pip invocation.

Comment 11 Ben Beasley 2021-12-07 16:04:24 UTC
You can see why I settled on installing the wheel into a temporary directory in these cases—I know that pyproject-rpm-macros will continue to build a wheel somehow, and it seems like almost everything else keeps changing. (The Python packaging ecosystem in a nutshell.)

I hadn’t found %{python3_platform}. That helps, and

> PYTHONPATH="${PWD}/build/lib.%{python3_platform}-%{python3_version}"

indeed works on F36. I’m happy enough to migrate to this in F36+. Maybe this path would be a good candidate for wrapping up in Yet Another RPM Macro.

----

Meanwhile, the version with “find” works on F34-F35 by obtaining something like:

> PYTHONPATH=/builddir/build/BUILD/cyipopt-1.1.0/.pyproject-builddir/pip-req-build-g_2omwec/build/lib.linux-armv7l-3.9

as long as I do “find "${PWD}" …” instead of “find "${PWD}/build …”. I can’t form that path without using something like “find” due to the temporary directory name in the middle.

So on F34-F35, and I guess in EPEL9 when pyproject-rpm-macros start working there instead of failing with “No matching package to install: 'python3dist(tox-current-env) >= 0.0.6'”, I can choose one of:

- PYTHONPATH="$(find "${PWD}" …)"
- Keep doing a temporary wheel install

I guess if there were a way to opt in to in-source builds to get a predictable PYTHONPATH I would use that. I’m not sure it’s worth it for F34–F35 alone, since both will be EOL within a year and it doesn’t simplify the spec file much compared to the version with “find”, but maybe it’s worth it if it can be made to work in EPEL9.

Comment 12 Miro Hrončok 2021-12-12 16:34:16 UTC
Package Review
==============


Package APPRVOED.


Legend:
[x] = Pass, [!] = Fail, [-] = Not applicable, [?] = Not evaluated

===== MUST items =====

C/C++:
[-]: Development (unversioned) .so files in -devel subpackage, if present.
     Note: Unversioned so-files in private %_libdir subdirectory (see
     attachment). Verify they are not in ld path.

Generic:
[x]: Package is licensed with an open-source compatible license and meets
     other legal requirements as defined in the legal section of Packaging
     Guidelines.
[x]: License field in the package spec file matches the actual license.
[x]: License file installed when any subpackage combination is installed.
[x]: %build honors applicable compiler flags or justifies otherwise.
[x]: Package contains no bundled libraries without FPC exception.
[x]: Changelog in prescribed format.
[x]: Sources contain only permissible code or content.
[-]: Package contains desktop file if it is a GUI application.
[-]: Development files must be in a -devel package
[x]: Package uses nothing in %doc for runtime.
[x]: Package consistently uses macros (instead of hard-coded directory
     names).
[x]: Package is named according to the Package Naming Guidelines.
[x]: Package does not generate any conflict.
[x]: Package obeys FHS, except libexecdir and /usr/target.
[-]: If the package is a rename of another package, proper Obsoletes and
     Provides are present.
[x]: Requires correct, justified where necessary.
[x]: Spec file is legible and written in American English.
[-]: Package contains systemd file(s) if in need.
[x]: Useful -debuginfo package or justification otherwise.
[x]: Package is not known to require an ExcludeArch tag.
[x]: Package complies to the Packaging Guidelines
[x]: Package successfully compiles and builds into binary rpms on at least
     one supported primary architecture.
[x]: Package installs properly.
[x]: Rpmlint is run on all rpms the build produces.
     Note: There are rpmlint messages (see attachment).
[x]: If (and only if) the source package includes the text of the
     license(s) in its own file, then that file, containing the text of the
     license(s) for the package is included in %license.
[x]: Package requires other packages for directories it uses.
[x]: Package must own all directories that it creates.
[x]: Package does not own files or directories owned by other packages.
[x]: Package uses either %{buildroot} or $RPM_BUILD_ROOT
[x]: Package does not run rm -rf %{buildroot} (or $RPM_BUILD_ROOT) at the
     beginning of %install.
[x]: Macros in Summary, %description expandable at SRPM build time.
[x]: Package does not contain duplicates in %files.
[x]: Permissions on files are set properly.
[x]: Package must not depend on deprecated() packages.
[x]: Package use %makeinstall only when make install DESTDIR=... doesn't
     work.
[x]: Package is named using only allowed ASCII characters.
[x]: Package does not use a name that already exists.
[x]: Package is not relocatable.
[x]: Sources used to build the package match the upstream source, as
     provided in the spec URL.
[x]: Spec file name must match the spec package %{name}, in the format
     %{name}.spec.
[x]: File names are valid UTF-8.
[x]: Large documentation must go in a -doc subpackage. Large could be size
     (~1MB) or number of files.
     Note: Documentation size is 0 bytes in 0 files.
[x]: Packages must not store files under /srv, /opt or /usr/local

Python:
[x]: Python eggs must not download any dependencies during the build
     process.
[x]: A package which is used by another package via an egg interface should
     provide egg info.
[x]: Package meets the Packaging Guidelines::Python
[x]: Package contains BR: python2-devel or python3-devel
[x]: Packages MUST NOT have dependencies (either build-time or runtime) on
     packages named with the unversioned python- prefix unless no properly
     versioned package exists. Dependencies on Python packages instead MUST
     use names beginning with python2- or python3- as appropriate.
[x]: Python packages must not contain %{pythonX_site(lib|arch)}/* in %files
[x]: Binary eggs must be removed in %prep

===== SHOULD items =====

Generic:
[-]: If the source package does not include license text(s) as a separate
     file from upstream, the packager SHOULD query upstream to include it.
[x]: Final provides and requires are sane (see attachments).
[-]: Fully versioned dependency in subpackages if applicable.
[?]: Package functions as described.
[?]: Latest version is packaged.
[x]: Package does not include license text files separate from upstream.
[x]: Patches link to upstream bugs/comments/lists or are otherwise
     justified.
[-]: Sources are verified with gpgverify first in %prep if upstream
     publishes signatures.
     Note: gpgverify is not used.
[?]: Package should compile and build into binary rpms on all supported
     architectures.
[x]: %check is present and all tests pass.
[?]: Packages should try to preserve timestamps of original installed
     files.
[x]: Reviewer should test that the package builds in mock.
[x]: Buildroot is not present
[x]: Package has no %clean section with rm -rf %{buildroot} (or
     $RPM_BUILD_ROOT)
[x]: No file requires outside of /etc, /bin, /sbin, /usr/bin, /usr/sbin.
[x]: Packager, Vendor, PreReq, Copyright tags should not be in spec file
[x]: Sources can be downloaded from URI in Source: tag
[x]: SourceX is a working URL.
[x]: Spec use %global instead of %define unless justified.

===== EXTRA items =====

Generic:
[x]: Rpmlint is run on all installed packages.
     Note: There are rpmlint messages (see attachment).
[x]: Large data in /usr/share should live in a noarch subpackage if package
     is arched.
[x]: Spec file according to URL is the same as in SRPM.


Rpmlint
-------
============================ rpmlint session starts ============================
rpmlint: 2.1.0
configuration:
    /usr/lib/python3.10/site-packages/rpmlint/configdefaults.toml
    /etc/xdg/rpmlint/fedora.toml
    /etc/xdg/rpmlint/licenses.toml
    /etc/xdg/rpmlint/scoring.toml
    /etc/xdg/rpmlint/users-groups.toml
    /etc/xdg/rpmlint/warn-on-functions.toml
checks: 31, packages: 5

python3-frozenlist-debuginfo.x86_64: W: unstripped-binary-or-object /usr/lib/debug/usr/lib64/python3.10/site-packages/frozenlist/_frozenlist.cpython-310-x86_64-linux-gnu.so-1.2.0-1.fc36.x86_64.debug
python3-frozenlist-debuginfo.x86_64: E: shared-library-without-dependency-information /usr/lib/debug/usr/lib64/python3.10/site-packages/frozenlist/_frozenlist.cpython-310-x86_64-linux-gnu.so-1.2.0-1.fc36.x86_64.debug
python-frozenlist-debugsource.x86_64: W: no-documentation
python3-frozenlist-debuginfo.x86_64: W: no-documentation
python3-frozenlist.x86_64: W: no-documentation
python-frozenlist.spec:111: W: macro-in-%changelog %autochangelog
python3-frozenlist-debuginfo.x86_64: W: dangling-relative-symlink /usr/lib/debug/.build-id/86/de7e36ca6ac3dca5b7cafdb447dfe1781fdd27 ../../../.build-id/86/de7e36ca6ac3dca5b7cafdb447dfe1781fdd27
 5 packages and 0 specfiles checked; 1 errors, 6 warnings, 1 badness; has taken 3.0 s 

All false positives or not relevant.


Unversioned so-files
--------------------
python3-frozenlist: /usr/lib64/python3.10/site-packages/frozenlist/_frozenlist.cpython-310-x86_64-linux-gnu.so

Source checksums
----------------
https://files.pythonhosted.org/packages/source/f/frozenlist/frozenlist-1.2.0.tar.gz :
  CHECKSUM(SHA256) this package     : 68201be60ac56aff972dc18085800b6ee07973c49103a8aba669dee3d71079de
  CHECKSUM(SHA256) upstream package : 68201be60ac56aff972dc18085800b6ee07973c49103a8aba669dee3d71079de


Requires
--------
python3-frozenlist (rpmlib, GLIBC filtered):
    libc.so.6()(64bit)
    python(abi)
    rtld(GNU_HASH)

python-frozenlist-doc (rpmlib, GLIBC filtered):

python-frozenlist-debugsource (rpmlib, GLIBC filtered):



Provides
--------
python3-frozenlist:
    python-frozenlist
    python3-frozenlist
    python3-frozenlist(x86-64)
    python3.10-frozenlist
    python3.10dist(frozenlist)
    python3dist(frozenlist)

python-frozenlist-doc:
    python-frozenlist-doc

python-frozenlist-debugsource:
    python-frozenlist-debugsource
    python-frozenlist-debugsource(x86-64)



Generated by fedora-review 0.7.0 (fed5495) last change: 2019-03-17
Command line :try-fedora-review -b 2029651 -m fedora-rawhide-x86_64
Buildroot used: fedora-rawhide-x86_64
Active plugins: Generic, Python, Shell-api
Disabled plugins: Ocaml, SugarActivity, PHP, R, Haskell, C/C++, Java, Perl, Ruby, fonts
Disabled flags: EPEL6, EPEL7, DISTTAG, BATCH, EXARCH

Comment 13 Ben Beasley 2021-12-12 16:46:38 UTC
Thanks! Repo requested.

Once I’ve built this and python-aiosignal in Rawhide I’ll look at making a PR to fix python-aiohttp.

Comment 14 Gwyn Ciesla 2021-12-13 16:04:26 UTC
(fedscm-admin):  The Pagure repository was created at https://src.fedoraproject.org/rpms/python-frozenlist

Comment 15 Fedora Update System 2021-12-13 17:33:21 UTC
FEDORA-2021-46a4e4ad5f has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2021-46a4e4ad5f

Comment 16 Fedora Update System 2021-12-13 17:37:56 UTC
FEDORA-2021-46a4e4ad5f has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 17 Fedora Update System 2021-12-13 19:57:24 UTC
FEDORA-2021-648bda03df has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2021-648bda03df

Comment 18 Fedora Update System 2021-12-13 20:11:46 UTC
FEDORA-2021-dc2c23a3d1 has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-dc2c23a3d1

Comment 19 Fedora Update System 2021-12-14 01:08:41 UTC
FEDORA-2021-648bda03df has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf install --enablerepo=updates-testing --advisory=FEDORA-2021-648bda03df \*`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-648bda03df

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 20 Fedora Update System 2021-12-14 01:35:51 UTC
FEDORA-2021-dc2c23a3d1 has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf install --enablerepo=updates-testing --advisory=FEDORA-2021-dc2c23a3d1 \*`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-dc2c23a3d1

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 21 Fedora Update System 2021-12-22 01:13:08 UTC
FEDORA-2021-dc2c23a3d1 has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 22 Fedora Update System 2021-12-22 02:06:26 UTC
FEDORA-2021-648bda03df has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.