Bug 2003287 - Review Request: python-llvmlite - A lightweight LLVM python binding for writing JIT compilers
Summary: Review Request: python-llvmlite - A lightweight LLVM python binding for writi...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: Package Review
Version: rawhide
Hardware: All
OS: Linux
medium
medium
Target Milestone: ---
Assignee: Zbigniew Jędrzejewski-Szmek
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: fedora-neuro, NeuroFedora
TreeView+ depends on / blocked
 
Reported: 2021-09-10 21:56 UTC by Ankur Sinha (FranciscoD)
Modified: 2021-09-24 20:22 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-09-21 15:32:06 UTC
Type: ---
Embargoed:
zbyszek: fedora-review+


Attachments (Terms of Use)

Description Ankur Sinha (FranciscoD) 2021-09-10 21:56:04 UTC
Spec URL: https://ankursinha.fedorapeople.org/python-llvmlite/python-llvmlite.spec

SRPM URL: https://ankursinha.fedorapeople.org/python-llvmlite/python-llvmlite-0.37.0-1.fc34.src.rpm

Description:
The old llvmpy binding exposes a lot of LLVM APIs but the mapping of C++-style
memory management to Python is error prone. Numba and many JIT compilers do not
need a full LLVM API. Only the IR builder, optimizer, and JIT compiler APIs are
necessary.

llvmlite is a project originally tailored for Numba's needs, using the
following approach:

- A small C wrapper around the parts of the LLVM C++ API we need that are not
  already exposed by the LLVM C API.
- A ctypes Python wrapper around the C API.
- A pure Python implementation of the subset of the LLVM IR builder that we
  need for Numba.

Key Benefits

- The IR builder is pure Python code and decoupled from LLVM’s
  frequently-changing C++ APIs.
- Materializing a LLVM module calls LLVM's IR parser which provides better
  error messages than step-by-step IR building through the C++ API (no more
  segfaults or process aborts).
- Most of llvmlite uses the LLVM C API which is small but very stable (low
  maintenance when changing LLVM version).
- The binding is not a Python C-extension, but a plain DLL accessed using
  ctypes (no need to wrestle with Python’s compiler requirements and C++ 11
compatibility).
- The Python binding layer has sane memory management.
- llvmlite is quite faster than llvmpy thanks to a much simpler architeture
  (the Numba test suite is twice faster than it was).

Fedora Account System Username: ankursinha

Comment 1 Zbigniew Jędrzejewski-Szmek 2021-09-11 07:11:20 UTC
It's great that you're working on this. Having Numba in Fedora would be wonderful: for
me it's one of the important remaining pieces for scientific Python work with a pure
Fedora installation.

> URL:            http://numba.pydata.org/
https://

The %description is written for a person who already knows what llvmlite is and is justifying
its existence. We need an explanation for people who have no idea. Maybe something like
this (and I'm paraphrasing the existing text based on my understanding only):

  llvmlite is a wrapper around the llvm compiler created for Numba to interact with the internal
  llvm internation representation of code. Only the IR builder, optimizer, and JIT compiler APIs
  are covered.

  llvmlite consists of a small C wrapper around some parts of the LLVM C++ API that are not
  already exposed by the LLVM C API, a ctypes wrapper around the C API to allow access
  from Python code, and a Python implementation of the subset of the LLVM IR builder that is
  needed for Numba.

The "Key Benefits" part could be included, but I'm not sure if it's useful to the users
of the package.

> %check

I think the tests should be enabled by default.
This is especially important when we're not using the same llvm version as upstream.
When enabled, they pass here without any issue.
(Also, maybe add '-v' or equivalent so we get a better log of what was executed…)

> LD_LIBRARY_PATH="$LD_LIBRARY_PATH:%{buildroot}/%{python3_sitearch}/llvmlite/binding/" PYTHONPATH="$PYTHONPATH:%{buildroot}/%{python3_sitearch}:%{buildroot}%{python3_sitelib}" %{__python3} runtests.py

Maybe:
LD_LIBRARY_PATH="%{buildroot}%{python3_sitearch}/llvmlite/binding/" PYTHONPATH="$PYTHONPATH:%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib}" %{python3} runtests.py
(less "/", we expect LD_LIBRARY_PATH to be unset, %python3 is now preferred)

'%license LICENSE' is missing.

+ package name is OK
+ latest version
+ license is acceptable for Fedora (BSD)
+ license is specified correctly
+ builds and installs OK
+ fedora-review and rpmlint find no issues (except bogus spelling suggestions and %autorelease misunderstandings)
- %check is disabled
+ BR/R/BR look correct
- license is missing from %files

Package is APPROVED, but please fix the minor issues pointed out above when importing.

Comment 2 Zbigniew Jędrzejewski-Szmek 2021-09-11 07:12:57 UTC
About Summary: maybe "Lightweight LLVM python binding for JIT compilers"
(Every other package starts with "A", so dropping it is recommended. And the "writing"
seems out of place in a title-like phrase.)

Comment 3 Ankur Sinha (FranciscoD) 2021-09-11 15:04:05 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #1)
> It's great that you're working on this. Having Numba in Fedora would be
> wonderful: for
> me it's one of the important remaining pieces for scientific Python work
> with a pure
> Fedora installation.

+1: a few of the packages on our Neuro-SIG queue also need numba, so it'll be good to have it packaged up

> 
> > URL:            http://numba.pydata.org/
> https://

Corrected

> 
> The %description is written for a person who already knows what llvmlite is
> and is justifying
> its existence. We need an explanation for people who have no idea. Maybe
> something like
> this (and I'm paraphrasing the existing text based on my understanding only):
> 
>   llvmlite is a wrapper around the llvm compiler created for Numba to
> interact with the internal
>   llvm internation representation of code. Only the IR builder, optimizer,
> and JIT compiler APIs
>   are covered.
> 
>   llvmlite consists of a small C wrapper around some parts of the LLVM C++
> API that are not
>   already exposed by the LLVM C API, a ctypes wrapper around the C API to
> allow access
>   from Python code, and a Python implementation of the subset of the LLVM IR
> builder that is
>   needed for Numba.
> 
> The "Key Benefits" part could be included, but I'm not sure if it's useful
> to the users
> of the package.

I've copied bits from here now:

https://llvmlite.readthedocs.io/en/latest/#philosophy

So it now says:

llvmlite provides a Python binding to LLVM for use in Numba.

Numba previously relied on llvmpy.  While llvmpy exposed large parts of the
LLVM C++ API for direct calls into the LLVM library, llvmlite takes an entirely
different approach. Llvmlite starts from the needs of a JIT compiler and splits
them into two decoupled tasks:

- Construction of a Module, function by function, Instruction by instruction.
- Compilation and optimization of the module into machine code.

The construction of an LLVM module does not call the LLVM C++ API. Rather, it
constructs the LLVM intermediate representation (IR) in pure Python. This is
the role of the IR layer.

The compilation of an LLVM module takes the IR in textual form and feeds it
into LLVM's parsing API. It then returns a thin wrapper around LLVM's C++
module object. This is the role of the binding layer.

Once parsed, the module's source code cannot be modified, which loses the
flexibility of the direct mapping of C++ APIs into Python that was provided by
llvmpy but saves a great deal of maintenance.



> 
> > %check
> 
> I think the tests should be enabled by default.
> This is especially important when we're not using the same llvm version as
> upstream.
> When enabled, they pass here without any issue.
> (Also, maybe add '-v' or equivalent so we get a better log of what was
> executed…)
> 
> > LD_LIBRARY_PATH="$LD_LIBRARY_PATH:%{buildroot}/%{python3_sitearch}/llvmlite/binding/" PYTHONPATH="$PYTHONPATH:%{buildroot}/%{python3_sitearch}:%{buildroot}%{python3_sitelib}" %{__python3} runtests.py

> 
> Maybe:
> LD_LIBRARY_PATH="%{buildroot}%{python3_sitearch}/llvmlite/binding/"
> PYTHONPATH="$PYTHONPATH:%{buildroot}%{python3_sitearch}:
> %{buildroot}%{python3_sitelib}" %{python3} runtests.py
> (less "/", we expect LD_LIBRARY_PATH to be unset, %python3 is now preferred)
> 

Enabled tests, updated the command, and increased the verbosity too.

> '%license LICENSE' is missing.

I've learned that it isnt needed since it gets included by the pyproject_save_files macro:

rpm -ql --licensefiles -p RPMS/python3-llvmlite-0.37.0-1.fc36.x86_64.rpm 
/usr/lib64/python3.10/site-packages/llvmlite-0.37.0.dist-info/LICENSE

> 
> + package name is OK
> + latest version
> + license is acceptable for Fedora (BSD)
> + license is specified correctly
> + builds and installs OK
> + fedora-review and rpmlint find no issues (except bogus spelling
> suggestions and %autorelease misunderstandings)
> - %check is disabled
> + BR/R/BR look correct
> - license is missing from %files
> 
> Package is APPROVED, but please fix the minor issues pointed out above when
> importing.


Thanks, requested SCM now.

Updated spec/srpm:

Spec URL: https://ankursinha.fedorapeople.org/python-llvmlite/python-llvmlite.spec
SRPM URL: https://ankursinha.fedorapeople.org/python-llvmlite/python-llvmlite-0.37.0-1.fc34.src.rpm

Comment 4 Tomas Hrcka 2021-09-13 07:55:59 UTC
(fedscm-admin):  The Pagure repository was created at https://src.fedoraproject.org/rpms/python-llvmlite

Comment 5 Fedora Update System 2021-09-13 08:39:09 UTC
FEDORA-2021-4ac55b1103 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2021-4ac55b1103

Comment 6 Fedora Update System 2021-09-13 08:39:10 UTC
FEDORA-2021-7ed19c890e has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-7ed19c890e

Comment 7 Fedora Update System 2021-09-13 14:29:18 UTC
FEDORA-2021-7ed19c890e has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf install --enablerepo=updates-testing --advisory=FEDORA-2021-7ed19c890e \*`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-7ed19c890e

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 8 Fedora Update System 2021-09-13 16:15:25 UTC
FEDORA-2021-4ac55b1103 has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf install --enablerepo=updates-testing --advisory=FEDORA-2021-4ac55b1103 \*`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-4ac55b1103

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 9 Fedora Update System 2021-09-21 15:32:06 UTC
FEDORA-2021-7ed19c890e has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 10 Fedora Update System 2021-09-24 20:22:15 UTC
FEDORA-2021-4ac55b1103 has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.