Bug 2000598 - Review Request: python-pathos - Parallel graph management and execution in heterogeneous computing
Summary: Review Request: python-pathos - Parallel graph management and execution in he...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: Package Review
Version: rawhide
Hardware: All
OS: Linux
medium
medium
Target Milestone: ---
Assignee: Zbigniew Jędrzejewski-Szmek
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: fedora-neuro, NeuroFedora 1914344
TreeView+ depends on / blocked
 
Reported: 2021-09-02 13:16 UTC by Ankur Sinha (FranciscoD)
Modified: 2021-09-07 19:07 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-09-07 19:07:34 UTC
Type: ---
Embargoed:
zbyszek: fedora-review+


Attachments (Terms of Use)

Description Ankur Sinha (FranciscoD) 2021-09-02 13:16:06 UTC
Spec URL: https://ankursinha.fedorapeople.org/python-pathos/python-pathos.spec
SRPM URL: https://ankursinha.fedorapeople.org/python-pathos/python-pathos-0.2.8-1.fc34.src.rpm

Description:
The pathos package provides a few basic tools to make parallel and distributed
computing more accessible to the end user. The goal of pathos is to enable the
user to extend their own code to parallel and distributed computing with
minimal refactoring.

pathos provides methods for configuring, launching, monitoring, and controlling
a service on a remote host. One of the most basic features of pathos is the
ability to configure and launch a RPC-based service on a remote host. pathos
seeds the remote host with the portpicker script, which allows the remote host
to inform the localhost of a port that is available for communication.

Beyond the ability to establish a RPC service, and then post requests, is the
ability to launch code in parallel. Unlike parallel computing performed at the
node level (typically with MPI), pathos enables the user to launch jobs in
parallel across heterogeneous distributed resources. pathos provides
distributed map and pipe algorithms, where a mix of local processors and
distributed workers can be selected. pathos also provides a very basic
automated load balancing service, as well as the ability for the user to
directly select the resources.

The high-level pool.map interface, yields a map implementation that hides the
RPC internals from the user. With pool.map, the user can launch their code in
parallel, and as a distributed service, using standard python and without
writing a line of server or parallel batch code.

RPC servers and communication in general is known to be insecure. However,
instead of attempting to make the RPC communication itself secure, pathos
provides the ability to automatically wrap any distributes service or
communication in a ssh-tunnel. Ssh is a universally trusted method. Using
ssh-tunnels, pathos has launched several distributed calculations on national
lab clusters, and to date has performed test calculations that utilize
node-to-node communication between several national lab clusters and a users
laptop. pathos allows the user to configure and launch at a very atomistic
level, through raw access to ssh and scp.

pathos is the core of a python framework for heterogeneous computing. pathos is
in active development, so any user feedback, bug reports, comments, or
suggestions are highly appreciated. A list of issues is located at
https://github.com/uqfoundation/pathos/issues, with a legacy list maintained at
https://uqfoundation.github.io/project/pathos/query.

Fedora Account System Username: ankursinha

Comment 1 Zbigniew Jędrzejewski-Szmek 2021-09-03 09:36:48 UTC
typos: "distributes service" → distributed service, "users laptop" → user's laptop, "atomistic" → atomic

+ package name is OK
+ license is acceptable for Fedora (BSD)
+ license is specified correctly
+ latest version
+ builds and installs without issue
+ fedora-review finds no problems
+ rpmlint only has false positive spelling suggestions
+ BR/R/P look OK
+ %check is present and passes

Package is APPROVED.

Comment 2 Ankur Sinha (FranciscoD) 2021-09-03 09:42:12 UTC
Thanks very much Zbigniew. Requesting SCM now. I'll correct the typos before import.

Comment 3 Miro Hrončok 2021-09-03 10:00:14 UTC
> export PYTHONPATH="$PYTHONPATH:%{python3_sitearch}:%{python3_sitelib}"

Just curious: Why is this used in %check? Should it include %{buidlroot}%{python3_sitearch/lib} instead?

Comment 4 Miro Hrončok 2021-09-03 10:02:23 UTC
Nitpicks:

- rm -rf %{pypi_name}.egg-info is not needed with pyproject RPM macros.
- %{python3} is prefered over %{__python3}

Comment 5 Ankur Sinha (FranciscoD) 2021-09-03 10:47:13 UTC
(In reply to Miro Hrončok from comment #3)
> > export PYTHONPATH="$PYTHONPATH:%{python3_sitearch}:%{python3_sitelib}"
> 
> Just curious: Why is this used in %check? Should it include
> %{buidlroot}%{python3_sitearch/lib} instead?

Aha, I've missed the %{buildroot} bit there. Correcting that now. `export PYTHONPATH="."` also works since it finds the built copy. So does `export PYTHONPATH=":"`---does a colon at the start ":" mean cwd? Maybe that's why my wrong command worked? 

Anyway, I'm using the full buildroot/ path now so it finds the installed copy.

(In reply to Miro Hrončok from comment #4)
> Nitpicks:
> 
> - rm -rf %{pypi_name}.egg-info is not needed with pyproject RPM macros.
> - %{python3} is prefered over %{__python3}

Thanks, corrected.

Updated spec/srpm:

Spec URL: https://ankursinha.fedorapeople.org/python-pathos/python-pathos.spec
SRPM URL: https://ankursinha.fedorapeople.org/python-pathos/python-pathos-0.2.8-1.fc34.src.rpm

Comment 6 Tomas Hrcka 2021-09-03 10:54:00 UTC
(fedscm-admin):  The Pagure repository was created at https://src.fedoraproject.org/rpms/python-pathos

Comment 7 Miro Hrončok 2021-09-03 11:05:14 UTC
> does a colon at the start ":" mean cwd?

It seems so.

Comment 8 Fedora Update System 2021-09-03 13:54:34 UTC
FEDORA-2021-040d0bf6ef has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2021-040d0bf6ef

Comment 9 Fedora Update System 2021-09-03 15:25:44 UTC
FEDORA-2021-040d0bf6ef has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf install --enablerepo=updates-testing --advisory=FEDORA-2021-040d0bf6ef \*`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-040d0bf6ef

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 10 Fedora Update System 2021-09-07 19:07:34 UTC
FEDORA-2021-040d0bf6ef has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.