Bug 2280116 (sherlock-project) - Review Request: sherlock-project - Simplify OSINT with username searches en masse
Summary: Review Request: sherlock-project - Simplify OSINT with username searches en m...
Keywords:
Status: CLOSED ERRATA
Alias: sherlock-project
Product: Fedora
Classification: Fedora
Component: Package Review
Version: rawhide
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Neil Hanlon
QA Contact: Fedora Extras Quality Assurance
URL: https://github.com/sherlock-project/s...
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2024-05-12 03:44 UTC by Paul Pfeister
Modified: 2024-08-18 03:55 UTC (History)
3 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2024-07-09 20:46:17 UTC
Type: ---
Embargoed:
neil: fedora-review+


Attachments (Terms of Use)
The .spec file difference from Copr build 7442563 to 7449613 (2.21 KB, patch)
2024-05-16 11:15 UTC, Fedora Review Service
no flags Details | Diff
The .spec file difference from Copr build 7449613 to 7473723 (1.25 KB, patch)
2024-05-21 21:58 UTC, Fedora Review Service
no flags Details | Diff
The .spec file difference from Copr build 7473723 to 7474944 (203 bytes, patch)
2024-05-22 01:39 UTC, Fedora Review Service
no flags Details | Diff

Description Paul Pfeister 2024-05-12 03:44:29 UTC
Spec URL: https://raw.githubusercontent.com/sherlock-project/sherlock/master/sherlock.spec
SRPM URL: https://download.copr.fedorainfracloud.org/results/ppfeister/sherlock/fedora-rawhide-x86_64/07436290-python-sherlock/python-sherlock-0.14.4-1.fc41.src.rpm
Description: This popular OSINT tool is used to search websites en masse for known usernames, helpful during the scoping and reconnaissance phase of an engagement.
Fedora Account System Username: ppfeister

Comment 1 Paul Pfeister 2024-05-12 08:35:14 UTC
Spec URL: https://raw.githubusercontent.com/sherlock-project/sherlock/master/python-sherlock.spec
SRPM URL: https://download.copr.fedorainfracloud.org/results/ppfeister/sherlock/fedora-rawhide-x86_64/07436481-python-sherlock/python-sherlock-0.14.4-1.fc41.src.rpm

Turns out there's an issue where the spec isn't included in the sprm when pulling spec from scm on a copr build. Works fine when using any other method though, so I adjusted for now. Updated spec and srpm with above links.

Note that this package has two depends (also requesting review). I have my doubts whether or not the fedora-review bot here will realize that or not. When built with those (interim) copr packages available (ppfeister/exrex, ppfeister/torrequest), the following fedora-review is generated.
https://download.copr.fedorainfracloud.org/results/ppfeister/sherlock/fedora-rawhide-x86_64/07436481-python-sherlock/fedora-review/review.txt
By my own review, package appears to meet all (applicable) MUSTs and SHOULDs. Open to critique.

Comment 2 Fedora Review Service 2024-05-12 20:11:39 UTC
Copr build:
https://copr.fedorainfracloud.org/coprs/build/7436951
(failed)

Build log:
https://download.copr.fedorainfracloud.org/results/@fedora-review/fedora-review-2280116-python-sherlock/srpm-builds/07436951/builder-live.log.gz

Please make sure the package builds successfully at least for Fedora Rawhide.

- If the build failed for unrelated reasons (e.g. temporary network
  unavailability), please ignore it.
- If the build failed because of missing BuildRequires, please make sure they
  are listed in the "Depends On" field


---
This comment was created by the fedora-review-service
https://github.com/FrostyX/fedora-review-service

If you want to trigger a new Copr build, add a comment containing new
Spec and SRPM URLs or [fedora-review-service-build] string.

Comment 3 Fedora Review Service 2024-05-12 20:13:14 UTC
Copr build:
https://copr.fedorainfracloud.org/coprs/build/7436953
(failed)

Build log:
https://download.copr.fedorainfracloud.org/results/@fedora-review/fedora-review-2280116-python-sherlock/fedora-rawhide-x86_64/07436953-python-sherlock/builder-live.log.gz

Please make sure the package builds successfully at least for Fedora Rawhide.

- If the build failed for unrelated reasons (e.g. temporary network
  unavailability), please ignore it.
- If the build failed because of missing BuildRequires, please make sure they
  are listed in the "Depends On" field


---
This comment was created by the fedora-review-service
https://github.com/FrostyX/fedora-review-service

If you want to trigger a new Copr build, add a comment containing new
Spec and SRPM URLs or [fedora-review-service-build] string.

Comment 4 Paul Pfeister 2024-05-12 20:20:06 UTC
Build failures due to missing dependencies, which are linked here as Blockers. Fed-Rev not reliable on this request. Reference comment 1 for more info, and for the ~proper~ fedora-review output.

Comment 5 Paul Pfeister 2024-05-14 07:48:21 UTC
Spec \ URL: https://raw.githubusercontent.com/sherlock-project/sherlock/master/sherlock-project.spec
SRPM \ URL: https://download.copr.fedorainfracloud.org/results/ppfeister/sherlock/fedora-rawhide-x86_64/07442546-sherlock-project/sherlock-project-0.14.4-1.fc41.src.rpm

Rewrote the spec with lessons learned from #177841 (python-exrex)
Renamed due to the fact that the python- prefix is only required for libraries.
Added manpage to satisfy rpmlint.
Removed shebangs in sitelib.
etc
All in all much cleaner.

Should meet the requirements for Fed-Rev plus a better rpmlint.

Since the build here will fail due to dependencies the bot doesn't realize (but are linked here as Depends), I've put a slash in the URLs to maybe avoid that unneeded (known-answer) check. The actual Fedora-Review with rpmlint, when ran with the paired depend, can be found here: https://download.copr.fedorainfracloud.org/results/ppfeister/sherlock/fedora-rawhide-x86_64/07442546-sherlock-project/fedora-review/review.txt

Comment 6 Fedora Review Service 2024-05-14 07:50:47 UTC
Copr build:
https://copr.fedorainfracloud.org/coprs/build/7442563
(failed)

Build log:
https://download.copr.fedorainfracloud.org/results/@fedora-review/fedora-review-2280116-sherlock-project/fedora-rawhide-x86_64/07442563-sherlock-project/builder-live.log.gz

Please make sure the package builds successfully at least for Fedora Rawhide.

- If the build failed for unrelated reasons (e.g. temporary network
  unavailability), please ignore it.
- If the build failed because of missing BuildRequires, please make sure they
  are listed in the "Depends On" field


---
This comment was created by the fedora-review-service
https://github.com/FrostyX/fedora-review-service

If you want to trigger a new Copr build, add a comment containing new
Spec and SRPM URLs or [fedora-review-service-build] string.

Comment 7 Paul Pfeister 2024-05-15 05:37:39 UTC
Pause review. Encountered some weird issues that I'd rather fix before pushing anything.

Will reply with new link shortly.

Comment 8 Paul Pfeister 2024-05-16 08:10:28 UTC
https://raw.githubusercontent.com/ppfeister/pkg/master/sherlock/sherlock-project.spec
https://download.copr.fedorainfracloud.org/results/ppfeister/sherlock/fedora-rawhide-x86_64/07448351-sherlock-project/sherlock-project-0.14.4-1.fc41.src.rpm

Completely rebuilt upstream packaging to address some concerns with installation. We're good to go.
NOTE: Fedora Review bot WILL fail if it runs. sherlock-project depends on python-exrex from request #2280062 (linked here as a dependency). When built with access to copr ppfeister/sherlock (which includes exrex in the interim), the following is the review provided:
https://download.copr.fedorainfracloud.org/results/ppfeister/sherlock/fedora-rawhide-x86_64/07448351-sherlock-project/fedora-review/review.txt
And rpmlint:
https://download.copr.fedorainfracloud.org/results/ppfeister/sherlock/fedora-rawhide-x86_64/07448351-sherlock-project/fedora-review/rpmlint.txt

Comment 9 Fedora Review Service 2024-05-16 11:15:55 UTC
Created attachment 2033543 [details]
The .spec file difference from Copr build 7442563 to 7449613

Comment 10 Fedora Review Service 2024-05-16 11:15:56 UTC
Copr build:
https://copr.fedorainfracloud.org/coprs/build/7449613
(failed)

Build log:
https://download.copr.fedorainfracloud.org/results/@fedora-review/fedora-review-2280116-sherlock-project/fedora-rawhide-x86_64/07449613-sherlock-project/builder-live.log.gz

Please make sure the package builds successfully at least for Fedora Rawhide.

- If the build failed for unrelated reasons (e.g. temporary network
  unavailability), please ignore it.
- If the build failed because of missing BuildRequires, please make sure they
  are listed in the "Depends On" field


---
This comment was created by the fedora-review-service
https://github.com/FrostyX/fedora-review-service

If you want to trigger a new Copr build, add a comment containing new
Spec and SRPM URLs or [fedora-review-service-build] string.

Comment 11 Paul Pfeister 2024-05-21 21:56:45 UTC
Spec URL: https://raw.githubusercontent.com/ppfeister/pkg/master/sherlock/sherlock-project.spec
SRPM URL: https://download.copr.fedorainfracloud.org/results/ppfeister/sherlock/fedora-rawhide-x86_64/07473720-sherlock-project/sherlock-project-0.14.4-1.fc41.src.rpm

python-exrex has been removed as a required dependency and sherlock-project is NOT blocked anymore. I still intend to package it for later projects, but these two requests for review are now decoupled.

Comment 12 Fedora Review Service 2024-05-21 21:58:56 UTC
Created attachment 2034513 [details]
The .spec file difference from Copr build 7449613 to 7473723

Comment 13 Fedora Review Service 2024-05-21 21:58:58 UTC
Copr build:
https://copr.fedorainfracloud.org/coprs/build/7473723
(failed)

Build log:
https://download.copr.fedorainfracloud.org/results/@fedora-review/fedora-review-2280116-sherlock-project/fedora-rawhide-x86_64/07473723-sherlock-project/builder-live.log.gz

Please make sure the package builds successfully at least for Fedora Rawhide.

- If the build failed for unrelated reasons (e.g. temporary network
  unavailability), please ignore it.
- If the build failed because of missing BuildRequires, please make sure they
  are listed in the "Depends On" field


---
This comment was created by the fedora-review-service
https://github.com/FrostyX/fedora-review-service

If you want to trigger a new Copr build, add a comment containing new
Spec and SRPM URLs or [fedora-review-service-build] string.

Comment 14 Paul Pfeister 2024-05-21 22:03:28 UTC
Build itself actually succeeds, unit tests just fail when built without internet access (as done by the review service bot). This is an internet-reliant tool and the unit tests validate internet-reaching probes.

tox succeeds in mock or on the copr when internet access is available.

Fedora Review Service output from copr build: https://download.copr.fedorainfracloud.org/results/ppfeister/sherlock/fedora-rawhide-x86_64/07473720-sherlock-project/fedora-review/review.txt

Comment 15 Ben Beasley 2024-05-21 22:11:27 UTC
(In reply to Paul Pfeister from comment #14)
> Build itself actually succeeds, unit tests just fail when built without
> internet access (as done by the review service bot). This is an
> internet-reliant tool and the unit tests validate internet-reaching probes.
> 
> tox succeeds in mock or on the copr when internet access is available.

Real Fedora builds are always done offline. You’ll need to selectively disable all tests that require network access. If you want to be able to run network tests manually in mock from time to time, you can gate this behind a build conditional, e.g.

  %bcond network_tests 0
  
  %if %{without network_tests}
  ...
  %endif
  
  %if %{with network_tests}
  ...
  %endif

If *all* of the tests really require network access and you have to disable tests entirely, you should at least run an import-only “smoke test” with %pyproject_check_import: https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_running_tests

Comment 16 Paul Pfeister 2024-05-21 22:19:58 UTC
Interesting. I hadn't accounted for that. A lot of the tests are network dependent, but a handful of them are still valid offline (and retaining them would be preferred, obviously)

I'll do some research and see what the best way to do this with tox would be. Worst case scenario I can just call pytest directly and specify what conditions/env to use. Will post an update here when adapted.

Comment 17 Paul Pfeister 2024-05-22 01:31:24 UTC
Spec URL: https://raw.githubusercontent.com/ppfeister/pkg/master/sherlock/sherlock-project.spec
SRPM URL: https://download.copr.fedorainfracloud.org/results/ppfeister/sherlock/fedora-rawhide-x86_64/07474940-sherlock-project/sherlock-project-0.14.4-1.fc41.src.rpm

Easy enough. Added a tox 'offline' env that calls pytest with online marks excluded. Lets the other tests run as expected.

Comment 18 Fedora Review Service 2024-05-22 01:39:17 UTC
Created attachment 2034524 [details]
The .spec file difference from Copr build 7473723 to 7474944

Comment 19 Fedora Review Service 2024-05-22 01:39:19 UTC
Copr build:
https://copr.fedorainfracloud.org/coprs/build/7474944
(succeeded)

Review template:
https://download.copr.fedorainfracloud.org/results/@fedora-review/fedora-review-2280116-sherlock-project/fedora-rawhide-x86_64/07474944-sherlock-project/fedora-review/review.txt

Please take a look if any issues were found.


---
This comment was created by the fedora-review-service
https://github.com/FrostyX/fedora-review-service

If you want to trigger a new Copr build, add a comment containing new
Spec and SRPM URLs or [fedora-review-service-build] string.

Comment 20 Paul Pfeister 2024-05-24 04:48:24 UTC
SRPM Permalink: https://github.com/ppfeister/pkg/raw/master/sherlock/sherlock-project-0.14.4-1.fc41.src.rpm
(copr build was removed)

Comment 21 Fedora Review Service 2024-05-24 04:54:59 UTC
Copr build:
https://copr.fedorainfracloud.org/coprs/build/7483756
(succeeded)

Review template:
https://download.copr.fedorainfracloud.org/results/@fedora-review/fedora-review-2280116-sherlock-project/fedora-rawhide-x86_64/07483756-sherlock-project/fedora-review/review.txt

Please take a look if any issues were found.


---
This comment was created by the fedora-review-service
https://github.com/FrostyX/fedora-review-service

If you want to trigger a new Copr build, add a comment containing new
Spec and SRPM URLs or [fedora-review-service-build] string.

Comment 22 Paul Pfeister 2024-06-25 04:11:11 UTC
Note regarding upstream --- we're changing the importable module name to have finally match the package name or similar. This will avoid future possible (but frankly unlikely) conflict with the unrelated 'sherlock' library on PyPI. This possible conflict was raised during discussions on the debian packaging side.

Probably won't trigger the review service for that type of change, but it was worth noting.

Comment 23 Neil Hanlon 2024-06-25 04:17:00 UTC
Notes:

* The URL in the spec does not build. However, it appears that this was recently merged into upstream; building against the latest commit for this github repo results in a successful build.
* No issues with review; once building against upstream


Please update the spec to build off of the latest branch [1]. With that, this package is **approved**, though I might recommend you wait until the library name is changed--up to you :) 

[1] https://docs.fedoraproject.org/en-US/packaging-guidelines/SourceURL/

Comment 24 Fedora Review Service 2024-06-25 04:17:37 UTC
Hello @ppfeister,
since this is your first Fedora package, you need to get sponsored by a package
sponsor before it can be accepted.

A sponsor is an experienced package maintainer who will guide you through
the processes that you will follow and the tools that you will use as a future
maintainer. A sponsor will also be there to answer your questions related to
packaging.

You can find all active sponsors here:
https://docs.pagure.org/fedora-sponsors/

I created a sponsorship request for you:
https://pagure.io/packager-sponsors/issue/665
Please take a look and make sure the information is correct.

Thank you, and best of luck on your packaging journey.

---
This comment was created by the fedora-review-service
https://github.com/FrostyX/fedora-review-service

Comment 25 Paul Pfeister 2024-06-25 04:34:00 UTC
Thank you Neil.

As time went on, that branch eventually got deleted it seems. Probably after merging upstream. Anyways, the SourceURL is now pointing to the upstream as it would normally. Should build fine, but just for the sake of testing, I'll trigger the service real quick...

[fedora-review-service-build]

Comment 26 Paul Pfeister 2024-06-25 04:38:24 UTC
Nevermind, I forgot the service doesn't run when approved. I'll run locally when I have a minute.

Cheers!

Comment 27 Fedora Admin user for bugzilla script actions 2024-06-26 05:34:04 UTC
The Pagure repository was created at https://src.fedoraproject.org/rpms/sherlock-project


Note You need to log in before you can comment on or make changes to this bug.