Bug 2280116 - Review Request: sherlock-project - Simplify OSINT with username searches en masse
Summary: Review Request: sherlock-project - Simplify OSINT with username searches en m...
Keywords:
Status: ASSIGNED
Alias: None
Product: Fedora
Classification: Fedora
Component: Package Review
Version: rawhide
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Neil Hanlon
QA Contact: Fedora Extras Quality Assurance
URL: https://github.com/sherlock-project/s...
Whiteboard:
Depends On:
Blocks: FE-NEEDSPONSOR
TreeView+ depends on / blocked
 
Reported: 2024-05-12 03:44 UTC by Paul Pfeister
Modified: 2024-05-24 04:54 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: ---
Embargoed:
neil: fedora-review?


Attachments (Terms of Use)
The .spec file difference from Copr build 7442563 to 7449613 (2.21 KB, patch)
2024-05-16 11:15 UTC, Fedora Review Service
no flags Details | Diff
The .spec file difference from Copr build 7449613 to 7473723 (1.25 KB, patch)
2024-05-21 21:58 UTC, Fedora Review Service
no flags Details | Diff
The .spec file difference from Copr build 7473723 to 7474944 (203 bytes, patch)
2024-05-22 01:39 UTC, Fedora Review Service
no flags Details | Diff

Description Paul Pfeister 2024-05-12 03:44:29 UTC
Spec URL: https://raw.githubusercontent.com/sherlock-project/sherlock/master/sherlock.spec
SRPM URL: https://download.copr.fedorainfracloud.org/results/ppfeister/sherlock/fedora-rawhide-x86_64/07436290-python-sherlock/python-sherlock-0.14.4-1.fc41.src.rpm
Description: This popular OSINT tool is used to search websites en masse for known usernames, helpful during the scoping and reconnaissance phase of an engagement.
Fedora Account System Username: ppfeister

Comment 1 Paul Pfeister 2024-05-12 08:35:14 UTC
Spec URL: https://raw.githubusercontent.com/sherlock-project/sherlock/master/python-sherlock.spec
SRPM URL: https://download.copr.fedorainfracloud.org/results/ppfeister/sherlock/fedora-rawhide-x86_64/07436481-python-sherlock/python-sherlock-0.14.4-1.fc41.src.rpm

Turns out there's an issue where the spec isn't included in the sprm when pulling spec from scm on a copr build. Works fine when using any other method though, so I adjusted for now. Updated spec and srpm with above links.

Note that this package has two depends (also requesting review). I have my doubts whether or not the fedora-review bot here will realize that or not. When built with those (interim) copr packages available (ppfeister/exrex, ppfeister/torrequest), the following fedora-review is generated.
https://download.copr.fedorainfracloud.org/results/ppfeister/sherlock/fedora-rawhide-x86_64/07436481-python-sherlock/fedora-review/review.txt
By my own review, package appears to meet all (applicable) MUSTs and SHOULDs. Open to critique.

Comment 2 Fedora Review Service 2024-05-12 20:11:39 UTC
Copr build:
https://copr.fedorainfracloud.org/coprs/build/7436951
(failed)

Build log:
https://download.copr.fedorainfracloud.org/results/@fedora-review/fedora-review-2280116-python-sherlock/srpm-builds/07436951/builder-live.log.gz

Please make sure the package builds successfully at least for Fedora Rawhide.

- If the build failed for unrelated reasons (e.g. temporary network
  unavailability), please ignore it.
- If the build failed because of missing BuildRequires, please make sure they
  are listed in the "Depends On" field


---
This comment was created by the fedora-review-service
https://github.com/FrostyX/fedora-review-service

If you want to trigger a new Copr build, add a comment containing new
Spec and SRPM URLs or [fedora-review-service-build] string.

Comment 3 Fedora Review Service 2024-05-12 20:13:14 UTC
Copr build:
https://copr.fedorainfracloud.org/coprs/build/7436953
(failed)

Build log:
https://download.copr.fedorainfracloud.org/results/@fedora-review/fedora-review-2280116-python-sherlock/fedora-rawhide-x86_64/07436953-python-sherlock/builder-live.log.gz

Please make sure the package builds successfully at least for Fedora Rawhide.

- If the build failed for unrelated reasons (e.g. temporary network
  unavailability), please ignore it.
- If the build failed because of missing BuildRequires, please make sure they
  are listed in the "Depends On" field


---
This comment was created by the fedora-review-service
https://github.com/FrostyX/fedora-review-service

If you want to trigger a new Copr build, add a comment containing new
Spec and SRPM URLs or [fedora-review-service-build] string.

Comment 4 Paul Pfeister 2024-05-12 20:20:06 UTC
Build failures due to missing dependencies, which are linked here as Blockers. Fed-Rev not reliable on this request. Reference comment 1 for more info, and for the ~proper~ fedora-review output.

Comment 5 Paul Pfeister 2024-05-14 07:48:21 UTC
Spec \ URL: https://raw.githubusercontent.com/sherlock-project/sherlock/master/sherlock-project.spec
SRPM \ URL: https://download.copr.fedorainfracloud.org/results/ppfeister/sherlock/fedora-rawhide-x86_64/07442546-sherlock-project/sherlock-project-0.14.4-1.fc41.src.rpm

Rewrote the spec with lessons learned from #177841 (python-exrex)
Renamed due to the fact that the python- prefix is only required for libraries.
Added manpage to satisfy rpmlint.
Removed shebangs in sitelib.
etc
All in all much cleaner.

Should meet the requirements for Fed-Rev plus a better rpmlint.

Since the build here will fail due to dependencies the bot doesn't realize (but are linked here as Depends), I've put a slash in the URLs to maybe avoid that unneeded (known-answer) check. The actual Fedora-Review with rpmlint, when ran with the paired depend, can be found here: https://download.copr.fedorainfracloud.org/results/ppfeister/sherlock/fedora-rawhide-x86_64/07442546-sherlock-project/fedora-review/review.txt

Comment 6 Fedora Review Service 2024-05-14 07:50:47 UTC
Copr build:
https://copr.fedorainfracloud.org/coprs/build/7442563
(failed)

Build log:
https://download.copr.fedorainfracloud.org/results/@fedora-review/fedora-review-2280116-sherlock-project/fedora-rawhide-x86_64/07442563-sherlock-project/builder-live.log.gz

Please make sure the package builds successfully at least for Fedora Rawhide.

- If the build failed for unrelated reasons (e.g. temporary network
  unavailability), please ignore it.
- If the build failed because of missing BuildRequires, please make sure they
  are listed in the "Depends On" field


---
This comment was created by the fedora-review-service
https://github.com/FrostyX/fedora-review-service

If you want to trigger a new Copr build, add a comment containing new
Spec and SRPM URLs or [fedora-review-service-build] string.

Comment 7 Paul Pfeister 2024-05-15 05:37:39 UTC
Pause review. Encountered some weird issues that I'd rather fix before pushing anything.

Will reply with new link shortly.

Comment 8 Paul Pfeister 2024-05-16 08:10:28 UTC
https://raw.githubusercontent.com/ppfeister/pkg/master/sherlock/sherlock-project.spec
https://download.copr.fedorainfracloud.org/results/ppfeister/sherlock/fedora-rawhide-x86_64/07448351-sherlock-project/sherlock-project-0.14.4-1.fc41.src.rpm

Completely rebuilt upstream packaging to address some concerns with installation. We're good to go.
NOTE: Fedora Review bot WILL fail if it runs. sherlock-project depends on python-exrex from request #2280062 (linked here as a dependency). When built with access to copr ppfeister/sherlock (which includes exrex in the interim), the following is the review provided:
https://download.copr.fedorainfracloud.org/results/ppfeister/sherlock/fedora-rawhide-x86_64/07448351-sherlock-project/fedora-review/review.txt
And rpmlint:
https://download.copr.fedorainfracloud.org/results/ppfeister/sherlock/fedora-rawhide-x86_64/07448351-sherlock-project/fedora-review/rpmlint.txt

Comment 9 Fedora Review Service 2024-05-16 11:15:55 UTC
Created attachment 2033543 [details]
The .spec file difference from Copr build 7442563 to 7449613

Comment 10 Fedora Review Service 2024-05-16 11:15:56 UTC
Copr build:
https://copr.fedorainfracloud.org/coprs/build/7449613
(failed)

Build log:
https://download.copr.fedorainfracloud.org/results/@fedora-review/fedora-review-2280116-sherlock-project/fedora-rawhide-x86_64/07449613-sherlock-project/builder-live.log.gz

Please make sure the package builds successfully at least for Fedora Rawhide.

- If the build failed for unrelated reasons (e.g. temporary network
  unavailability), please ignore it.
- If the build failed because of missing BuildRequires, please make sure they
  are listed in the "Depends On" field


---
This comment was created by the fedora-review-service
https://github.com/FrostyX/fedora-review-service

If you want to trigger a new Copr build, add a comment containing new
Spec and SRPM URLs or [fedora-review-service-build] string.

Comment 11 Paul Pfeister 2024-05-21 21:56:45 UTC
Spec URL: https://raw.githubusercontent.com/ppfeister/pkg/master/sherlock/sherlock-project.spec
SRPM URL: https://download.copr.fedorainfracloud.org/results/ppfeister/sherlock/fedora-rawhide-x86_64/07473720-sherlock-project/sherlock-project-0.14.4-1.fc41.src.rpm

python-exrex has been removed as a required dependency and sherlock-project is NOT blocked anymore. I still intend to package it for later projects, but these two requests for review are now decoupled.

Comment 12 Fedora Review Service 2024-05-21 21:58:56 UTC
Created attachment 2034513 [details]
The .spec file difference from Copr build 7449613 to 7473723

Comment 13 Fedora Review Service 2024-05-21 21:58:58 UTC
Copr build:
https://copr.fedorainfracloud.org/coprs/build/7473723
(failed)

Build log:
https://download.copr.fedorainfracloud.org/results/@fedora-review/fedora-review-2280116-sherlock-project/fedora-rawhide-x86_64/07473723-sherlock-project/builder-live.log.gz

Please make sure the package builds successfully at least for Fedora Rawhide.

- If the build failed for unrelated reasons (e.g. temporary network
  unavailability), please ignore it.
- If the build failed because of missing BuildRequires, please make sure they
  are listed in the "Depends On" field


---
This comment was created by the fedora-review-service
https://github.com/FrostyX/fedora-review-service

If you want to trigger a new Copr build, add a comment containing new
Spec and SRPM URLs or [fedora-review-service-build] string.

Comment 14 Paul Pfeister 2024-05-21 22:03:28 UTC
Build itself actually succeeds, unit tests just fail when built without internet access (as done by the review service bot). This is an internet-reliant tool and the unit tests validate internet-reaching probes.

tox succeeds in mock or on the copr when internet access is available.

Fedora Review Service output from copr build: https://download.copr.fedorainfracloud.org/results/ppfeister/sherlock/fedora-rawhide-x86_64/07473720-sherlock-project/fedora-review/review.txt

Comment 15 Ben Beasley 2024-05-21 22:11:27 UTC
(In reply to Paul Pfeister from comment #14)
> Build itself actually succeeds, unit tests just fail when built without
> internet access (as done by the review service bot). This is an
> internet-reliant tool and the unit tests validate internet-reaching probes.
> 
> tox succeeds in mock or on the copr when internet access is available.

Real Fedora builds are always done offline. You’ll need to selectively disable all tests that require network access. If you want to be able to run network tests manually in mock from time to time, you can gate this behind a build conditional, e.g.

  %bcond network_tests 0
  
  %if %{without network_tests}
  ...
  %endif
  
  %if %{with network_tests}
  ...
  %endif

If *all* of the tests really require network access and you have to disable tests entirely, you should at least run an import-only “smoke test” with %pyproject_check_import: https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_running_tests

Comment 16 Paul Pfeister 2024-05-21 22:19:58 UTC
Interesting. I hadn't accounted for that. A lot of the tests are network dependent, but a handful of them are still valid offline (and retaining them would be preferred, obviously)

I'll do some research and see what the best way to do this with tox would be. Worst case scenario I can just call pytest directly and specify what conditions/env to use. Will post an update here when adapted.

Comment 17 Paul Pfeister 2024-05-22 01:31:24 UTC
Spec URL: https://raw.githubusercontent.com/ppfeister/pkg/master/sherlock/sherlock-project.spec
SRPM URL: https://download.copr.fedorainfracloud.org/results/ppfeister/sherlock/fedora-rawhide-x86_64/07474940-sherlock-project/sherlock-project-0.14.4-1.fc41.src.rpm

Easy enough. Added a tox 'offline' env that calls pytest with online marks excluded. Lets the other tests run as expected.

Comment 18 Fedora Review Service 2024-05-22 01:39:17 UTC
Created attachment 2034524 [details]
The .spec file difference from Copr build 7473723 to 7474944

Comment 19 Fedora Review Service 2024-05-22 01:39:19 UTC
Copr build:
https://copr.fedorainfracloud.org/coprs/build/7474944
(succeeded)

Review template:
https://download.copr.fedorainfracloud.org/results/@fedora-review/fedora-review-2280116-sherlock-project/fedora-rawhide-x86_64/07474944-sherlock-project/fedora-review/review.txt

Please take a look if any issues were found.


---
This comment was created by the fedora-review-service
https://github.com/FrostyX/fedora-review-service

If you want to trigger a new Copr build, add a comment containing new
Spec and SRPM URLs or [fedora-review-service-build] string.

Comment 20 Paul Pfeister 2024-05-24 04:48:24 UTC
SRPM Permalink: https://github.com/ppfeister/pkg/raw/master/sherlock/sherlock-project-0.14.4-1.fc41.src.rpm
(copr build was removed)

Comment 21 Fedora Review Service 2024-05-24 04:54:59 UTC
Copr build:
https://copr.fedorainfracloud.org/coprs/build/7483756
(succeeded)

Review template:
https://download.copr.fedorainfracloud.org/results/@fedora-review/fedora-review-2280116-sherlock-project/fedora-rawhide-x86_64/07483756-sherlock-project/fedora-review/review.txt

Please take a look if any issues were found.


---
This comment was created by the fedora-review-service
https://github.com/FrostyX/fedora-review-service

If you want to trigger a new Copr build, add a comment containing new
Spec and SRPM URLs or [fedora-review-service-build] string.


Note You need to log in before you can comment on or make changes to this bug.