Bug 1824848
Summary: | Review Request: jc - JSONifies the output of dozens of commands | ||
---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Kelly Brazil <kellyjonbrazil> |
Component: | Package Review | Assignee: | Artur Frenszek-Iwicki <fedora> |
Status: | CLOSED DUPLICATE | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
Severity: | medium | Docs Contact: | |
Priority: | unspecified | ||
Version: | rawhide | CC: | fedora, mail, package-review, ppisar, pviktori, zebob.m |
Target Milestone: | --- | Flags: | fedora:
fedora-review+
|
Target Release: | --- | ||
Hardware: | All | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2020-06-05 11:51:10 UTC | Type: | --- |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | 1776922 | ||
Bug Blocks: | 177841 |
Description
Kelly Brazil
2020-04-16 14:39:35 UTC
To submit a package for Fedora, you need to provide a spec. The spec file does not have to written by hand - i.e. many golang-based packages use go2rpm to create a spec - but it does have to conform to Fedora's packaging guidelines (and ensuring that is partly what the review process is for). If you are able to build a RPM, then you should also be able to build a SRPM. Do you just get the RPM from fpm, or have you tried building it from spec with rpmbuild? Ah, I see. I can provide a spec, but I think the challenge will be with a SRPM. Since this is a python3 app, typically users use pip to install it and the source-code is freely available on pypi.org and github. For a python app, can I just create an SRPM that includes the tgz distribution file that I build for pypi? Then I would expect the user to run pip to install it manually. I'm not keen on recreating pip's functionality within a SRPM, so I'm trying to find the best practice here. The examples I have seen for python apps don't seem to fit the modern way of distribution - requiring manual creation of a wrapper script to execute the application. This should be handled by pip. Should I just add `BuildRequires python3 and python3-wheel` to the spec so pip is available and can 'install' the python source? I'm far from a guru when it comes to packaging Python stuff in Fedora, but a lot of the questions you ask have been thought out throughout the years and put in the Python packaging guidelines: https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/ That said, if the package is pure Python (i.e. not a Python module written in C or whatever else), then packaging is quite straightforward. You can take a look at some of the python-SomeThing packages already present in Fedora: https://src.fedoraproject.org/rpms/python-whois/raw/master/f/python-whois.spec https://src.fedoraproject.org/rpms/python-rcssmin/raw/master/f/python-rcssmin.spec This is very helpful, thank you! I'll dig into this. My app requires the following python modules (available via pip): - 'ifconfig-parser>=0.0.5' - 'ruamel.yaml>=0.15.0' - 'xmltodict>=0.12.0' - 'Pygments>=2.5.2' It looks like the following are available: - python3-pygments 2.4.2-7 - python3-ruamel-yaml 0.16.6-2 - python3-ruamel-yaml-clib 0.1.2-2 - python3-xmltodict 0.12.0-7 And python-ifconfig-parser does not exist. I can test to see if I can reduce the pygments requirement to 2.4.2, but will I need to create a python-ifconfig-parser package? Maybe since it's such a small module I should just vendorize it. There is an open RHBZ ticket to have python-pygments updated: https://bugzilla.redhat.com/show_bug.cgi?id=1776922 I have pinged the maintainer and asked them to take a look at it. Thanks - I think I'm good to start building the RPM and SRPM. I have successfully tested with pygments 2.4.2 so I'm ok there. I have decided to vendorize ifconfig-parser since it is a very small package and is easy to integrate and remove the dependency. Here is my source RPM: https://jc-packages.s3-us-west-1.amazonaws.com/fedora/python-jc-1.10.7-1.fc32.src.rpm Here is my RPM: https://jc-packages.s3-us-west-1.amazonaws.com/fedora/python3-jc-1.10.7-1.fc32.noarch.rpm Here is my python-jc.spec # Created by pyp2rpm-3.3.4 %global pypi_name jc Name: python-%{pypi_name} Version: 1.10.7 Release: 1%{?dist} Summary: This tool serializes the output of popular command line tools and filetypes to structured JSON output License: MIT URL: https://github.com/kellyjonbrazil/jc Source0: %{pypi_source} BuildArch: noarch BuildRequires: python3-devel BuildRequires: python3dist(pygments) >= 2.4.2 BuildRequires: python3dist(ruamel.yaml) >= 0.15 BuildRequires: python3dist(setuptools) BuildRequires: python3dist(xmltodict) >= 0.12 %description JSON CLI output utilityjc is used to JSONify the output of many standard linux cli tools and file types for easier parsing in scripts. %package -n python3-%{pypi_name} Summary: %{summary} %{?python_provide:%python_provide python3-%{pypi_name}} Requires: python3dist(pygments) >= 2.4.2 Requires: python3dist(ruamel.yaml) >= 0.15 Requires: python3dist(setuptools) Requires: python3dist(xmltodict) >= 0.12 %description -n python3-%{pypi_name} JSON CLI output utilityjc is used to JSONify the output of many standard linux cli tools and file types for easier parsing in scripts. %prep %autosetup -n %{pypi_name}-%{version} # Remove bundled egg-info rm -rf %{pypi_name}.egg-info %build %py3_build %install %py3_install %check %{__python3} setup.py test %files -n python3-%{pypi_name} %doc README.md %{_bindir}/jc %{python3_sitelib}/%{pypi_name} %{python3_sitelib}/tests %{python3_sitelib}/%{pypi_name}-%{version}-py%{python3_version}.egg-info %changelog * Fri May 01 2020 kbrazil - 1.10.7-1 - Initial package. The binary RPM link is not needed. The spec should be provided as a link, instead of being pasted here. Either way, here's my nitpicking: >Summary: This tool serializes the output of popular command line tools and filetypes to structured JSON output The summary text should be no longer than 80 characters. Maybe "Serialize the output of command line tools and filetypes to structured JSON"? https://docs.fedoraproject.org/en-US/packaging-guidelines/#_summary_and_description >%description >JSON CLI output utilityjc is used to JSONify the output of many standard linux >cli tools and file types for easier parsing in scripts. >... >%description -n python3-%{pypi_name} >JSON CLI output utilityjc is used to JSONify the output of many standard linux >cli tools and file types for easier parsing in scripts. 1. "utilityjc" - missing space 2. "cli tools" - please use "CLI" 3. Consider putting the description into a macro (i.e. %global macro_name macro_body), so you don't have to repeat this text in the spec and just use the macro instead. >%{?python_provide:%python_provide python3-%{pypi_name}} I believe this is not needed; it will get auto-generated. https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_automatic_provides_with_a_standardized_name >Requires: python3dist(pygments) >= 2.4.2 >Requires: python3dist(ruamel.yaml) >= 0.15 >Requires: python3dist(setuptools) >Requires: python3dist(xmltodict) >= 0.12 Again, these are not needed; the automatic Python dependency generator should pick these up for you. https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_automatically_generated_dependencies Thanks for the feedback! It looks like this works: https://github.com/kellyjonbrazil/jc-packaging/blob/master/rpmbuild/SPEC/python-jc.spec $ rpm -q -i -p python3-jc-1.10.7-1.fc32.noarch.rpm Name : python3-jc Version : 1.10.7 Release : 1.fc32 Architecture: noarch Install Date: (not installed) Group : Unspecified Size : 17702993 License : MIT Signature : (none) Source RPM : python-jc-1.10.7-1.fc32.src.rpm Build Date : Sat 02 May 2020 10:42:16 AM PDT Build Host : localhost URL : https://github.com/kellyjonbrazil/jc Summary : Serialize the output of command line tools and filetypes to structured JSON Description : JSON CLI output utility. JC is used to JSONify the output of many standard linux CLI tools and file types for easier parsing in scripts. Just checking if there is anything more I need to do here. Thanks! >https://github.com/kellyjonbrazil/jc-packaging/blob/master/rpmbuild/SPEC/python-jc.spec This points to a syntax-highlighted HTML rendition of the spec. Please use "raw file" links. >%global desc JSON CLI output utility. JC is used to JSONify the output of many standard linux \ >CLI tools and file types for easier parsing in scripts. This macro is later used for the %description and the first line is 81 characters long. Either remove the space at the end ("linux\") so it's exactly 80 characters, or move a word or two to the second line. https://docs.fedoraproject.org/en-US/packaging-guidelines/#_summary_and_description >Name: python-%{pypi_name} jc is a Python application, not a library. As such, it's not necessary to use "python-jc" for the name; it can be just "jc". https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_naming The rest looks good to me. I'll do a proper review in the upcoming days. Thanks for the suggestions. I updated the spec here: https://raw.githubusercontent.com/kellyjonbrazil/jc-packaging/master/rpmbuild/SPEC/python-jc.spec The: %{python3_sitelib}/tests in %files means you're installing the tests system-wide, so `import test` will import tests of jc. If another package would also try to do this, it would conflict. I think you should leave the tests out of the RPM. If you really need them, could you rename them to e.g. `test_jc`? Could you include a LICENSE file? Usually it's done like this: %files -n ... %doc README.md %license LICENSE.md %{_bindir}/jc.md ... And of course, the license should be included in the archive on PyPI. Otherwise it's basically illegal to redistribute the package: if you read the MIT license, you'll find that it basically says you need to distribute the license (specific parts of it) with the software. Thanks for letting me know about that! It looks like a lot of other python packages do the same thing because the python packaging docs (setup_tools, not RPM/Fedora) are not very clear on how to deal with tests and licenses. I have made the suggested updates to the pypi package and updated the spec here: https://raw.githubusercontent.com/kellyjonbrazil/jc-packaging/master/rpmbuild/SPEC/python-jc.spec My only concern is the naming of the rpm. I'm not sure I'm in compliance now that I changed the Name to just 'jc' in the spec, while other parts of the spec point to 'python3-jc'. Notice the difference between the Name field and the Source RPM name field. $ rpm -q -i -p python3-jc-1.10.8-1.fc32.noarch.rpm Name : python3-jc Version : 1.10.8 Release : 1.fc32 Architecture: noarch Install Date: (not installed) Group : Unspecified Size : 672992 License : MIT Signature : (none) Source RPM : jc-1.10.8-1.fc32.src.rpm Build Date : Fri 08 May 2020 09:58:17 AM PDT Build Host : localhost URL : https://github.com/kellyjonbrazil/jc Summary : Serialize the output of command line tools and filetypes to structured JSON Description : JSON CLI output utility. JC is used to JSONify the output of many linux CLI tools and file types for easier parsing in scripts. Quick update - I bumped the version to 1.10.9 which includes the MIT license info for a vendorized package used in jc (ifconfig-parser) Bumped the version to 1.10.10 which fixed a couple bugs relevant to the Fedora platform. Should be the last update. Please upload the latest spec and SRPM somewhere and provide the links, I'll do a proper review then. Here is the latest spec: https://raw.githubusercontent.com/kellyjonbrazil/jc-packaging/master/rpmbuild/SPEC/python-jc.spec Here is the source RPM from the spec: https://jc-packages.s3-us-west-1.amazonaws.com/fedora/jc-1.10.10-1.fc32.src.rpm Package Review ============== Legend: [x] = Pass, [!] = Fail, [-] = Not applicable, [?] = Not evaluated [ ] = Manual review needed Issues: ======= - The spec produces a python binary in a "python3-jc" package, instead of just "jc". "python3-XXX" is only required for python modules. - Spec file name must match the spec package %{name}, in the format %{name}.spec - "python-jc.spec" should be "jc.spec". - Upstream sources contain a test suite, but the spec does not contain a %check section that would execute those tests. - There are extra executable bits on README.md and LICENSE.md. ===== MUST items ===== Generic: [x]: Package is licensed with an open-source compatible license and meets other legal requirements as defined in the legal section of Packaging Guidelines. [x]: License field in the package spec file matches the actual license. [x]: Package contains no bundled libraries without FPC exception. [x]: Changelog in prescribed format. [x]: Sources contain only permissible code or content. [-]: Package contains desktop file if it is a GUI application. [-]: Development files must be in a -devel package [x]: Package uses nothing in %doc for runtime. [x]: Package consistently uses macros (instead of hard-coded directory names). [!]: Package is named according to the Package Naming Guidelines. [x]: Package does not generate any conflict. [x]: Package obeys FHS, except libexecdir and /usr/target. [-]: If the package is a rename of another package, proper Obsoletes and Provides are present. [x]: Requires correct, justified where necessary. [x]: Spec file is legible and written in American English. [-]: Package contains systemd file(s) if in need. [x]: Package is not known to require an ExcludeArch tag. [-]: Large documentation must go in a -doc subpackage. Large could be size (~1MB) or number of files. [!]: Package complies to the Packaging Guidelines [x]: Package successfully compiles and builds into binary rpms on at least one supported primary architecture. [x]: Package installs properly. [x]: Rpmlint is run on all rpms the build produces. Note: There are rpmlint messages (see attachment). [x]: If (and only if) the source package includes the text of the license(s) in its own file, then that file, containing the text of the license(s) for the package is included in %license. [x]: Package requires other packages for directories it uses. [x]: Package must own all directories that it creates. [x]: Package does not own files or directories owned by other packages. [x]: Package uses either %{buildroot} or $RPM_BUILD_ROOT [x]: Package does not run rm -rf %{buildroot} (or $RPM_BUILD_ROOT) at the beginning of %install. [x]: Macros in Summary, %description expandable at SRPM build time. [x]: Dist tag is present. [x]: Package does not contain duplicates in %files. [x]: Permissions on files are set properly. [x]: Package must not depend on deprecated() packages. [x]: Package use %makeinstall only when make install DESTDIR=... doesn't work. [x]: Package is named using only allowed ASCII characters. [x]: Package does not use a name that already exists. [x]: Package is not relocatable. [x]: Sources used to build the package match the upstream source, as provided in the spec URL. [x]: File names are valid UTF-8. [x]: Packages must not store files under /srv, /opt or /usr/local Python: [x]: Binary eggs must be removed in %prep Note: Cannot find any build in BUILD directory (--prebuilt option?) [x]: Python eggs must not download any dependencies during the build process. [-]: A package which is used by another package via an egg interface should provide egg info. [!]: Package meets the Packaging Guidelines::Python [x]: Package contains BR: python2-devel or python3-devel [x]: Packages MUST NOT have dependencies (either build-time or runtime) on packages named with the unversioned python- prefix unless no properly versioned package exists. Dependencies on Python packages instead MUST use names beginning with python2- or python3- as appropriate. [x]: Python packages must not contain %{pythonX_site(lib|arch)}/* in %files ===== SHOULD items ===== Generic: [-]: If the source package does not include license text(s) as a separate file from upstream, the packager SHOULD query upstream to include it. [x]: Final provides and requires are sane (see attachments). [x]: Package functions as described. [x]: Latest version is packaged. [x]: Package does not include license text files separate from upstream. [!]: Sources are verified with gpgverify first in %prep if upstream publishes signatures. Note: gpgverify is not used. [-]: Description and summary sections in the package spec file contains translations for supported Non-English languages, if available. [-]: Package should compile and build into binary rpms on all supported architectures. [!]: %check is present and all tests pass. [x]: Packages should try to preserve timestamps of original installed files. [x]: Reviewer should test that the package builds in mock. [x]: Buildroot is not present [x]: Package has no %clean section with rm -rf %{buildroot} (or $RPM_BUILD_ROOT) [x]: No file requires outside of /etc, /bin, /sbin, /usr/bin, /usr/sbin. [x]: Packager, Vendor, PreReq, Copyright tags should not be in spec file [x]: Sources can be downloaded from URI in Source: tag [x]: SourceX is a working URL. [x]: Spec use %global instead of %define unless justified. ===== EXTRA items ===== Generic: [x]: Rpmlint is run on all installed packages. Note: There are rpmlint messages (see attachment). Rpmlint ------- Checking: python3-jc-1.10.10-1.fc33.noarch.rpm jc-1.10.10-1.fc33.src.rpm python3-jc.noarch: W: spelling-error Summary(en_US) filetypes -> file types, file-types, teletypes python3-jc.noarch: E: non-executable-script /usr/lib/python3.8/site-packages/jc/cli.py 644 /usr/bin/env python3 python3-jc.noarch: W: spurious-executable-perm /usr/share/doc/python3-jc/README.md python3-jc.noarch: E: script-without-shebang /usr/share/licenses/python3-jc/LICENSE.md python3-jc.noarch: W: no-manual-page-for-binary jc jc.src: W: spelling-error Summary(en_US) filetypes -> file types, file-types, teletypes jc.src: E: invalid-spec-name 2 packages and 0 specfiles checked; 3 errors, 4 warnings. Rpmlint (installed packages) ---------------------------- python3-jc.noarch: W: spelling-error Summary(en_US) filetypes -> file types, file-types, teletypes python3-jc.noarch: E: non-executable-script /usr/lib/python3.8/site-packages/jc/cli.py 644 /usr/bin/env python3 python3-jc.noarch: W: spurious-executable-perm /usr/share/doc/python3-jc/README.md python3-jc.noarch: E: script-without-shebang /usr/share/licenses/python3-jc/LICENSE.md python3-jc.noarch: W: no-manual-page-for-binary jc 1 packages and 0 specfiles checked; 2 errors, 4 warnings. Source checksums ---------------- https://files.pythonhosted.org/packages/source/j/jc/jc-1.10.10.tar.gz : CHECKSUM(SHA256) this package : d76995a8416ee2ddb02314e02ef21ea2a2289ea2a6f2e95f9606faedd55e3ddf CHECKSUM(SHA256) upstream package : d76995a8416ee2ddb02314e02ef21ea2a2289ea2a6f2e95f9606faedd55e3ddf Requires -------- python3-jc (rpmlib, GLIBC filtered): /usr/bin/python3 python(abi) python3.8dist(pygments) python3.8dist(ruamel.yaml) python3.8dist(setuptools) python3.8dist(xmltodict) Provides -------- python3-jc: python-jc python3-jc python3.8-jc python3.8dist(jc) python3dist(jc) Generated by fedora-review 0.7.5 (5fa5b7e) last change: 2020-02-16 Command line :/usr/bin/fedora-review -b 1824848 Buildroot used: fedora-rawhide-x86_64 Active plugins: Python, Shell-api, Generic Disabled plugins: Java, Haskell, C/C++, Perl, PHP, fonts, SugarActivity, R, Ocaml Disabled flags: EPEL6, EPEL7, DISTTAG, BATCH, EXARCH Also, looking at the upstream repo - you could add the contents of the docs/ directory, either as %doc in the main jc package, or as jc-docs sub-package. This would probably require using both the pypi tarball and the GitHub tarball as sources, since the pypi one ships without the docs - but on the other hand, the pypi tarball also ships without tests, so using the GH tarball would allow you to execute those during %check. I have updated the upstream source and spec and rebuilt the SRPM per the above. rpmlint is coming back clean: Issues: ======= - The spec produces a python binary in a "python3-jc" package, instead of just "jc". "python3-XXX" is only required for python modules. I think I have fixed this, but not exactly sure how to test if this is fixed. Also, jc can be used as a python module but will probably be used as a CLI utility 99% of the time. When I tried to change all references of "python3-%{pypi_name}" to "%{pypi_name}", rpm gave me the error: "error: line 24: %package -n jc: package jc already exists" So I reverted those back to "python3-%{pypi_name}". - Spec file name must match the spec package %{name}, in the format %{name}.spec - "python-jc.spec" should be "jc.spec". Fixed - Upstream sources contain a test suite, but the spec does not contain a %check section that would execute those tests. Fixed - I removed the tests from the upstream (pypi) since the fixtures were quite large. - There are extra executable bits on README.md and LICENSE.md. Fixed - removed executable bits. https://raw.githubusercontent.com/kellyjonbrazil/jc-packaging/master/rpmbuild/SPEC/jc.spec https://jc-packages.s3-us-west-1.amazonaws.com/fedora/jc-1.10.12-1.fc32.src.rpm >When I tried to change all references of "python3-%{pypi_name}" to "%{pypi_name}", rpm gave me the error:
>"error: line 24: %package -n jc: package jc already exists"
Since the name of the spec file is "jc.spec" the jc package is created implicitly - "%package -n jc" is thus a re-definition. The "default" %description and %files list (i.e. just "%files", not "%files somename" nor "%files -n somename") apply to the implicit package (i.e. the one with the same name as the spec file).
I went ahead and removed the extra %package and %description sections per your comment above and that seemed to build fine. Is that acceptable? If so, I'll update the SRPM. Can't review it if you don't post it. ;) You got it! Links updated: https://raw.githubusercontent.com/kellyjonbrazil/jc-packaging/master/rpmbuild/SPEC/jc.spec https://jc-packages.s3-us-west-1.amazonaws.com/fedora/jc-1.10.12-1.fc32.src.rpm Looks good to me! I'm marking the package as *approved*, though you'll still need to find a sponsor who will accept your Fedora Account into the "packagers" group. I'll take the liberty and page Robert-André Mauchin, who's a member of the "sponsors" group and who reviewed some of my Python packages in the past. Thank you very much! Appreciate your guidance through this process. Package reviews remain assigned to the reviewer. Let me know if there is anything I need to do. Thanks! Latest version (v1.11.2) update: https://raw.githubusercontent.com/kellyjonbrazil/jc-packaging/master/rpmbuild/SPEC/jc.spec https://jc-packages.s3-us-west-1.amazonaws.com/fedora/jc-1.11.2-1.fc32.src.rpm Kelly, your package has already been approved (fedora-review flag is +). You has not yet been sponsored into the packager group. Without that you cannot request for creating a dist-git repository for the package. According to <https://fedoraproject.org/wiki/Package_Review_Process#Contributor>, you should ask for the sponsorship at <https://pagure.io/packager-sponsors/>. Can I request someone else become the packager? I am the author of this application, so I'm not sure if it is appropriate for me to be the package maintainer. Other distros like OpenSuse, NixOS, FreeBSD ports, and macOS Homebrew have their own package maintainers of this software. A package maintainer does not have the author of the software. You can ask someone else to maintain this package in Fedora. (But don't count on me. I already look after another thousand packages.) There's nothing in Fedora policies preventing upstream developers from being package maintainers for their own software in Fedora, and there's quite a few packages where the maintainer is upstream. I could take this, but I guess that would require someone to do a re-review, since otherwise I'd be reviewing and approving my own package. I can do the review. Artur, could you please open a new review request and mark this as a duplicate? Sure. Made a couple changes to the spec along the way. *** This bug has been marked as a duplicate of bug 1844435 *** |