Bug 2453414 - Review Request: lemonade-server - Local LLM serving with GPU and NPU acceleration
Summary: Review Request: lemonade-server - Local LLM serving with GPU and NPU acceler...
Keywords:
Status: ASSIGNED
Alias: None
Product: Fedora
Classification: Fedora
Component: Package Review
Version: rawhide
Hardware: All
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Tom.Rix
QA Contact: Fedora Extras Quality Assurance
URL: https://github.com/lemonade-sdk/lemonade
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2026-03-31 16:13 UTC by Kurtis
Modified: 2026-05-14 19:07 UTC (History)
2 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed:
Type: ---
Embargoed:


Attachments (Terms of Use)
The .spec file difference from Copr build 10280942 to 10326544 (2.51 KB, patch)
2026-04-15 03:34 UTC, Fedora Review Service
no flags Details | Diff
The .spec file difference from Copr build 10426885 to 10460881 (1.60 KB, patch)
2026-05-14 19:07 UTC, Fedora Review Service
no flags Details | Diff

Description Kurtis 2026-03-31 16:13:36 UTC
Spec URL: https://download.copr.fedorainfracloud.org/results/kmarinos/lemonade-server/srpm-builds/10279508/lemonade-server.spec
SRPM URL: https://download.copr.fedorainfracloud.org/results/kmarinos/lemonade-server/srpm-builds/10279508/lemonade-server-10.0.1-1.fc45.src.rpm
Description: Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. It supports GGUF, FLM, and ONNX models. It operates as a local backend server.
Fedora Account System Username: kmarinos

Comment 1 Fedora Review Service 2026-04-01 04:03:47 UTC
Copr build:
https://copr.fedorainfracloud.org/coprs/build/10280942
(succeeded)

Review template:
https://download.copr.fedorainfracloud.org/results/@fedora-review/fedora-review-2453414-lemonade-server/fedora-rawhide-x86_64/10280942-lemonade-server/fedora-review/review.txt

Found issues:

- Upstream MD5sum check error, diff is in /var/lib/copr-rpmbuild/results/lemonade-server/diff.txt
  Read more: https://docs.fedoraproject.org/en-US/packaging-guidelines/SourceURL/
- Systemd service file(s) in lemonade-server
  Read more: https://docs.fedoraproject.org/en-US/packaging-guidelines/Scriptlets/#_scriptlets

Please know that there can be false-positives.

---
This comment was created by the fedora-review-service
https://github.com/FrostyX/fedora-review-service

If you want to trigger a new Copr build, add a comment containing new
Spec and SRPM URLs or [fedora-review-service-build] string.

Comment 2 Tom.Rix 2026-04-05 13:08:22 UTC
- Upstream MD5sum check error, diff is in /var/lib/copr-rpmbuild/results/lemonade-server/diff.txt
  Read more: https://docs.fedoraproject.org/en-US/packaging-guidelines/SourceURL/

diff.txt:

diff -U2 -r /sfs/fedora-review/review-lemonade-server/upstream-unpacked/Source0/lemonade-10.0.1/.github/workflows/cpp_server_build_test_release.yml /sfs/fedora-review/review-lemonade-server/srpm-unpacked/lemonade-server-10.0.1.tar.gz-extract/lemonade-10.0.1/.github/workflows/cpp_server_build_test_release.yml
--- /sfs/fedora-review/review-lemonade-server/upstream-unpacked/Source0/lemonade-10.0.1/.github/workflows/cpp_server_build_test_release.yml	2026-03-24 10:43:14.000000000 -0700
+++ /sfs/fedora-review/review-lemonade-server/srpm-unpacked/lemonade-server-10.0.1.tar.gz-extract/lemonade-10.0.1/.github/workflows/cpp_server_build_test_release.yml	2026-03-24 10:00:21.000000000 -0700
@@ -1272,6 +1272,4 @@
       - test-rpm-package
     if: startsWith(github.ref, 'refs/tags/v')
-    env:
-      LEMONADE_VERSION: ${{ needs.build-lemonade-rpm.outputs.version }}
     steps:
       - name: Checkout for release notes action

the lemonade tarball has several hidden dirs like '.github'
It is ok to rm -rf them in the %prep stage.
I do this in several of my packages to make the license check (see below) easier, if the file or dir isn't used it is ok to remove it.

- Systemd service file(s) in lemonade-server
  Read more: https://docs.fedoraproject.org/en-US/packaging-guidelines/Scriptlets/#_scriptlets

I believe this is catching how you are adding the lemonade user.
Review this document 
https://docs.fedoraproject.org/en-US/packaging-guidelines/UsersAndGroups/
My package ollama adds the ollama user, following the dynamic user case in the doc.
I think this is what you want to do.

License:        MIT

[!]: License field in the package spec file matches the actual license.
     Note: Checking patched sources after %prep for licenses. Licenses
     found: "Unknown or generated", "*No copyright* Apache License 2.0",
     "GNU General Public License, Version 2 and/or ISC License and/or MIT
     License", "*No copyright* Apache License", "MIT License", "Apache
     License 2.0", "Apache License 2.0 and/or GNU General Public License,
     Version 2 and/or ISC License and/or MIT License", "*No copyright* MIT
     License", "GNU General Public License, Version 2". 350 files have
     unknown license. Detailed output of licensecheck in /sfs/fedora-
     review/review-lemonade-server/licensecheck.txt

Review the licensecheck.txt file from fedora-review, this looks like Apache-2.0

# Restrict secrets file to be readable only by root and the lemonade service group
%attr(0660, root, lemonade) %config(noreplace) %{_sysconfdir}/lemonade/conf.d/zz-secrets.conf

     review/review-lemonade-server/licensecheck.txt
[!]: Package requires other packages for directories it uses.
     Note: No known owner of /etc/lemonade/conf.d
[!]: Package must own all directories that it creates.
     Note: Directories without known owners: lemonade-
     server:/etc/lemonade/conf.d

This will likely be reworked as part of adding lemonade user

URL:            https://github.com/lemonade-sdk/lemonade
Source0:        https://github.com/lemonade-sdk/lemonade/archive/refs/tags/v%{version}.tar.gz#/%{name}-%{version}.tar.gz

should use %{url}/archives/... for source0

Summary:        Local LLM serving with GPU and NPU acceleration

Should there be some Requires: to pick up needed gpu or npu packages to run ?

rpmlint 

lemonade-server.spec:10: W: unversioned-explicit-provides user(lemonade)
lemonade-server.spec:11: W: unversioned-explicit-provides group(lemonade)
lemonade-server.x86_64: W: non-standard-gid /etc/lemonade/conf.d/zz-secrets.conf lemonade
lemonade-server.x86_64: E: non-readable /etc/lemonade/conf.d/zz-secrets.conf 660

this will change with how adding lemonade user changes. 

lemonade-server.spec: W: no-%check-section

How do we test this package ?
If if depends on a real gpu, consider adding some conditional logic like the rocblas package does.

Comment 3 Kurtis 2026-04-14 16:28:09 UTC
Spec URL: https://download.copr.fedorainfracloud.org/results/kmarinos/lemonade-server/srpm-builds/10323178/lemonade-server.spec
SRPM URL: https://download.copr.fedorainfracloud.org/results/kmarinos/lemonade-server/srpm-builds/10323178/lemonade-server-10.0.1-1.fc45.src.rpm

Here are the changes I made after reading your feedback:

- Removed hidden upstream directories (.github, etc.) in %prep since 
  they are not used in the build and were causing the tarball
  diff. I also updated Source0 to use %{url}.

- Systemd service scriptlets: Added %post, %preun, and %postun sections using
  %systemd_post, %systemd_preun, and %systemd_postun_with_restart macros per
  the Scriptlets guidelines.

- User/group creation: Replaced manual groupadd/useradd in %pre with sysusers.d
  allocation in accordance to the documentation for Users and Groups. This also resolves the
  rpmlint unversioned-explicit-provides warnings since the Provides are now
  auto-generated.

- Added %dir %{_sysconfdir}/lemonade/conf.d to ensure the
  package owns all directories it creates.

- Changed secrets.conf permissions from 0660 to 0640. File is intentionally not
  world-readable since it contains secrets, which rpmlint makes it appear as an error, though I 
  believe this a false-positive.

- I added a conditional %check gated behind %bcond_with check since
  tests require GPU hardware.

- As for ROCm support, lemonade-server uses the default system backend for now. My plan was to add
  ROCm support once the package has been reviewed and added to the Fedora ecosystem, where I could then update
  the package with ROCm support as optional.

- Modified the license in the SPEC. I forgot to modify that portion of the SPEC when I first auto-generated the template, my mistake. It should be okay now.

Comment 4 Fedora Review Service 2026-04-15 03:34:52 UTC
Created attachment 2137145 [details]
The .spec file difference from Copr build 10280942 to 10326544

Comment 5 Fedora Review Service 2026-04-15 03:34:54 UTC
Copr build:
https://copr.fedorainfracloud.org/coprs/build/10326544
(succeeded)

Review template:
https://download.copr.fedorainfracloud.org/results/@fedora-review/fedora-review-2453414-lemonade-server/fedora-rawhide-x86_64/10326544-lemonade-server/fedora-review/review.txt

Found issues:

- Upstream MD5sum check error, diff is in /var/lib/copr-rpmbuild/results/lemonade-server/diff.txt
  Read more: https://docs.fedoraproject.org/en-US/packaging-guidelines/SourceURL/

Please know that there can be false-positives.

---
This comment was created by the fedora-review-service
https://github.com/FrostyX/fedora-review-service

If you want to trigger a new Copr build, add a comment containing new
Spec and SRPM URLs or [fedora-review-service-build] string.

Comment 6 Tom.Rix 2026-04-19 16:28:54 UTC
Trying an install
https://github.com/trixirt/rocm-distro-containers/blob/main/fedora/rawhide/lemonade-server/Dockerfile

0.130 Verifying...                          ########################################                                                                 
0.134 Preparing...                          ########################################                                                                 
0.210 <stdin>:1: '%{_sharedstatedir}/lemonade' is not a valid home directory field.                                                                  
0.211 error: %sysusers(lemonade-server-10.0.1-1.fc45.x86_64) scriptlet failed, exit status 1
0.211 error: lemonade-server-10.0.1-1.fc45.x86_64: install failed

Some problem in the install in at least containers

lemonade-server looks like a network server plugging into existing local llm applications as its backends
I would expect there to be something like

%Requires llama-cpp

for at least the cpu to work.

Though this application could build on all arches, the list of aches should be the intersection of the backends.
Ex/ llama-cpp only supports x86_64 and aarch64

https://src.fedoraproject.org/rpms/llama-cpp/blob/rawhide/f/llama-cpp.spec#_46

Which arches should also reflect what you are willing to support.
If ex/ you have no powerpc hw, I would recommend not adding that but rather reach out to someone who does and would be interested in comaintaining.

I am not sure if there is a ctest tests, it looks like tests are some python scripts here
https://github.com/lemonade-sdk/lemonade/tree/v10.0.1/test

I see the upstream has v10.2
Please review and update if you think it is worth it, generally package reviews should be updated with the latest but that may not be practical if the upstream is moving crazy fast like llama-cpp.

Comment 7 Kurtis 2026-05-05 17:14:48 UTC
Spec URL: https://download.copr.fedorainfracloud.org/results/kmarinos/lemonade-server/srpm-builds/10426623/lemonade-server.spec
SRPM URL: https://download.copr.fedorainfracloud.org/results/kmarinos/lemonade-server/srpm-builds/10426623/lemonade-server-10.2.0-1.fc45.src.rpm

Changes:

- Fixed sysusers.d Container Install Failure
- Changed home directory in lemonade-server.sysusers from RPM RPM macro %{_sharedstatedir}/lemonade to literal path /var/lib/lemonade

This should address the install failure in containers.

- Updated to v10.2.0. I will have to update to 10.3 shortly, but wanted to provide an update until then.
- Updated %files section: Added lemond binary and man page (new in 10.2.0), and added %{_datadir}/lemonade/ directory (also new in 10.2.0)
- Removed lemonade-router binary and lemonade.conf (removed upstream)

Fixed Licensing Field:
- Changed from "Apache-2.0" to "Apache-2.0 AND GPL-2.0-only WITH Linux-syscall-note AND MIT". Added respective License breakdown comments documenting each component.

- Added Bundled Library Declaration: aixlog is a header-only library not available as a Fedora package

- Added ExclusiveArch: x86_64. This is to match llama-cpp backend support, as well as the architecture I can support and test.

- Added Requires: llama-cpp

- Updated %check Section. Changed from %ctest to %pytest. Upstream tests are Python-based tests.
- Added conditional BuildRequires for python3-pytest and python3-requests
- Tests continue to be behind %bcond_with check to meet GPU/APU hardware requirement

There were 4 warnings provided when running rpmlint: (setlocale, non-standard-gid,
non-readable secrets file, empty-%pre on systemd 260+). These are expected.

Comment 8 Fedora Review Service 2026-05-05 17:27:50 UTC
Copr build:
https://copr.fedorainfracloud.org/coprs/build/10426885
(succeeded)

Review template:
https://download.copr.fedorainfracloud.org/results/@fedora-review/fedora-review-2453414-lemonade-server/fedora-rawhide-x86_64/10426885-lemonade-server/fedora-review/review.txt

Please take a look if any issues were found.


---
This comment was created by the fedora-review-service
https://github.com/FrostyX/fedora-review-service

If you want to trigger a new Copr build, add a comment containing new
Spec and SRPM URLs or [fedora-review-service-build] string.

Comment 9 Kurtis 2026-05-14 18:39:51 UTC
Spec URL: https://download.copr.fedorainfracloud.org/results/kmarinos/lemonade-server/srpm-builds/10460803/lemonade-server.spec
SRPM URL: https://download.copr.fedorainfracloud.org/results/kmarinos/lemonade-server/srpm-builds/10460803/lemonade-server-10.4.0-1.fc45.src.rpm

Updated package to v10.4.0

Systemd Service File Renamed:
- Renamed upstream service file from lemonade-server.service to lemond.service
- Service file was no longer being installed by cmake so I added a manual install in %install
- Updated %post, %preun, %postun scriptlets to reference lemond.service and updated %files to reference lemond.service

Fixed %check Section:
- I took a look at the --with-check option while building, and investigated the py tests. From my understanding it appears that the upstream
  tests are integration tests that require a running lemonade-server instance, the binary at the upstream dev build path (build/lemonade-server), and GPU hardware.
  I documented this in a comment within the check section of the spec file. The tests cannot run in a build environment

- Replaced %pytest macro with direct python3 -m pytest call, the %pytest macro was not expanding correctly when troubleshooting.
- Added python3-openai to conditional BuildRequires required by the tests.

Comment 10 Fedora Review Service 2026-05-14 19:07:09 UTC
Created attachment 2141302 [details]
The .spec file difference from Copr build 10426885 to 10460881

Comment 11 Fedora Review Service 2026-05-14 19:07:11 UTC
Copr build:
https://copr.fedorainfracloud.org/coprs/build/10460881
(succeeded)

Review template:
https://download.copr.fedorainfracloud.org/results/@fedora-review/fedora-review-2453414-lemonade-server/fedora-rawhide-x86_64/10460881-lemonade-server/fedora-review/review.txt

Please take a look if any issues were found.


---
This comment was created by the fedora-review-service
https://github.com/FrostyX/fedora-review-service

If you want to trigger a new Copr build, add a comment containing new
Spec and SRPM URLs or [fedora-review-service-build] string.


Note You need to log in before you can comment on or make changes to this bug.