Bug 2359764 - rpm-ostree fails to upgrade if systemtap-runtime is layered
Summary: rpm-ostree fails to upgrade if systemtap-runtime is layered
Keywords:
Status: NEW
Alias: None
Product: Fedora
Classification: Fedora
Component: rpm-ostree
Version: 42
Hardware: Unspecified
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Colin Walters
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2025-04-15 13:24 UTC by Marco Huenseler
Modified: 2025-06-11 20:37 UTC (History)
21 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed:
Type: ---
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github coreos rpm-ostree issues 5365 0 None open Cannot layer any package on Fedora 42 2025-05-16 16:53:15 UTC
Github fedora-silverblue issue-tracker issues 643 0 None closed Error layering Qemu: error: While applying overrides for pkg systemtap-runtime: Could not find group 'stapusr' in group ... 2025-05-16 16:53:15 UTC
Red Hat Issue Tracker FC-1656 0 None None None 2025-05-12 13:27:11 UTC

Description Marco Huenseler 2025-04-15 13:24:57 UTC
I have layered several qemu and libvirt packages which depend on systemtap. When I try to upgrade my system, the following happens:

$ sudo rpm-ostree update
[sudo] Passwort für user:
⠉ Receiving metadata objects: 0/(estimating) -/s 0 Bytes...                                                                                                                                                                                                   2 metadata, 0 content objects fetched; 788 B transferred in 7 seconds; 0 Bytes content written
Receiving metadata objects: 0/(estimating) -/s 0 Bytes... done
Checking out tree ed4e319... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora rpmfusion-free-updates rpmfusion-free rpmfusion-nonfree-updates rpmfusion-nonfree updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2024-08-21T16:04:02Z solvables: 3
rpm-md repo 'updates' (cached); generated: 2025-04-13T04:06:21Z solvables: 3236
rpm-md repo 'fedora' (cached); generated: 2025-04-11T05:17:07Z solvables: 76879
rpm-md repo 'rpmfusion-free-updates' (cached); generated: 2025-02-06T18:04:47Z solvables: 0
rpm-md repo 'rpmfusion-free' (cached); generated: 2025-04-12T09:12:27Z solvables: 358
rpm-md repo 'rpmfusion-nonfree-updates' (cached); generated: 2025-02-06T18:07:03Z solvables: 0
rpm-md repo 'rpmfusion-nonfree' (cached); generated: 2025-04-12T09:32:45Z solvables: 218
rpm-md repo 'updates-archive' (cached); generated: 2025-02-06T19:33:32Z solvables: 0
Resolving dependencies... done
Checking out packages... done
Running systemd-sysusers... done
Running pre scripts... done
Running post scripts... done
error: While applying overrides for pkg systemtap-runtime: Could not find group 'stapusr' in group file

Apparently, there is already a bug report in the Silverblue issue tracker describing the same issue: https://github.com/fedora-silverblue/issue-tracker/issues/643

While I'm not quite sure what's wrong, the cause seems to be with the systemtap package and its use of sysusers (see Github Issue).

Reproducible: Always

Steps to Reproduce:
1. Layer qemu in Fedora 41
2. Upgrade to Fedora 42 (which works)
3. Try to update to a more recent version

According to the issue linked above, simply running "rpm-ostree install qemu" on F42 also works to reproduce this issue

Comment 1 Alex Haydock 2025-04-21 20:52:27 UTC
I've submitted a PR upstream which I hope would fix this issue by updating the systemtap spec to use the modern method of creating users via sysusers:

https://src.fedoraproject.org/rpms/systemtap/pull-request/32

I'm new to Fedora Packaging so it's possible I've missed some things here. Apologies if so, but hopefully this does at least a chunk of the work and helps out.

Comment 2 Daniel Berrangé 2025-04-22 16:15:38 UTC
AFAICT this was fixed upstream by

commit f892ad26592f97d32984ae088634e82c3c973138
Author: Zbigniew Jędrzejewski-Szmek <zbyszek.pl>
Date:   Mon Mar 10 18:32:19 2025 -0400

    systemtap.spec: Only run sysusers creation scriptlets in Fedora <= 41
    
    In later releases, rpm will do this automatically if it sees the
    sysusers file in the payload.
    
    https://fedoraproject.org/wiki/Changes/RPMSuportForSystemdSysusers

but that change needs to be pulled into Fedora, which hasn't had any changes synced from upstream since Feb

Comment 3 thrashwerk 2025-05-08 13:03:27 UTC
> 2. Upgrade to Fedora 42 (which works)

It doesn't work for me. I've got qemu-kvm qemu-tools layered and when I try to upgrade to 42 (`rpm-ostree rebase fedora/42/x86_64/kinoite`) I get the same error.

Updating to new 41 versions works but I can't upgrade.

My setup:
● fedora:fedora/41/x86_64/kinoite (index: 0)
                  Version: 41.20250508.0 (2025-05-08T02:25:14Z)
               BaseCommit: b82deaa4f1a6946693fcda2864580a2af336c7179738b2dc4d366472d3394c80
                           ├─ repo-0 (2024-10-24T13:55:59Z)
                           ├─ repo-1 (2025-05-08T01:56:33Z)
                           └─ repo-2 (2025-05-08T02:03:25Z)
                   Commit: 69796bd5cc3e26a239da22f7341a667b0d27dd433be1793d4586c1d50a8ee98a
                           ├─ fedora (2024-10-24T13:55:59Z)
                           ├─ fedora-cisco-openh264 (2024-03-11T19:22:31Z)
                           ├─ hashicorp (2025-05-07T17:33:04Z)
                           ├─ rpmfusion-free (2024-10-27T07:49:25Z)
                           ├─ rpmfusion-free-updates (2025-05-01T16:49:04Z)
                           ├─ updates (2025-05-07T03:51:46Z)
                           └─ updates-archive (2025-05-08T03:36:02Z)
                   Staged: no
                StateRoot: fedora
             GPGSignature: 1 signature
                           Signature made 2025 m. gegužės 08 d. 05:29:55 using RSA key ID D0622462E99D6AD1
                           Good signature from "Fedora <fedora-41-primary>"
      RemovedBasePackages: firefox firefox-langpacks 138.0.1-1.fc41
          LayeredPackages: ansible cowsay distrobox google-noto-fonts-all-vf google-roboto-mono-fonts guestfs-tools ksshaskpass libvirt-daemon-config-network libvirt-daemon-kvm libvirt-devel lm_sensors packer python3-libguestfs qemu-kvm qemu-tools
                           rpmfusion-free-release rsms-inter-fonts rubygem-ruby-libvirt vagrant vim virt-install virt-manager virt-top virt-viewer VirtualBox yubikey-manager

Comment 4 slentz 2025-05-08 18:12:38 UTC
(In reply to thrashwerk from comment #3)
> > 2. Upgrade to Fedora 42 (which works)
> 
> It doesn't work for me. I've got qemu-kvm qemu-tools layered and when I try
> to upgrade to 42 (`rpm-ostree rebase fedora/42/x86_64/kinoite`) I get the
> same error.
...

Same goes for me for Fedora SilverBlue 42 (still no update possible for more than 2 weeks) : 
May 08 18:57:37 silver42 rpm-ostree[3358]: Downloading: http://ftp.lip6.fr/ftp/pub/linux/distributions/fedora/updates/42/Everything/x86_64/Packages/s/systemtap-runtime-5.3-1.fc42.x86_64.rpm
May 08 18:57:37 silver42 rpm-ostree[3358]: Downloading: http://ftp.lip6.fr/ftp/pub/linux/distributions/fedora/updates/42/Everything/x86_64/Packages/s/systemtap-client-5.3-1.fc42.x86_64.rpm
May 08 18:57:48 silver42 rpm-ostree[5466]: Creating user 'stapunpriv' (systemtap unprivileged user) with UID 159 and GID 159.
May 08 18:58:43 silver42 rpm-ostree[3358]: Txn Upgrade on /org/projectatomic/rpmostree1/fedora failed: While applying overrides for pkg systemtap-runtime: Could not find group 'stapusr' in group file
May 08 19:40:21 silver42 rpm-ostree[4567]: Creating user 'stapunpriv' (systemtap unprivileged user) with UID 159 and GID 159.
May 08 19:41:01 silver42 rpm-ostree[3348]: Txn Upgrade on /org/projectatomic/rpmostree1/fedora failed: While applying overrides for pkg systemtap-runtime: Could not find group 'stapusr' in group file
May 08 19:44:19 silver42 rpm-ostree[4584]: Creating user 'stapunpriv' (systemtap unprivileged user) with UID 159 and GID 159.
May 08 19:44:56 silver42 rpm-ostree[3343]: Txn Upgrade on /org/projectatomic/rpmostree1/fedora failed: While applying overrides for pkg systemtap-runtime: Could not find group 'stapusr' in group file

My set-up : 
● fedora:fedora/42/x86_64/silverblue
                  Version: 42.20250416.0 (2025-04-16T02:32:27Z)
               BaseCommit: 6f6e1710b3428eeb94f49ebd561a62df99566c499bbaac76eb4bba9f0da80c14
             GPGSignature: Valid signature by B0F4950458F69E1150C6C5EDC8AC4916105EF944
      RemovedBasePackages: mesa-va-drivers 25.0.2-3.fc42 noopenh264 2.5.0-2.fc42
          LayeredPackages: android-tools asunder autofs distrobox file-roller-nautilus gstreamer1-plugin-openh264 gstreamer1-plugins-ugly libvirt mesa-va-drivers-freeworld mesa-vdpau-drivers-freeworld
                           mozilla-openh264 nnn openssl qemu qemu-kvm rpmfusion-free-release rpmfusion-nonfree-release tcpdump virt-manager
             InitramfsEtc: /etc/vconsole.conf

Comment 5 Alex Haydock 2025-05-10 07:47:36 UTC
I can confirm I'm still seeing the above issue too. I think there must be more in the upstream spec file that needs fixing.

I had hoped that the promotion of this package build to stable in F42 a few days ago would solve this: https://bodhi.fedoraproject.org/updates/FEDORA-2025-fb27254eca

That package build is the first stable release to incorporate f892ad26592f97d32984ae088634e82c3c973138 which looked like it was specifically targeting this issue. (Commit msg: 'Only run sysusers creation scriptlets in Fedora <= 41')

But it seems unfortunately that this hasn't solved the issue and we're still unable to layer `qemu` on any Fedora Atomic distros.

Comment 6 Zbigniew Jędrzejewski-Szmek 2025-05-12 13:26:33 UTC
Looks like the rpm-ostree tooling is not creating the stapusr group, even though
it creates the stapunpriv user… Reassigning for comments.

  $ rpm -qP systemtap-runtime | rg 'user|group'
  group(stapdev) = ZyBzdGFwZGV2IDE1OAAA
  group(stapsys) = ZyBzdGFwc3lzIDE1NwAA
  group(stapunpriv)
  group(stapunpriv) = ZyBzdGFwdW5wcml2IDE1OQAA
  group(stapusr) = ZyBzdGFwdXNyIDE1NgAA
  groupmember(stapunpriv/stapunpriv) = bSBzdGFwdW5wcml2IHN0YXB1bnByaXYA
  user(stapunpriv) =
dSBzdGFwdW5wcml2IDE1OSAic3lzdGVtdGFwIHVucHJpdmlsZWdlZCB1c2VyIiAvdmFyL2xpYi9zdGFwdW5wcml2IC9zYmluL25vbG9naW4A
  or
  $ rpm -q --qf='[%{SYSUSERS}\n]' systemtap-runtime
  g stapdev 158
  g stapsys 157
  g stapunpriv 159
  g stapusr 156
  m stapunpriv stapunpriv
  u stapunpriv 159 "systemtap unprivileged user" /var/lib/stapunpriv /sbin/nologin

  $ rpm -ql systemtap-runtime | rg sysusers
  /usr/lib/sysusers.d/systemtap-runtime.conf)

  $ cat /usr/lib/sysusers.d/systemtap-runtime.conf
  g     stapusr  156
  g     stapsys  157
  g     stapdev  158
  g     stapunpriv 159
  u     stapunpriv 159      "systemtap unprivileged user"   /var/lib/stapunpriv   /sbin/nologin
  m     stapunpriv stapunpriv

  A lot of this is redundant. The following would be enough:
  g     stapusr  156
  g     stapsys  157
  g     stapdev  158
  u     stapunpriv 159      "systemtap unprivileged user"   /var/lib/stapunpriv
  But the redundancy should not cause a problem.

Comment 7 Timothée Ravier 2025-05-15 16:34:32 UTC
This is really weird. The wireshark package has a very similar setup (a binary with a specific group ownership) and if I overlay just his package, itworks. But if I install both wireshark & systemtap-runtime, it fails, even for the wireshark case as well.

Comment 8 Timothée Ravier 2025-05-15 17:11:43 UTC
Hum, and now it does not work for wireshark anymore. It looks like we regressed in 2025.08 as it works in 2025.07 for wireshark.

Comment 9 Timothée Ravier 2025-05-16 16:49:17 UTC
And I can't make it work for wireshark with 2025.07. This is really weird.

Comment 10 james 2025-05-17 10:42:29 UTC
I have this same issue with openvpn, which I have reported in #2366260.  Neither my fedora 42 coreos, 41 coreos or 41/42 silverblue are working with openvpn installed/prepared for deploy.

I get the same error - unable to find openvpn group.

No fix or workaround has been suggested anywhere, and I have a server about to go to production, so I am a little bit frustrated...

Comment 11 Timothée Ravier 2025-05-19 14:13:47 UTC
There is a workaround: use a sysext instead of layering: https://extensions.fcos.fr/

Comment 12 thrashwerk 2025-05-20 14:13:12 UTC
(In reply to Timothée Ravier from comment #11)
> There is a workaround: use a sysext instead of layering:
> https://extensions.fcos.fr/

Another option would be to use Distrobox, they already have some docs for libvirt https://github.com/89luca89/distrobox/blob/main/docs/posts/run_libvirt_in_distrobox.md

In my case, I used the registry.fedoraproject.org/fedora-toolbox:42 image just so the container and the main OS are the same.
Instead of `--unshare-all` I used `--unshare-groups --unshare-ipc --unshare-netns --unshare-process` because I needed /dev mounted for VirtualBox (funky setup at work) to work.

Would probably work for Wireshark too if you don't use `--unshare-netns` since the container would be connected to the same interfaces as the host.

Comment 13 Alex Haydock 2025-06-11 20:37:36 UTC
I like the workarounds mentioned in the thread so far, but one of the main packages impacted by this is `qemu` which can run in Distrobox or Toolbx as mentioned, but that doesn't allow for PCI Passthrough or some of the more low-level features (that I can tell), so I think this is still quite an impactful bug.

So I'm keen to see if there's any paths towards resolution upstream that might be worth exploring. Willing to try some testing but I'm not too sure where to start.


Note You need to log in before you can comment on or make changes to this bug.