Rawhide builds have started failing sporadically. Here's and example: https://koji.fedoraproject.org/koji/taskinfo?taskID=105649466 from root.log: ``` DEBUG util.py:444: Total 91 MB/s | 161 MB 00:01 DEBUG util.py:444: Running transaction check DEBUG util.py:444: Transaction check succeeded. DEBUG util.py:444: Running transaction test DEBUG util.py:444: Transaction test succeeded. DEBUG util.py:444: Running transaction DEBUG util.py:444: warning: /etc/hosts created as /etc/hosts.rpmnew DEBUG util.py:444: warning: /etc/dnf/dnf.conf created as /etc/dnf/dnf.conf.rpmnew DEBUG util.py:444: useradd: Warning: missing or non-executable shell '/usr/sbin/nologin' DEBUG util.py:444: Creating group 'input' with GID 104. DEBUG util.py:444: Creating group 'kvm' with GID 36. DEBUG util.py:444: Creating group 'render' with GID 105. DEBUG util.py:444: Creating group 'sgx' with GID 106. DEBUG util.py:444: Creating group 'systemd-journal' with GID 190. DEBUG util.py:444: Creating group 'systemd-oom' with GID 999. DEBUG util.py:444: Creating user 'systemd-oom' (systemd Userspace OOM Killer) with UID 999 and GID 999. DEBUG util.py:444: Created symlink /etc/systemd/system/dbus-org.freedesktop.resolve1.service → /usr/lib/systemd/system/systemd-resolved.service. DEBUG util.py:444: Created symlink /etc/systemd/system/sysinit.target.wants/systemd-resolved.service → /usr/lib/systemd/system/systemd-resolved.service. DEBUG util.py:444: Created symlink /etc/systemd/system/dbus-org.freedesktop.home1.service → /usr/lib/systemd/system/systemd-homed.service. DEBUG util.py:444: Created symlink /etc/systemd/system/multi-user.target.wants/systemd-homed.service → /usr/lib/systemd/system/systemd-homed.service. DEBUG util.py:444: Created symlink /etc/systemd/system/systemd-homed.service.wants/systemd-homed-activate.service → /usr/lib/systemd/system/systemd-homed-activate.service. DEBUG util.py:444: Created symlink /etc/systemd/system/sysinit.target.wants/systemd-pstore.service → /usr/lib/systemd/system/systemd-pstore.service. DEBUG util.py:444: Created symlink /etc/systemd/system/multi-user.target.wants/remote-cryptsetup.target → /usr/lib/systemd/system/remote-cryptsetup.target. DEBUG util.py:444: Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service → /usr/lib/systemd/system/NetworkManager.service. DEBUG util.py:444: Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service → /usr/lib/systemd/system/NetworkManager-dispatcher.service. DEBUG util.py:444: Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service → /usr/lib/systemd/system/NetworkManager-wait-online.service. DEBUG util.py:444: Created symlink /etc/systemd/system/sockets.target.wants/iscsiuio.socket → /usr/lib/systemd/system/iscsiuio.socket. DEBUG util.py:444: Created symlink /etc/systemd/system/sysinit.target.wants/iscsi-starter.service → /usr/lib/systemd/system/iscsi-starter.service. DEBUG util.py:444: Created symlink /etc/systemd/system/sockets.target.wants/iscsid.socket → /usr/lib/systemd/system/iscsid.socket. DEBUG util.py:444: Created symlink /etc/systemd/system/sysinit.target.wants/iscsi-onboot.service → /usr/lib/systemd/system/iscsi-onboot.service. DEBUG util.py:444: Created symlink /etc/systemd/system/sockets.target.wants/dm-event.socket → /usr/lib/systemd/system/dm-event.socket. DEBUG util.py:444: Created symlink /etc/systemd/system/sysinit.target.wants/lvm2-monitor.service → /usr/lib/systemd/system/lvm2-monitor.service. DEBUG util.py:444: Created symlink /etc/systemd/system/sysinit.target.wants/lvm2-lvmpolld.socket → /usr/lib/systemd/system/lvm2-lvmpolld.socket. DEBUG util.py:444: Created symlink /etc/systemd/system/multi-user.target.wants/chronyd.service → /usr/lib/systemd/system/chronyd.service. DEBUG util.py:444: Created symlink /etc/systemd/system/local-fs.target.wants/ostree-remount.service → /usr/lib/systemd/system/ostree-remount.service. DEBUG util.py:444: Created symlink /etc/systemd/system/multi-user.target.wants/mdmonitor.service → /usr/lib/systemd/system/mdmonitor.service. DEBUG util.py:444: Created symlink /etc/systemd/system/timers.target.wants/raid-check.timer → /usr/lib/systemd/system/raid-check.timer. DEBUG util.py:444: Created symlink /etc/systemd/system/graphical.target.wants/udisks2.service → /usr/lib/systemd/system/udisks2.service. DEBUG util.py:444: Created symlink /etc/systemd/system/sysinit.target.wants/selinux-autorelabel-mark.service → /usr/lib/systemd/system/selinux-autorelabel-mark.service. DEBUG util.py:444: Creating group 'systemd-coredump' with GID 997. DEBUG util.py:444: Creating user 'systemd-coredump' (systemd Core Dumper) with UID 997 and GID 997. DEBUG util.py:444: Creating group 'systemd-resolve' with GID 193. DEBUG util.py:444: Creating user 'systemd-resolve' (systemd Resolver) with UID 193 and GID 193. DEBUG util.py:444: Creating group 'systemd-timesync' with GID 996. DEBUG util.py:444: Creating user 'systemd-timesync' (systemd Time Synchronization) with UID 996 and GID 996. DEBUG util.py:595: Child return code was: 255 ``` I think this change: https://github.com/rpm-software-management/dnf/pull/1854 may have caused this? Reproducible: Always
A complete DNF command from the root.log: DEBUG util.py:539: Executing command: ['/usr/bin/systemd-nspawn', '-q', '-M', 'b837ad60f27c4bf5af1a37a227009650', '-D', '/var/lib/mock/f40-build-45245253-5394000-bootstrap/root', '-a', '--capability=cap_ipc_lock', '--bind=/tmp/mock-resolv.nk4k7_1q:/etc/resolv.conf', '--console=pipe', '--setenv=TERM=vt100', '--setenv=SHELL=/bin/bash', '--setenv=HOME=/var/lib/mock/f40-build-45245253-5394000/root/installation-homedir', '--setenv=HOSTNAME=mock', '--setenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin', '--setenv=PROMPT_COMMAND=printf "\\033]0;<mock-chroot>\\007"', '--setenv=PS1=<mock-chroot> \\s-\\v\\$ ', '--setenv=LANG=C.UTF-8', '--setenv=LC_MESSAGES=C.UTF-8', '--resolv-conf=off', '/usr/bin/dnf-3', '--installroot', '/var/lib/mock/f40-build-45245253-5394000/root/', '--setopt=install_weak_deps=0', '--disableplugin=local', '--disableplugin=spacewalk', '--disableplugin=versionlock', 'groupinstall', 'livemedia-build', '--setopt=tsflags=nocontexts'] with env {'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOME': '/var/lib/mock/f40-build-45245253-5394000/root/installation-homedir', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'PROMPT_COMMAND': 'printf "\\033]0;<mock-chroot>\\007"', 'PS1': '<mock-chroot> \\s-\\v\\$ ', 'LANG': 'C.UTF-8', 'LC_MESSAGES': 'C.UTF-8', 'SYSTEMD_NSPAWN_TMPFS_TMP': '0', 'SYSTEMD_SECCOMP': '0'} and shell False
Which version of python3-dnf, python3-libdnf, rpm-libs etc. do you use? Which repository defines "livemedia-build" group? I tried the "/usr/bin/dnf-3 ..." command with Fedora 40 python3-libdnf-0.70.2-1.fc39.x86_64 on f40-build repository. It complains that "livemedia-build" group does not exist. If I use "build" group, it works for me. But I did not run it through systemd-nspawn.
I believe the problem is connected to currently untagged bodhi update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-8782a26bc2, according to messages from related "rawhide buildroot seems weird state, Child return code was: 255" thread on devel mailing list. So specifically: librepo 1.16.0 libdnf 0.71.0 dnf 4.17.0 This is the suspicious merged PR: https://github.com/rpm-software-management/dnf/pull/1984 (which was already merged in the past and then reverted due to rpm 4.18.0 not available on several RHEL/Fedora systems. Although it's most likely the causing PR, the problem is still not clear to me. The provided fix should just harden the RPM transaction from the outside signals, the rpm.blockSignals calls are in the end calling just the rpmsqBlock with SIG_BLOCK or SIG_UNBLOCK argument defined here: https://github.com/rpm-software-management/rpm/blob/master/rpmio/rpmsq.c.
Today, we attempted to collaborate with the guys from Mock and RPM team to address the issue. However, we were unable to reproduce the problem locally without involving the Koji build. Using Koji, it appears that the issue can be easily reproduced by having the problematic packages in the side tag and attempting a scratch build of a package that previously failed with dnf 4.17.0 when it was in Rawhide: https://koji.fedoraproject.org/koji/taskinfo?taskID=105898343. I will explore debugging options with the Koji admin team now.
I'll have a look. This comes from following my suggestion, to use code written by me, so it seems appropriate I take the blame when it doesn't work :D
This bug appears to have been reported against 'rawhide' during the Fedora Linux 40 development cycle. Changing version to 40.
This message is a reminder that Fedora Linux 40 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora Linux 40 on 2025-05-13. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a 'version' of '40'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, change the 'version' to a later Fedora Linux version. Note that the version field may be hidden. Click the "Show advanced fields" button if you do not see it. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora Linux 40 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora Linux, you are encouraged to change the 'version' to a later version prior to this bug being closed.
Fedora Linux 40 entered end-of-life (EOL) status on 2025-05-13. Fedora Linux 40 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora Linux please feel free to reopen this bug against that version. Note that the version field may be hidden. Click the "Show advanced fields" button if you do not see the version field. If you are unable to reopen this bug, please file a new report against an active release. Thank you for reporting this bug and we are sorry it could not be fixed.