Since Fedora-42-20250213.n.0 and Fedora-Rawhide-20250211.n.1 , systemd-remount-fs.service is consistently failing on first boot of Silverblue after install, in openQA testing. The journal shows: Feb 27 04:08:43 fedora systemd-remount-fs[693]: /usr/bin/mount for / exited with exit status 32. Feb 27 04:08:43 fedora systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE Feb 27 04:08:43 fedora systemd[1]: systemd-remount-fs.service: Failed with result 'exit-code'. Feb 27 04:08:43 fedora systemd[1]: Failed to start systemd-remount-fs.service - Remount Root and Kernel File Systems. Feb 27 04:08:43 fedora systemd-remount-fs[699]: mount: /: fsconfig system call failed: overlay: No changes allowed in reconfigure. Feb 27 04:08:43 fedora systemd-remount-fs[699]: dmesg(1) may have more information after failed mount system call. I'm having trouble isolating what might have caused this. systemd did not change in either affected compose. Neither did util-linux. The kernel changed in the first affected compose on Rawhide, but not on F42. But the bug itself is clear enough, and reproduces every compose.
This is triggered on Atomic Desktops & IoT by the migration to composefs: https://gitlab.com/fedora/ostree/sig/-/issues/35 This is tracked in: https://github.com/containers/bootc/issues/971 We will likely end up masking this unit on Atomic Desktops.
Another option would be to have it remount /sysroot (the real root mount point on ostree/bootc systems) instead of /
Previous discussion in https://github.com/ostreedev/ostree/issues/3193
Hmm, various ideas were proposed in 3193, but then it was closed as "Not planned". I still think that systemd-remount-fs can and should be changed to support the setup where the root filesystem is not what is listed in /etc/fstab. It certainly shouldn't error out. Masking the service does not seem like a good idea because the users may rely on it for other filesystems.
It'd be really nice if we could address this, because this is the *only* test failure in openQA for Fedora 42 right now :D
Proposing as an FE for Final. If it were in a release-blocking image this would violate Final criterion "All system services present after installation with one of the release-blocking package sets must start properly, unless they require hardware which is not present." Silverblue is not release blocking, and this doesn't seem to be happening on IoT for whatever reason, but it does seem reasonable to give it an FE for Silverblue.
+4 in https://pagure.io/fedora-qa/blocker-review/issue/1834 , marking accepted FE.
This does not happen on IoT anymore as they tweaked the fstab entry to match: https://github.com/fedora-iot/iot-distro/issues/81 For Atomic Desktops, this is triggered by `compress=zstd:1` which is only in /etc/fstab and not on the kernel command line: https://gitlab.com/fedora/ostree/sig/-/issues/72
https://github.com/systemd/systemd/compare/main...travier:systemd:remount-fs-sysroot appears to make the error go away, but I had an error previously so I don't know if there is a race or not. It also does not solve the issue for compression (but that could be https://bugzilla.redhat.com/show_bug.cgi?id=2332319 instead or something else)
Script like version of what we would have to do to fix the compression karg (would still fail on first boot): ``` # Check stamp or check command line? if [[ -f "/var/lib/.stamp" ]]; then exit 0 fi fstype="$(findmnt /sysroot --json --output FSTYPE | jq -r '.filesystems[0].fstype')" if [[ "${fstype}" != "btrfs" ]]; then echo "Not using btrfs for /" touch /var/lib/.stamp exit 0 fi compress="$(cat /etc/fstab | grep -vE "^#|^$" | awk '{ if ($2 == "/") { print $4 } }' | grep -c '^subvol=root,compress=zstd:1,ro$')" if [[ "${compress}" != "1" ]]; then echo "Not using btrfs for /" touch /var/lib/.stamp exit 0 fi rpm-ostree kargs --delete=rootflags=subvol=root --append=rootflags=subvol=root,compress=zstd:1 touch /var/lib/.stamp ``` This should only touch default configs. Maybe we could add that to bootc alongside the workaround that already exists.
Common Issue description: https://discussion.fedoraproject.org/t/common-issue/148562