Bug 2348934 - systemd-remount-fs.service fails on boot of Silverblue - "mount: /: fsconfig system call failed: overlay: No changes allowed in reconfigure"
Summary: systemd-remount-fs.service fails on boot of Silverblue - "mount: /: fsconfig ...
Keywords:
Status: NEW
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: 42
Hardware: All
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: systemd-maint
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: openqa AcceptedFreezeException https:...
Depends On:
Blocks: F42FinalFreezeException
TreeView+ depends on / blocked
 
Reported: 2025-02-27 22:53 UTC by Adam Williamson
Modified: 2025-04-28 18:54 UTC (History)
15 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed:
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Adam Williamson 2025-02-27 22:53:38 UTC
Since Fedora-42-20250213.n.0 and Fedora-Rawhide-20250211.n.1 , systemd-remount-fs.service is consistently failing on first boot of Silverblue after install, in openQA testing. The journal shows:

Feb 27 04:08:43 fedora systemd-remount-fs[693]: /usr/bin/mount for / exited with exit status 32.
Feb 27 04:08:43 fedora systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE
Feb 27 04:08:43 fedora systemd[1]: systemd-remount-fs.service: Failed with result 'exit-code'.
Feb 27 04:08:43 fedora systemd[1]: Failed to start systemd-remount-fs.service - Remount Root and Kernel File Systems.
Feb 27 04:08:43 fedora systemd-remount-fs[699]: mount: /: fsconfig system call failed: overlay: No changes allowed in reconfigure.
Feb 27 04:08:43 fedora systemd-remount-fs[699]:        dmesg(1) may have more information after failed mount system call.

I'm having trouble isolating what might have caused this. systemd did not change in either affected compose. Neither did util-linux. The kernel changed in the first affected compose on Rawhide, but not on F42. But the bug itself is clear enough, and reproduces every compose.

Comment 1 Timothée Ravier 2025-03-04 21:14:23 UTC
This is triggered on Atomic Desktops & IoT by the migration to composefs: https://gitlab.com/fedora/ostree/sig/-/issues/35

This is tracked in: https://github.com/containers/bootc/issues/971

We will likely end up masking this unit on Atomic Desktops.

Comment 2 Timothée Ravier 2025-03-04 21:33:15 UTC
Another option would be to have it remount /sysroot (the real root mount point on ostree/bootc systems) instead of /

Comment 3 Timothée Ravier 2025-03-04 21:35:56 UTC
Previous discussion in https://github.com/ostreedev/ostree/issues/3193

Comment 4 Zbigniew Jędrzejewski-Szmek 2025-03-24 21:09:58 UTC
Hmm, various ideas were proposed in 3193, but then it was closed as "Not planned".
I still think that systemd-remount-fs can and should be changed to support the
setup where the root filesystem is not what is listed in /etc/fstab. It certainly
shouldn't error out. Masking the service does not seem like a good idea because
the users may rely on it for other filesystems.

Comment 5 Adam Williamson 2025-04-03 22:22:35 UTC
It'd be really nice if we could address this, because this is the *only* test failure in openQA for Fedora 42 right now :D

Comment 6 Adam Williamson 2025-04-03 22:24:54 UTC
Proposing as an FE for Final. If it were in a release-blocking image this would violate Final criterion "All system services present after installation with one of the release-blocking package sets must start properly, unless they require hardware which is not present." Silverblue is not release blocking, and this doesn't seem to be happening on IoT for whatever reason, but it does seem reasonable to give it an FE for Silverblue.

Comment 7 Adam Williamson 2025-04-04 15:41:27 UTC
+4 in https://pagure.io/fedora-qa/blocker-review/issue/1834 , marking accepted FE.

Comment 8 Timothée Ravier 2025-04-07 15:50:00 UTC
This does not happen on IoT anymore as they tweaked the fstab entry to match: https://github.com/fedora-iot/iot-distro/issues/81

For Atomic Desktops, this is triggered by `compress=zstd:1` which is only in /etc/fstab and not on the kernel command line: https://gitlab.com/fedora/ostree/sig/-/issues/72

Comment 9 Timothée Ravier 2025-04-07 17:00:07 UTC
https://github.com/systemd/systemd/compare/main...travier:systemd:remount-fs-sysroot appears to make the error go away, but I had an error previously so I don't know if there is a race or not. It also does not solve the issue for compression (but that could be https://bugzilla.redhat.com/show_bug.cgi?id=2332319 instead or something else)

Comment 10 Timothée Ravier 2025-04-07 17:27:33 UTC
Script like version of what we would have to do to fix the compression karg (would still fail on first boot):

```
# Check stamp or check command line?
if [[ -f "/var/lib/.stamp" ]]; then
    exit 0
fi

fstype="$(findmnt /sysroot --json --output FSTYPE | jq -r '.filesystems[0].fstype')"
if [[ "${fstype}" != "btrfs" ]]; then
    echo "Not using btrfs for /"
    touch /var/lib/.stamp
    exit 0
fi

compress="$(cat /etc/fstab | grep -vE "^#|^$" | awk '{ if ($2 == "/") { print $4 } }' | grep -c '^subvol=root,compress=zstd:1,ro$')"
if [[ "${compress}" != "1" ]]; then
    echo "Not using btrfs for /"
    touch /var/lib/.stamp
    exit 0
fi

rpm-ostree kargs --delete=rootflags=subvol=root --append=rootflags=subvol=root,compress=zstd:1
touch /var/lib/.stamp
```

This should only touch default configs. Maybe we could add that to bootc alongside the workaround that already exists.

Comment 11 Kamil Páral 2025-04-16 12:58:23 UTC
Common Issue description:
https://discussion.fedoraproject.org/t/common-issue/148562


Note You need to log in before you can comment on or make changes to this bug.