Bug 2348934 - systemd-remount-fs.service fails on boot of Silverblue - "mount: /: fsconfig system call failed: overlay: No changes allowed in reconfigure"
Summary: systemd-remount-fs.service fails on boot of Silverblue - "mount: /: fsconfig ...
Keywords:
Status: NEW
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: rawhide
Hardware: All
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: systemd-maint
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: openqa AcceptedFreezeException https:...
: 2379895 (view as bug list)
Depends On:
Blocks: F42FinalFreezeException
TreeView+ depends on / blocked
 
Reported: 2025-02-27 22:53 UTC by Adam Williamson
Modified: 2026-03-07 19:57 UTC (History)
24 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed:
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github ostreedev ostree issues 3193 0 None closed systemd-remount-fs.service fail with composefs enabled 2026-01-04 18:53:58 UTC
Github systemd systemd pull 36867 0 None open Skip foreign mounts in remount-fs 2026-01-04 18:53:58 UTC
Gitlab fedora/ostree sig issues 72 0 None opened root mount options from /etc/fstab are not taken into account anymore with composefs (symptom: `systemd-remount-fs.servi... 2026-01-04 18:53:58 UTC

Description Adam Williamson 2025-02-27 22:53:38 UTC
Since Fedora-42-20250213.n.0 and Fedora-Rawhide-20250211.n.1 , systemd-remount-fs.service is consistently failing on first boot of Silverblue after install, in openQA testing. The journal shows:

Feb 27 04:08:43 fedora systemd-remount-fs[693]: /usr/bin/mount for / exited with exit status 32.
Feb 27 04:08:43 fedora systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE
Feb 27 04:08:43 fedora systemd[1]: systemd-remount-fs.service: Failed with result 'exit-code'.
Feb 27 04:08:43 fedora systemd[1]: Failed to start systemd-remount-fs.service - Remount Root and Kernel File Systems.
Feb 27 04:08:43 fedora systemd-remount-fs[699]: mount: /: fsconfig system call failed: overlay: No changes allowed in reconfigure.
Feb 27 04:08:43 fedora systemd-remount-fs[699]:        dmesg(1) may have more information after failed mount system call.

I'm having trouble isolating what might have caused this. systemd did not change in either affected compose. Neither did util-linux. The kernel changed in the first affected compose on Rawhide, but not on F42. But the bug itself is clear enough, and reproduces every compose.

Comment 1 Timothée Ravier 2025-03-04 21:14:23 UTC
This is triggered on Atomic Desktops & IoT by the migration to composefs: https://gitlab.com/fedora/ostree/sig/-/issues/35

This is tracked in: https://github.com/containers/bootc/issues/971

We will likely end up masking this unit on Atomic Desktops.

Comment 2 Timothée Ravier 2025-03-04 21:33:15 UTC
Another option would be to have it remount /sysroot (the real root mount point on ostree/bootc systems) instead of /

Comment 3 Timothée Ravier 2025-03-04 21:35:56 UTC
Previous discussion in https://github.com/ostreedev/ostree/issues/3193

Comment 4 Zbigniew Jędrzejewski-Szmek 2025-03-24 21:09:58 UTC
Hmm, various ideas were proposed in 3193, but then it was closed as "Not planned".
I still think that systemd-remount-fs can and should be changed to support the
setup where the root filesystem is not what is listed in /etc/fstab. It certainly
shouldn't error out. Masking the service does not seem like a good idea because
the users may rely on it for other filesystems.

Comment 5 Adam Williamson 2025-04-03 22:22:35 UTC
It'd be really nice if we could address this, because this is the *only* test failure in openQA for Fedora 42 right now :D

Comment 6 Adam Williamson 2025-04-03 22:24:54 UTC
Proposing as an FE for Final. If it were in a release-blocking image this would violate Final criterion "All system services present after installation with one of the release-blocking package sets must start properly, unless they require hardware which is not present." Silverblue is not release blocking, and this doesn't seem to be happening on IoT for whatever reason, but it does seem reasonable to give it an FE for Silverblue.

Comment 7 Adam Williamson 2025-04-04 15:41:27 UTC
+4 in https://pagure.io/fedora-qa/blocker-review/issue/1834 , marking accepted FE.

Comment 8 Timothée Ravier 2025-04-07 15:50:00 UTC
This does not happen on IoT anymore as they tweaked the fstab entry to match: https://github.com/fedora-iot/iot-distro/issues/81

For Atomic Desktops, this is triggered by `compress=zstd:1` which is only in /etc/fstab and not on the kernel command line: https://gitlab.com/fedora/ostree/sig/-/issues/72

Comment 9 Timothée Ravier 2025-04-07 17:00:07 UTC
https://github.com/systemd/systemd/compare/main...travier:systemd:remount-fs-sysroot appears to make the error go away, but I had an error previously so I don't know if there is a race or not. It also does not solve the issue for compression (but that could be https://bugzilla.redhat.com/show_bug.cgi?id=2332319 instead or something else)

Comment 10 Timothée Ravier 2025-04-07 17:27:33 UTC
Script like version of what we would have to do to fix the compression karg (would still fail on first boot):

```
# Check stamp or check command line?
if [[ -f "/var/lib/.stamp" ]]; then
    exit 0
fi

fstype="$(findmnt /sysroot --json --output FSTYPE | jq -r '.filesystems[0].fstype')"
if [[ "${fstype}" != "btrfs" ]]; then
    echo "Not using btrfs for /"
    touch /var/lib/.stamp
    exit 0
fi

compress="$(cat /etc/fstab | grep -vE "^#|^$" | awk '{ if ($2 == "/") { print $4 } }' | grep -c '^subvol=root,compress=zstd:1,ro$')"
if [[ "${compress}" != "1" ]]; then
    echo "Not using btrfs for /"
    touch /var/lib/.stamp
    exit 0
fi

rpm-ostree kargs --delete=rootflags=subvol=root --append=rootflags=subvol=root,compress=zstd:1
touch /var/lib/.stamp
```

This should only touch default configs. Maybe we could add that to bootc alongside the workaround that already exists.

Comment 11 Kamil Páral 2025-04-16 12:58:23 UTC
Common Issue description:
https://discussion.fedoraproject.org/t/common-issue/148562

Comment 12 Adam Williamson 2025-11-10 22:35:42 UTC
Still valid in current Rawhide.

Comment 13 Jiri Kortus 2026-03-03 18:16:48 UTC
I believe this is the same bug as described in https://bugzilla.redhat.com/show_bug.cgi?id=2439682#c3, CC'ing the reporter (Christopher Klooz).

Comment 14 David Tardon 2026-03-05 11:27:29 UTC
*** Bug 2379895 has been marked as a duplicate of this bug. ***

Comment 15 solver-our-1j 2026-03-07 18:05:22 UTC
I have to plead extreme ignorance on most Linux topics, so I apologize if ‘butting in’ here is inappropriate, but after needing to use a Live ISO (Bazzite KDE in my case) to restore the boot loader, I was dumped into the ‘emergency’ mode that needed a root password, when my original Fedora 42 Kenoite Atomic had “discouraged” me against setting up a root account/password.

I tried a number of possibilities in case maybe I had set one, to no avail, and now root is locked out.

I have tried a number of times to follow various guides to “use a Live ISO to replace root’s password”, also unsuccessfully.

Currently I am using rd.break and various commands to ‘remount /sysroot with read-write’ (<mount -oremount,rw /sysroot> being the latest) but I get the following output:

mount: /sysroot: fsconfig() failed: overlay: No changes allowed in reconfigure.
       dmesg(1) may have more information after failed mount system call.


Is there anything else I might be able to try? Either to fix this mounting issue or to ‘bypass’ the emergency mode to complete a regular boot into the OS?

Thanks!

Comment 16 solver-our-1j 2026-03-07 19:57:42 UTC
Nothing against this forum, but I’ve decided “I’m done” with my current quagmire and will simply overwrite my installation/OS. Those few files I most worry might be irreplaceable, I’ve copied onto another drive/partition for safekeeping.


Note You need to log in before you can comment on or make changes to this bug.