Description of problem: The journal of an XFS root fs gets recovered after every reboot or shutdown. Dmesg shows: Starting recovery (logdev: internal) Ending recovery (logdev: internal) This affects only the root partition, other XFS fs are fine. This happens on three different Fedora machines. Version-Release number of selected component (if applicable): Fedora 27 Kernel 4.14.12 How reproducible: Always. Install F27, format the root fs with XFS, reboot, check dmesg. Steps to Reproduce: 1. Install F27 2. Format the root partition with XFS. 3. Reboot. The XFS journal is corrupted and gets recovered. Can be repeated endlessly. Actual results: The XFS journal is corrupt and gets recovered on next start. Expected results: The XFS journal is clean, and there is no recovery on bootup. Additional info:
Note, the journal is not /corrupt/ it is dirty, which is expected when the filesystem is not cleanly unmounted. The kernel is doing the right thing - recovering a filesystem which was not cleanly unmounted. systemd controls whether that is so, so moving this bug. -Eric
While debugging, I noticed that the problem is solved by simply placing a script in /usr/lib/systemd/systemd-shutdown that mounts the root fs rw and then ro: #!/bin/sh mount -o remount,rw / mount -o remount,ro / Now, the root fs is clean on every reboot/shutdown. Seems like something in the systemd reboot/shutdown process got messed up.
In fact, creating the aforementioned debug.sh script in /usr/lib/systemd/systemd-shutdown with only that lsingle ine is suffice to resolve the problem: mount -o remount,ro / Looks like systemd fails to remount the root fs read-only when shutting down. [htd@chiara ~]$ rpm -qa | grep -i systemd systemd-libs-234-9.fc27.x86_64 systemd-udev-234-9.fc27.x86_64 python3-systemd-234-3.fc27.x86_64 systemd-pam-234-9.fc27.x86_64 rpm-plugin-systemd-inhibit-4.14.0-2.fc27.x86_64 python-systemd-doc-234-3.fc27.x86_64 systemd-bootchart-233-1.fc27.x86_64 systemd-234-9.fc27.x86_64
Please see my report of ext4 journal recovery on reboot, Bug 1560149. Appears to be same behavior.
And refer to Bug 1508984. Seems like several disparate installations are observing this "dirty" journal.
Yes, the root filesystem is not getting cleanly unmounted on a reboot. Systemd folks, any plans to address this bug?
Can try to reproduce with plymouth disabled (boot with plymouth.enable=0 on kernel command line)? IIRC, I've seen similar issues in the past and they were related to plymouth keeping open files on root filesystem while being excluded from killing at shutdown.
On my systems, there is no plymouth installed. This is not plymouth-related.
Installed plymouth, the fs is still not unmounted cleanly. Disabled it, with the same result. FWIW: the workaround I mentioned above works both with and without plymouth installed.
Bug #1508984 claims that after updating to systemd-234-10.git5f8984e.fc27 the problem went away. Can you confirm?
I've updated today, to the same systemid version as mentioned in Comment 10. The problem persists.
system-d, obviously, not systemid. Sorry! Anyway, the problem is still there.
FWIW: Tried on my Laptop today: root@kiera ~]# rpm -qa | grep -i -e ^systemd-[0-9]+* systemd-234-10.git5f8984e.fc27.x86_64 [root@kiera ~]# uname -a Linux kiera.fritha.org 4.16.4 #1 SMP PREEMPT Tue Apr 24 09:59:18 CEST 2018 x86_64 x86_64 x86_64 GNU/Linux [htd@kiera ~]$ dmesg | grep -i xfs [ 9.155378] SGI XFS with ACLs, security attributes, no debug enabled [ 9.160902] XFS (sda2): Mounting V4 Filesystem [ 9.217017] XFS (sda2): Starting recovery (logdev: internal) [ 9.262375] XFS (sda2): Ending recovery (logdev: internal) Still not cleanly unmounted.
This message is a reminder that Fedora 27 is nearing its end of life. On 2018-Nov-30 Fedora will stop maintaining and issuing updates for Fedora 27. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '27'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 27 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 27 changed to end-of-life (EOL) status on 2018-11-30. Fedora 27 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.