Created attachment 1459838 [details] SOSreport captured from inside Emergency Shell at time of problem Description of problem: After updating the kernel from 4.17.5-200.fc28.x86_64 to 4.17.6-200.fc28.x86_64 and rebooting, the system drops to the Emergency Shell. The root filesystem (and several others) are formatted as ext4 . The journald logs show errors suggesting that these ext4 filesystems are corrupted (see below). However, no errors are reported after rebooting into the last known-good kernel and running "fsck.ext4 -f -C0" on said filesystems. This problem appears on two different machines (different hardware, different vendors). Version-Release number of selected component (if applicable): Fedora 28, with kernel 4.17.6-200.fc28.x86_64 How reproducible: Reproducible on every reboot with 4.17.6-200.fc28.x86_64 . Steps to Reproduce: 1. Install Fedora 28 with an ext4-formatted root filesystem. 2. Run "dnf upgrade" such that the kernel 4.17.6-200.fc28.x86_64 is installed. 3. Reboot. Actual results: The Emergency Shell appears; the system does not finish booting. Expected results: The system boots as usual. Additional info: Below are selected highlights from the journald log: root@grout:~# journalctl -l -b -1 | grep -i -E 'EXT4-fs.+(ext4_check_descriptors|group descriptors)|system call failed|emergency' Jul 18 19:15:44 grout.halibutdepot.org kernel: EXT4-fs (dm-4): ext4_check_descriptors: Block bitmap for group 0 overlaps block group descriptors Jul 18 19:15:44 grout.halibutdepot.org kernel: EXT4-fs (dm-4): group descriptors corrupted! Jul 18 19:15:44 grout.halibutdepot.org mount[920]: mount: /opt/nobackup: mount(2) system call failed: Structure needs cleaning. Jul 18 19:15:44 grout.halibutdepot.org systemd[1]: Started Emergency Shell. Jul 18 19:15:44 grout.halibutdepot.org audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=emergency comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Jul 18 19:15:44 grout.halibutdepot.org systemd[1]: Reached target Emergency Mode. Jul 18 19:15:45 grout.halibutdepot.org kernel: EXT4-fs (dm-5): ext4_check_descriptors: Block bitmap for group 0 overlaps block group descriptors Jul 18 19:15:45 grout.halibutdepot.org kernel: EXT4-fs (dm-5): group descriptors corrupted! Jul 18 19:15:45 grout.halibutdepot.org mount[973]: mount: /var/lib/libvirt: mount(2) system call failed: Structure needs cleaning. Jul 18 19:20:27 grout.halibutdepot.org systemd[1]: Stopped target Emergency Mode. Jul 18 19:20:27 grout.halibutdepot.org systemd[1]: Stopping Emergency Shell... Jul 18 19:20:27 grout.halibutdepot.org systemd[1]: Stopped Emergency Shell. Jul 18 19:20:27 grout.halibutdepot.org audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=emergency comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Same for me on Fedora release 27 after updating kernel. Disk won't mount with kernel-4.17.6-100.fc27.x86_64 /var/log/messages says when mounting: kernel: EXT4-fs (dm-2): ext4_check_descriptors: Block bitmap for group 0 overlaps block group descriptors kernel: EXT4-fs (dm-2): group descriptors corrupted! Mounts OK with kernel-4.16.16-200.fc27.x86_64 Is it this: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=903838 ? See Message #26 in the above: https://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.git/commit/?id=44de022c4382541cebdd6de4465d1f4f465ff1dd
Thanks, Phil Smith, for the bug links in comment #1 . Yes, I believe that this is the same bug. (0) Debian backported a fix for this bug on 2018-07-17. Can we have the same fix applied as well? (1) I can reliably duplicate this problem on kernel 4.17.6-200.fc28.x86_64: Create an ext4 filesystem with the "meta_bg" filesystem feature turned on, and the this newest kernel refuses to mount the filesystem. See output (1) below. (2) Likewise, this problem *does not* appear on the last known-good kernel 4.17.5-200.fc28.x86_64 when following the same steps. See output (2) below. (3) It doesn't look like we can work around this bug merely by turning off the "meta_bg" feature in situ. tune2fs refuses to do so, saying "Clearing filesystem feature 'meta_bg' not supported.". ======== OUTPUT FROM TEST (1) ======== -------- (a) Using kernel 4.17.6-200.fc28.x86_64, create a new ext4 filesystem with the bug-tickling "meta_bg" feature enabled. The filesystem cannot be mounted. -------- # cat /etc/redhat-release Fedora release 28 (Twenty Eight) # uname -r 4.17.6-200.fc28.x86_64 # lvcreate --size 1g --name demo fedora_localhost-live Logical volume "demo" created. # mkfs.ext4 -O ^resize_inode,+meta_bg /dev/fedora_localhost-live/demo mke2fs 1.44.2 (14-May-2018) Creating filesystem with 262144 4k blocks and 65536 inodes Filesystem UUID: dc048f73-cd2a-4116-aead-7f2f7bac35a7 Superblock backups stored on blocks: 32768, 98304, 163840, 229376 Allocating group tables: done Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done # mount /dev/fedora_localhost-live/demo /mnt mount: /mnt: mount(2) system call failed: Structure needs cleaning. # journalctl -l --since=-1min -- Logs begin at Fri 2018-05-11 22:58:12 EDT, end at Fri 2018-07-20 10:35:23 EDT. -- Jul 20 10:35:23 virtual5.halibutdepot.org kernel: EXT4-fs (dm-2): ext4_check_descriptors: Block bitmap for group 0 overlaps block group descriptors Jul 20 10:35:23 virtual5.halibutdepot.org kernel: EXT4-fs (dm-2): group descriptors corrupted! -------- (b) Using the same kernel 4.17.6-200.fc28.x86_64, create an ext4 filesystem *without* the bug-tickling "meta_bg" filesystem feature. The filesystem now mounts successfully. -------- # wipefs -a /dev/fedora_localhost-live/demo /dev/fedora_localhost-live/demo: 2 bytes were erased at offset 0x00000438 (ext4): 53 ef # mkfs.ext4 /dev/fedora_localhost-live/demo mke2fs 1.44.2 (14-May-2018) Creating filesystem with 262144 4k blocks and 65536 inodes Filesystem UUID: 876401b0-f825-41b8-b286-e263cadf28bc Superblock backups stored on blocks: 32768, 98304, 163840, 229376 Allocating group tables: done Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done # mount /dev/fedora_localhost-live/demo /mnt # echo $? 0 ======== OUTPUT FROM TEST (2) ======== On the last known-good kernel, 4.17.5-200.fc28.x86_64, create an ext4 filesystem with the "meta_bg" filesystem feature enabled. This kernel mounts the filesystem successfully. # uname -r 4.17.5-200.fc28.x86_64 # mkfs.ext4 -O ^resize_inode,+meta_bg /dev/vg_paul/demo mke2fs 1.44.2 (14-May-2018) Discarding device blocks: done Creating filesystem with 262144 4k blocks and 65536 inodes Filesystem UUID: 1c8c93db-5486-4eec-bbe8-e781d507738c Superblock backups stored on blocks: 32768, 98304, 163840, 229376 Allocating group tables: done Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done # mount /dev/vg_paul/demo /mnt # echo $? 0
Thanks for the detailed information. I see the patch linked is still in the Maintainer's dev branch. I don't see it in Linus' tree yet though. I know Debian picked up the patch but I'm a little hesitant to pick up FS patches too quickly because they have the potential for data corruption. This wouldn't be picked up in a build before 4.17.9 comes out anyway so let's double check where the patch is at when 4.17.9 comes out (probably sometime on Monday).
kernel-4.17.9-200.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-ab23a7ca88
kernel-4.17.9-100.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-5b59d78d1c
Thanks! kernel-4.17.9-200.fc28.x86_64 is confirmed to resolve this issue. Positive feedback left on bodhi.
Thank you all. The new kernel-4.17.9-100.fc27.x86_64 resolved the issue for me. Positive feedback left on bodhi.
kernel-4.17.9-200.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-ab23a7ca88
kernel-4.17.9-100.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-5b59d78d1c
kernel-4.17.9-100.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.
kernel-4.17.9-200.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.