Bug 1174945 - Filesystem corruption after resume from suspend to disk (hibernation)
Summary: Filesystem corruption after resume from suspend to disk (hibernation)
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: dracut
Version: 21
Hardware: All
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: Harald Hoyer
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1198012 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-12-16 20:10 UTC by Przemysław Palacz
Modified: 2016-08-05 17:49 UTC (History)
13 users (show)

Fixed In Version: dracut-038-33.git20141216.fc21
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-03-24 18:46:51 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
Possible patch borrowed from openSUSE (1.31 KB, patch)
2014-12-16 20:10 UTC, Przemysław Palacz
no flags Details | Diff
Updated patch (1.25 KB, patch)
2014-12-17 09:31 UTC, Przemysław Palacz
no flags Details | Diff
dmesg (2.09 KB, text/plain)
2014-12-17 09:32 UTC, Przemysław Palacz
no flags Details
journal (25.01 KB, text/plain)
2014-12-17 09:33 UTC, Przemysław Palacz
no flags Details
dracut commit 733c71c adjusted for f21 branch (1.27 KB, patch)
2015-02-10 21:25 UTC, Leo Wolf
no flags Details | Diff

Description Przemysław Palacz 2014-12-16 20:10:06 UTC
Created attachment 969732 [details]
Possible patch borrowed from openSUSE

Description of problem:
After resuming from hibernation kernel starts to report failed attempts to insert new inode (__ext4_new_inode:1010: comm ... failed to insert inode ... doubly allocated?) which latter on results in filesystem corruption (as advertised in logs).

Version-Release number of selected component (if applicable):
dracut 038-31.git20141204.fc21
kernel 3.17.6-300.fc21

How reproducible:
Always

Additional info:
This might be the same bug as reported in SUSE's bugzilla:
https://bugzilla.suse.com/show_bug.cgi?id=906592

They are using older dracut version, same main version as in Fedora 20 although, I had no problems with hibernation on F20.

Going to rebuild dracut with attached patch now to check if it's working...

Comment 1 Przemysław Palacz 2014-12-17 09:30:57 UTC
Looks like it's working :)

Comment 2 Przemysław Palacz 2014-12-17 09:31:34 UTC
Created attachment 969997 [details]
Updated patch

Comment 3 Przemysław Palacz 2014-12-17 09:32:59 UTC
Created attachment 969999 [details]
dmesg

Comment 4 Przemysław Palacz 2014-12-17 09:33:53 UTC
Created attachment 970000 [details]
journal

Comment 5 Harald Hoyer 2014-12-17 10:48:35 UTC
(In reply to Przemysław Palacz from comment #3)
> Created attachment 969999 [details]
> dmesg

I guess, this is without the patch

Comment 6 Przemysław Palacz 2014-12-17 19:25:29 UTC
Yes, sorry... forgot to attach it to the first comment. After applying the patch hibernation does not cause filesystem errors any more.

Comment 7 Leo Wolf 2015-02-10 21:25:46 UTC
Created attachment 990261 [details]
dracut commit 733c71c adjusted for f21 branch

Cherrypicking dracut commit 733c71c "resume: make use of systemd-hibernate-resume, if existant" works for me.

Comment 8 Przemysław Palacz 2015-02-16 18:08:41 UTC
Thanks for the info.
I actually took the easy way out and downloaded dracut 41 rpms from koji which also works very nicely on my F21.

Comment 9 Till Maas 2015-03-06 10:25:22 UTC
*** Bug 1198012 has been marked as a duplicate of this bug. ***

Comment 10 Till Maas 2015-03-06 10:37:52 UTC
attachment 990261 [details] fixes this for me as well. I also added a resume kernel commandline parameter to grub, not sure if this is necessary.

Comment 11 Fedora Update System 2015-03-06 12:23:18 UTC
dracut-038-33.git20141216.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/dracut-038-33.git20141216.fc21

Comment 12 Fedora Update System 2015-03-09 08:34:53 UTC
Package dracut-038-33.git20141216.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing dracut-038-33.git20141216.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-3484/dracut-038-33.git20141216.fc21
then log in and leave karma (feedback).

Comment 13 Andrew J. Schorr 2015-03-09 13:05:16 UTC
I installed these rpms on my f21 laptop and ran "dracut -f" to rebuild the initramfs, but hibernate/resume no longer works.  When I hibernate, it appears to work, and the system powers down.  When I then open the lid, it does power on automatically, but it does not resume -- it goes through a regular boot sequence, and I have to login again.  Any ideas on how to troubleshoot?  I searched for error messages in journalctl, and I see this:

sh-4.3# journalctl -b | egrep -i 'susp|resu|hib'
Mar 09 04:44:18 ajs-t530 kernel: PM: Hibernation image not present or could not be loaded.
Mar 09 08:44:38 ajs-t530 swapon[650]: swapon: /dev/mapper/fedora-swap: software suspend data detected. Rewriting the swap signature.

Regards,
Andy

Comment 14 Przemysław Palacz 2015-03-09 20:24:02 UTC
I think I had the same problem at some point.
I use quite? standard partition scheme (UEFI, swap file), no LVM but still.
Check if dracut detects your swap partition during image generation:
dracut -fv | grep resume
You can also check if swap is added to your current image with:
lsinitrd -f etc/cmdline.d/95resume.conf

If resume module isn't executed during image generation (or missing 95resume.conf) then you have to add your swap manually (man dracut.conf).

Comment 15 Andrew J. Schorr 2015-03-09 21:49:17 UTC
Hi, thanks for the tips, but I don't think that explains it:

[root@ajs-t530 ~]# lsinitrd -f etc/cmdline.d/95resume.conf
 resume=/dev/mapper/fedora-swap
[root@ajs-t530 ~]# dracut -fv > /tmp/dracut.out 2>&1 && echo YES
YES
[root@ajs-t530 ~]# grep resume /tmp/dracut.out 
*** Including module: resume ***

Thanks!

-Andy

Comment 16 Przemysław Palacz 2015-03-24 18:46:51 UTC
Update fixes the problem, thank you :)

Comment 17 Fedora Update System 2015-03-29 04:51:02 UTC
dracut-038-33.git20141216.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 18 Andrew J. Schorr 2015-03-29 15:52:10 UTC
Hi,

I'm afraid hibernate/resume still doesn't work for me.  I followed the following steps:

- Fresh install of Fedora 21 onto my laptop using the Fedora 21/x86_64 live workstation usb stick.  This installs the old rpm versions from the released ISO.

- Then I updated dracut by running "yum -y update dracut".  I wanted to make sure that dracut was updated before the kernel so that the new initramfs generated would use the new version of dracut.

- I updated all the other software: "yum -y update"

- I also installed some other software, in case it matters, including XFCE, tcsh, tigervnc, etc.  And I disabled SElinux.

- I rebooted into the new 3.19.2 kernel and logged into an XFCE desktop.  I then requested the system to hibernate from the XFCE panel widget.

- I closed the lid and reopened it, and the system booted up to a login screen.  It did not resume properly.

What am I doing wrong?

Regards,
Andy

Comment 19 Till Maas 2015-03-29 17:41:04 UTC
(In reply to Andrew J. Schorr from comment #18)

> What am I doing wrong?

Your kernel commandline needs to contain a proper resume= parameter.

Comment 20 Andrew J. Schorr 2015-03-29 19:45:52 UTC
I have a vanilla configuration, as far as I know.  It used to work fine.  Is this supposed to be in /boot/grub2/grub.cfg?  I don't see it there:

[schorr@ajs-r705 grub2]$ grep resume /boot/grub2/grub.cfg
[schorr@ajs-r705 grub2]$ cat /etc/sysconfig/grub 
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.lvm.lv=vg_sys/f21 rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
[schorr@ajs-r705 grub2]$ cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-3.19.2-201.fc21.x86_64 root=/dev/mapper/vg_sys-f21 ro rd.lvm.lv=vg_sys/f21 rhgb quiet LANG=en_US.UTF-8

Is there somehow a configuration problem here?  I did not touch anything related to grub2.  Shouldn't this work out of the box?  It always has for me in the past...

Thanks,
Andy

Comment 21 Till Maas 2015-03-29 20:45:51 UTC
(In reply to Andrew J. Schorr from comment #20)

> Is there somehow a configuration problem here?  I did not touch anything
> related to grub2.  Shouldn't this work out of the box?  It always has for me
> in the past...

unfortunately dracut/systemd are not properly integrated into Fedora 21, therefore you need to set the resume option manually.

Comment 22 Andrew J. Schorr 2015-03-29 21:37:20 UTC
Thanks for the help, but this doesn't seem quite right.

I just wiped my laptop again and did a vanilla Fedora 21 installation from a USB stick with the distributed ISO.  So all the software is from the original F21 release.

I logged in, su'ed to root, and typed "systemctl hibernate".  I then closed the lid.  After opening the lid, the system resumed properly.

Note that the word "resume" does not appear anywhere in /etc/default/grub or in /boot/grub2/grub.cfg or in /proc/cmdline.

I did this twice, and it is repeatable.  The vanilla F21 hibernate/resume works out of the box before updating any software.

I then ran "yum update dracut".  After this, hibernate/resume still works.  

I then rebuilt the initramfs by running "dracut --force".  I rebooted the machine.  I then ran "systemctl hibernate", but when I power back up, resume no longer works.  It does a fresh boot.

Does this behavior make sense?  It surely seems as if something in the new dracut package is breaking normal hibernate/resume behavior.  I think there's a regression here; am I confused?

Regards,
Andy

Comment 23 Andrew J. Schorr 2015-03-29 21:44:19 UTC
And yes, you are right -- adding "resume=<swap device>" to /etc/default/grub in the GRUB_CMDLINE_LINUX value and then running grub2-mkconfig /boot/grub2/grub.cfg does fix the problem.  But this should not be necessary.  The original F21 version of dracut did not require this.  I don't understand the boot process well enough to troubleshoot what's going wrong, but I think there's a regression here.

Regards,
Andy

Comment 24 Till Maas 2015-03-30 06:33:18 UTC
(In reply to Andrew J. Schorr from comment #23)

> /boot/grub2/grub.cfg does fix the problem.  But this should not be
> necessary.  The original F21 version of dracut did not require this.  I
> don't understand the boot process well enough to troubleshoot what's going
> wrong, but I think there's a regression here.

yes, the requirement cam with the update fixing this, because without the filesystem will be corrupted during resume, because there was an fsck check performed. To fix this it seems systemd needed to be used, but it only accepts a resume parameter from the kernel commandline.

Comment 25 Andrew J. Schorr 2015-03-30 13:01:53 UTC
Ah, now I understand.  I think you are saying that Anaconda needs to be patched to add "resume=<swap device>" to the boot loader configuration files.  Has an Anaconda bug been opened for this issue?

Regards,
Andy

Comment 26 Adam Pribyl 2015-04-24 19:14:57 UTC
Follow up bug https://bugzilla.redhat.com/show_bug.cgi?id=1206912

Comment 27 Garrett Mitchener 2016-08-05 17:49:41 UTC
I don't know if I'm seeing the same problem or not.  I've been trying to set up Deja Dup to put backups on a USB hard drive, and I keep getting corrupted backups and errors like those in the description:

[ 7663.363525] EXT4-fs error (device sdk1): __ext4_new_inode:1066: comm python2: failed to insert inode 17301505: doubly allocated?
[ 7693.426957] EXT4-fs error (device sdk1): __ext4_new_inode:1066: comm python2: failed to insert inode 17301506: doubly allocated?

I'm using suspend but not hibernate. I thought the hard drive was failing, so I bought a new one, but it's giving me the same error messages.

This is a Fedora 24 desktop machine, Dell Precision 3620

kernel-4.6.4-301.fc24.x86_64
systemd-229-9.fc24.x86_64
dracut-044-20.fc24.x86_64


Note You need to log in before you can comment on or make changes to this bug.