Bug 981841 - Unable to hibernate properly with multiple encrypted volumes due to ordering changes [NEEDINFO]
Unable to hibernate properly with multiple encrypted volumes due to ordering ...
Status: CLOSED INSUFFICIENT_DATA
Product: Fedora
Classification: Fedora
Component: kernel (Show other bugs)
20
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: dracut-maint
Fedora Extras Quality Assurance
: Regression
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-06 04:23 EDT by Robert Hancock
Modified: 2015-04-28 14:23 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-04-28 14:23:30 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
kernel-team: needinfo?


Attachments (Terms of Use)

  None (edit)
Description Robert Hancock 2013-07-06 04:23:18 EDT
Description of problem:
dracut appears to be setting an incorrect device in /sys/power/resume on boot, seemingly resulting in hibernate not working - I get a "PM: Cannot find swap device, try swapon -a" error from the kernel.

This computer is set up with an encrypted /home and swap partition (root is not encrypted). The device file that ends up being set in /sys/power/resume is for /home, not the swap partition.

This setup was previously working in Fedora 18 but stopped working after upgrading.

Version-Release number of selected component (if applicable):
dracut-029-1.fc19.x86_64

How reproducible:
Every time

Steps to Reproduce:
1. Use encrypted swap and /home with unencrypted root FS
2. Try to hibernate
3.

Actual results:
Hibernate fails

Expected results:
Hibernate and resume succeeds

Additional info:
cat /sys/power/resume 
253:2

cat /proc/partitions 
major minor  #blocks  name

   8        0  488386584 sda
   8        1   26214400 sda1
   8        2     102400 sda2
   8        3  195354624 sda3
   8        4          1 sda4
   8        5     512000 sda5
   8        6  266200064 sda6
  11        0    1048575 sr0
 253        0    6291456 dm-0
 253        1   52428800 dm-1
 253        2  207454208 dm-2
 253        3    6289408 dm-3
 253        4  207452160 dm-4

ll /dev/mapper/
total 0
crw-------. 1 root root 10, 236 Jul  6 02:08 control
lrwxrwxrwx. 1 root root       7 Jul  6 02:08 luks-21197db5-29e6-44a3-8d0e-7bb0678dd950 -> ../dm-3
lrwxrwxrwx. 1 root root       7 Jul  6 02:08 luks-76947c1b-9cb4-4c1e-aa7e-5a41abbc3a74 -> ../dm-4
lrwxrwxrwx. 1 root root       7 Jul  6 02:08 vg_sandy-lv_home -> ../dm-2
lrwxrwxrwx. 1 root root       7 Jul  6 02:08 vg_sandy-lv_root -> ../dm-1
lrwxrwxrwx. 1 root root       7 Jul  6 02:08 vg_sandy-lv_swap -> ../dm-0

cat /etc/fstab
...
/dev/mapper/luks-21197db5-29e6-44a3-8d0e-7bb0678dd950 swap                    swap    defaults        0 0

cat /etc/default/grub 
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Fedora"
GRUB_DEFAULT=saved
GRUB_CMDLINE_LINUX="rd.md=0 vconsole.keymap=us rd.lvm.lv=vg_sandy/lv_swap rd.luks.uuid=21197db5-29e6-44a3-8d0e-7bb0678dd950 rd.lvm.lv=vg_sandy/lv_root LANG=en_US.UTF-8 resume=/dev/mapper/luks-21197db5-29e6-44a3-8d0e-7bb0678dd950 rhgb quiet pcie_aspm=force"
#GRUB_THEME="/boot/grub2/themes/system/theme.txt"

The correct device is dm-3 but for some reason dm-2 is the one pointed to.

Also see these errors on bootup after entering the passphrase:

dracut-initqueue[213]: ln: failed to create symbolic link '/dev/resume': File exists
dracut-initqueue[213]: rm: cannot remove '/lib/dracut/hooks/initqueue/settled/resume.sh': No such file or directory
dracut-initqueue[213]: rm: cannot remove '/lib/dracut/hooks/initqueue/timeout/resume.sh': No such file or directory
Comment 1 Robert Hancock 2013-07-06 20:59:19 EDT
From what I can tell, the kernel is storing the resume= device from the kernel command line on boot. Later on, when the kernel gets told to attempt resuming, the device name is converted into a major/minor pair. It appears that this device is the one the kernel will later attempt to hibernate to when required.

The resume part with dracut seems reasonable - the encrypted volume gets activated and a resume is attempted from the device matching the one passed on the command line. At this point the swap encrypted volume is on dm-3 and the home volume (if it's listed on the boot command line) is on dm-4.

The problem comes later, as for some reason those cryptsetup mappings get torn down and recreated when the "main" boot process starts. The lines in /etc/fstab for the /home partition and swap partition were in the opposite order from the order in the dracut command line. This means that now the swap encrypted volume is dm-4 and the home volume is dm-3. But the kernel still remembers that the resume device was minor number 3 and so when you hibernate it tries to write the image to the home partition, which fails as it's not a swap partition.

Switching the order of lines in /etc/fstab so that the swap partition comes before the home partition fixed the problem in this case. But this all seems very fragile. Arguably it would be better if the kernel used the stored device filename when hibernating, rather than the major/minor numbers it remembered from the attempted resume, as that seems more likely to have a stable mapping to the actual partition/device. But I'm also not sure why the cryptsetup mappings are being torn down and recreated. I would think if that didn't happen, it would avoid this problem, and speed up the boot as well.
Comment 2 Harald Hoyer 2013-07-08 05:08:55 EDT
(In reply to Robert Hancock from comment #0)
> Description of problem:
> dracut appears to be setting an incorrect device in /sys/power/resume on
> boot, seemingly resulting in hibernate not working - I get a "PM: Cannot
> find swap device, try swapon -a" error from the kernel.
> 

dracut does not set /sys/power/resume, if you are not resuming. Writing to /sys/power/resume does result in resuming.

But if dracut encounters a partition with the ID_FS_TYPE=suspend or ID_FS_TYPE=swsuspend, it will echo the major:minor to /sys/power/resume.
Comment 3 Harald Hoyer 2013-07-08 05:10:48 EDT
(In reply to Robert Hancock from comment #1)
> From what I can tell, the kernel is storing the resume= device from the
> kernel command line on boot. Later on, when the kernel gets told to attempt
> resuming, the device name is converted into a major/minor pair. It appears
> that this device is the one the kernel will later attempt to hibernate to
> when required.

So, you mean:

1. cold boot
2. hibernate -> works
3. resume
4. hibernate -> fails, because the kernel wants to resuse the major/minor?

I can't believe that.
Comment 4 Robert Hancock 2013-07-08 16:55:27 EDT
(In reply to Harald Hoyer from comment #2)
> (In reply to Robert Hancock from comment #0)
> > Description of problem:
> > dracut appears to be setting an incorrect device in /sys/power/resume on
> > boot, seemingly resulting in hibernate not working - I get a "PM: Cannot
> > find swap device, try swapon -a" error from the kernel.
> > 
> 
> dracut does not set /sys/power/resume, if you are not resuming. Writing to
> /sys/power/resume does result in resuming.
> 
> But if dracut encounters a partition with the ID_FS_TYPE=suspend or
> ID_FS_TYPE=swsuspend, it will echo the major:minor to /sys/power/resume.

I don't think I have any partition labelled as suspend or swsuspend (I'm not sure what would get the swap partition into a state so that it would be so labelled). But I do have an explicit resume= entry on the kernel command line, so it does try to resume during the dracut boot sequence.

(In reply to Harald Hoyer from comment #3)
> (In reply to Robert Hancock from comment #1)
> > From what I can tell, the kernel is storing the resume= device from the
> > kernel command line on boot. Later on, when the kernel gets told to attempt
> > resuming, the device name is converted into a major/minor pair. It appears
> > that this device is the one the kernel will later attempt to hibernate to
> > when required.
> 
> So, you mean:
> 
> 1. cold boot
> 2. hibernate -> works
> 3. resume
> 4. hibernate -> fails, because the kernel wants to resuse the major/minor?
> 
> I can't believe that.

No, not that complicated. Only one boot is involved.

1. cold boot
2. dm-crypt mappings get set up in order as listed on command line
3. dracut sets /sys/power/resume attempting resume - fails, no hibernate image in swap, but major-minor pair of resume device is stored by kernel
4. dm-crypt mappings torn down
5. dm-crypt mappings set up in order as listed in /etc/fstab
6. user hibernate - fails, stored major/minor is no longer a swap partition

If, after booting up, I echoed the proper major/minor numbers for the resume partition into /sys/power/resume and hibernated, then on powering up it resumed fine. So it's the hibernate portion that's broken.
Comment 5 Harald Hoyer 2013-07-09 07:26:52 EDT
(In reply to Robert Hancock from comment #4)
> If, after booting up, I echoed the proper major/minor numbers for the resume
> partition into /sys/power/resume and hibernated, then on powering up it
> resumed fine. So it's the hibernate portion that's broken.

Yeah, but dracut does not hibernate. It only resumes.
Comment 6 Robert Hancock 2013-07-09 23:25:27 EDT
(In reply to Harald Hoyer from comment #5)
> Yeah, but dracut does not hibernate. It only resumes.

It's true, the hibernate portion is done (and gets broken) after dracut is all finished. Not sure what component this should be assigned to then - it seems like we shouldn't be tearing down the encrypted volumes just to create them, and we shouldn't be storing the resume device in the kernel in as fragile of a fashion.
Comment 7 Robert Hancock 2013-08-04 01:43:47 EDT
OK, after installing some updates this problem is happening again. Apparently the order of the rd.luks.uuid lines on the kernel command line isn't reliably controlling the order in which the devices are activated in dracut, which means they don't reliably match the order in which they're listed in fstab. Switching the lines in fstab avoids the problem for now but who knows if some future change will switch them around again.

Since there seems to be no reliable workaround, something needs to be rethought here I think. I can only presume this doesn't commonly happen since default setups generally encrypt a single drive either entirely or not at all.
Comment 8 Harald Hoyer 2013-08-05 04:46:36 EDT
No, the lines on the kernel command line do not specify the order of activation. Activation happens with hotplug, as soon as the underlying device is discovered.
Comment 9 Robert Hancock 2013-09-28 02:40:18 EDT
Am going to reassign this to kernel at this point, as this doesn't really seem like a dracut problem. It seems like it all kind of comes back to that major:minor number pair that the kernel holds onto from bootup. It seems a bit naive to expect that device ID to still be valid by the time the user finally triggers a hibernate. In setups like this with multiple encrypted volumes, it frequently is not.
Comment 10 Justin M. Forbes 2014-01-03 17:05:34 EST
*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 19 kernel bugs.

Fedora 19 has now been rebased to 3.12.6-200.fc19.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you have moved on to Fedora 20, and are still experiencing this issue, please change the version to Fedora 20.

If you experience different issues, please open a new bug report for those.
Comment 11 Robert Hancock 2014-01-04 02:51:17 EST
Changing version to F20 since as far as I can tell, nothing has really changed in this regard.
Comment 12 Justin M. Forbes 2014-02-24 08:53:02 EST
*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 20 kernel bugs.

Fedora 20 has now been rebased to 3.13.4-200.fc20.  Please test this kernel update and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you experience different issues, please open a new bug report for those.
Comment 13 Robert Hancock 2014-02-24 22:32:54 EST
As far as I can tell there has been no change in this behavior.
Comment 14 Robert Hancock 2014-02-24 22:49:54 EST
Reported to kernel mailing list here:

https://lkml.org/lkml/2014/2/24/762
Comment 15 Justin M. Forbes 2014-05-21 15:37:45 EDT
*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 20 kernel bugs.

Fedora 20 has now been rebased to 3.14.4-200.fc20.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you experience different issues, please open a new bug report for those.
Comment 16 Robert Hancock 2014-05-21 21:03:54 EDT
Still no change as far as I'm aware.
Comment 17 Justin M. Forbes 2014-11-13 10:56:25 EST
*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 20 kernel bugs.

Fedora 20 has now been rebased to 3.17.2-200.fc20.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you have moved on to Fedora 21, and are still experiencing this issue, please change the version to Fedora 21.

If you experience different issues, please open a new bug report for those.
Comment 18 Robert Hancock 2014-11-13 21:40:09 EST
Not aware of any change in this behavior.
Comment 19 collura 2015-01-16 17:42:30 EST
got this in fc21 install:

  dracut-initqueue ln failed to create symbolic link '/dev/resume' file exists

so still seems to be there as of 

  kernel-3.17.7-300.fc21.86_64
  dracut-038-32.git20141216.fc21.x86_64
Comment 20 Fedora Kernel Team 2015-02-24 11:22:29 EST
*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 20 kernel bugs.

Fedora 20 has now been rebased to 3.18.7-100.fc20.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you have moved on to Fedora 21, and are still experiencing this issue, please change the version to Fedora 21.

If you experience different issues, please open a new bug report for those.
Comment 21 Fedora Kernel Team 2015-04-28 14:23:30 EDT
*********** MASS BUG UPDATE **************
This bug is being closed with INSUFFICIENT_DATA as there has not been a response in over 4 weeks. If you are still experiencing this issue, please reopen and attach the relevant data from the latest kernel you are running and any data that might have been requested previously.

Note You need to log in before you can comment on or make changes to this bug.