Bug 1053045 - kdump.service complains about using encrypted rootfs as dump target even when crash ramdisk is capable of unlocking LUKS partitions
Summary: kdump.service complains about using encrypted rootfs as dump target even when...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: kexec-tools
Version: 7.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: rc
: 7.0
Assignee: arthur
QA Contact: Qiao Zhao
URL:
Whiteboard:
Depends On:
Blocks: 1077625
TreeView+ depends on / blocked
 
Reported: 2014-01-14 15:45 UTC by Martin
Modified: 2016-01-18 20:48 UTC (History)
11 users (show)

Fixed In Version: kexec-tools-2.0.4-25
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1077625 (view as bug list)
Environment:
Last Closed: 2014-06-13 12:05:53 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Martin 2014-01-14 15:45:49 UTC
Description of problem:
kdump tries to use encrypted rootfs as a default dump target

I propose, kdump should be more intelligent by default and automatically use /boot partition when /var/crash is located on encrypted partition.

Version-Release number of selected component (if applicable):
kexec-tools-2.0.4-19.el7.x86_64

How reproducible:
always

Steps to Reproduce:
Enable and run kdump.service

Actual results:
# systemctl status kdump.service 
kdump.service - Crash recovery kernel arming
   Loaded: loaded (/usr/lib/systemd/system/kdump.service; enabled)
   Active: failed (Result: exit-code) since Tue 2014-01-14 16:32:15 CET; 5min ago
  Process: 1291 ExecStart=/usr/bin/kdumpctl start (code=exited, status=1/FAILURE)
 Main PID: 1291 (code=exited, status=1/FAILURE)
   CGroup: /system.slice/kdump.service

Jan 14 16:32:13 martin-holec kdumpctl[1291]: No kdump initial ramdisk found.
Jan 14 16:32:13 martin-holec kdumpctl[1291]: Rebuilding /boot/initramfs-3.10.0-67.el7.x86_64kdump.img
Jan 14 16:32:15 martin-holec kdumpctl[1291]: Device /dev/sda2 is encrypted, can not be used in kdump.
Jan 14 16:32:15 martin-holec kdumpctl[1291]: Can not save vmcore to target device /dev/mapper/rhel_dhcp131--108-root.
Jan 14 16:32:15 martin-holec kdumpctl[1291]: mkdumprd: failed to make kdump initrd
Jan 14 16:32:15 martin-holec kdumpctl[1291]: Starting kdump: [FAILED]
Jan 14 16:32:15 martin-holec systemd[1]: kdump.service: main process exited, code=exited, status=1/FAILURE
Jan 14 16:32:15 martin-holec systemd[1]: Failed to start Crash recovery kernel arming.
Jan 14 16:32:15 martin-holec systemd[1]: Unit kdump.service entered failed state.

Expected results:
/boot is used as default crash dump target when using encrypted rootfs

Comment 2 Martin 2014-01-14 17:37:26 UTC
Additional informations:
I added "xfs /dev/sda1" to /etc/kdump.conf to set crash dump destination to /boot partition and started kdump.service (successfully).

Results:
1) When kernel panic occurred, kexec run crash kernel and ramdisk and asked for LUKS password for rootfs.

2) After I provided password, kdump writed dump file and rebooted. But I can't find any files on /boot/var/crash/ nor /var/crash/.

Discussion:

1) Why kdump.service complains about encrypted /dev/sda2 when crash ramdisk can unlock rootfs on LUKS?

2) Why kdump doesn't write crash dump to configured location? If this is a bug, not misconfiguration, I can report separate bug report.

Comment 3 Vivek Goyal 2014-01-14 18:32:24 UTC
So far we did not allow dump to encrypted partition because it required manually entering the password. And we don't expect that after the crash somebody will be sitting there to enter the password to unlock the encrypted disk. We want to quickly dump and reboot back to first kernel.

Comment 4 Vivek Goyal 2014-01-14 18:35:03 UTC
(In reply to Martin Holec from comment #0)
> Description of problem:
> kdump tries to use encrypted rootfs as a default dump target

May be. But I don't feel too excited about choosing /boot as default dump destination. vmcores can be big and it can completely fill up /boot easily. I would rather let user specify to use /boot as dump partition.

Comment 5 Martin 2014-01-14 19:26:55 UTC
(In reply to Vivek Goyal from comment #3)
> So far we did not allow dump to encrypted partition because it required
> manually entering the password.

This doesn't work as you expect. Because when I set dump target to unencrypted /boot, start kdump.service and wait for crash, crash ramdisk asks me for LUKS password anyway before it runs kdump.

> And we don't expect that after the crash
> somebody will be sitting there to enter the password to unlock the encrypted
> disk. We want to quickly dump and reboot back to first kernel.

Even on Workstation installations? I understand this default for servers. But I think you can detect workstations from servers when running Anaconda Initial Setup or Firstboot in GUI mode instead of TUI or scripted installation and allow this "wait for user password input" scenario on workstations.

Question about workaround: How can I force kdump.service to start when /var/crash is on encrypted partition?

Comment 6 Vivek Goyal 2014-01-14 19:46:23 UTC
(In reply to Martin Holec from comment #5)
> (In reply to Vivek Goyal from comment #3)
> > So far we did not allow dump to encrypted partition because it required
> > manually entering the password.
> 
> This doesn't work as you expect. Because when I set dump target to
> unencrypted /boot, start kdump.service and wait for crash, crash ramdisk
> asks me for LUKS password anyway before it runs kdump.

Hmm..., Thinking more about it. I think it is happening because we always
mount root even if dump destination is not root. One reason being that root
acts as backup target if dump to primary target fails. And second reason
might be that dracut has not been written to be able to work without being
passed a root parameter.

So I agree that it might not be working as one might expect. If root is encrypted, dracut initrams still might be asking for password. We will have
to do some testing.

So bottom line is that to avoid encrypted root restriction, we need to
come up with a way where we don't try to mount root in second kernel and
only mount dump partition which is not encrypted.


> 
> > And we don't expect that after the crash
> > somebody will be sitting there to enter the password to unlock the encrypted
> > disk. We want to quickly dump and reboot back to first kernel.
> 
> Even on Workstation installations?

I think so. You lock your workstation and go home and workstation crashes. I
would prefer that it saves the dump automatically and reboot back so that I 
can atleast log into it remotely.

> 
> Question about workaround: How can I force kdump.service to start when
> /var/crash is on encrypted partition?

I thought just changing dump destination will do it. But looks like you are not getting expected results. I will play with it a bit to figure out if there is a quick workaround.

Comment 7 Martin 2014-01-14 19:53:49 UTC
> > > And we don't expect that after the crash
> > > somebody will be sitting there to enter the password to unlock the encrypted
> > > disk. We want to quickly dump and reboot back to first kernel.
> > 
> > Even on Workstation installations?
> 
> I think so. You lock your workstation and go home and workstation crashes. I
> would prefer that it saves the dump automatically and reboot back so that I 
> can atleast log into it remotely.

When my workstation reboots it asks me for rootfs LUKS password locally before it boots. I can't login to it remotely.

Comment 8 Vivek Goyal 2014-01-14 20:14:20 UTC
(In reply to Martin Holec from comment #7)
> > I think so. You lock your workstation and go home and workstation crashes. I
> > would prefer that it saves the dump automatically and reboot back so that I 
> > can atleast log into it remotely.
> 
> When my workstation reboots it asks me for rootfs LUKS password locally
> before it boots. I can't login to it remotely.

That's a good point. So if root is encrypted, one can't reboot back without manual intervention anyway. This is true for servers too. 

Problem will happen only if we dump target is encrypted and root is not. 

May be a better idea is to just warn if an encrypted target is in dump path and not fail the dump operation.

Comment 9 arthur 2014-01-15 07:18:05 UTC
(In reply to Martin Holec from comment #2)
> 
> 2) After I provided password, kdump writed dump file and rebooted. But I
> can't find any files on /boot/var/crash/ nor /var/crash/.

Hi, Martin Holec

Can you offer the steps to reproduce this issue?  Because I dump successfully with
a encrypted rootfs and using /boot as dump target. After input the passwd, kdump do dump and system reboot. I got the vmcore file in the /boot/var/crash.

Thanks
arthur

Comment 10 Dave 2014-02-13 16:48:08 UTC
I have the same problem with Fedora 20.

Feb 10 09:54:52 xxx systemd: Started Hostname Service.
Feb 10 09:54:53 xxx systemd: kdump.service: main process exited, code=exited, status=1/FAILURE
Feb 10 09:54:53 xxx systemd: Failed to start Crash recovery kernel arming.
Feb 10 09:54:53 xxx systemd: Unit kdump.service entered failed state.
Feb 10 09:54:53 xxx kdumpctl: No kdump initial ramdisk found.
Feb 10 09:54:53 xxx kdumpctl: Rebuilding /boot/initramfs-3.12.9-301.fc20.x86_64kdump.img
Feb 10 09:54:53 xxx libvirtd: 2014-02-10 14:54:50.771+0000: 1582: info : libvirt version: 1.1.3.3, package: 2.fc20 (Fedora Project, 2014-01-20-22:28:02, buildvm-21.phx2.fedoraproject.org)
Feb 10 09:54:53 xxx libvirtd: 2014-02-10 14:54:50.771+0000: 1582: warning : virGetHostname:625 : getaddrinfo failed for 'xxx': Name or service not known
Feb 10 09:54:53 xxx colord: (colord:1526): Cd-WARNING **: failed to get session [pid 1347]: Unknown error -2
Feb 10 09:54:53 xxx colord: (colord:1526): Cd-WARNING **: failed to get session [pid 1347]: Unknown error -2
Feb 10 09:54:53 xxx colord: Profile added: MFC-9440CN-Gray..
Feb 10 09:54:53 xxx colord: Profile added: MFC-9440CN-CMYK..
Feb 10 09:54:53 xxx colord: Device added: cups-MFC-9440CN
Feb 10 09:54:53 xxx kdumpctl: Device /dev/sda2 is encrypted, can not be used in kdump.
Feb 10 09:54:53 xxx kdumpctl: Can not save vmcore to target device /dev/mapper/luks-fc1ab12d-bdc1-4fe0-bd3a-43e7afb7790e.
Feb 10 09:54:53 xxx kdumpctl: mkdumprd: failed to make kdump initrd
Feb 10 09:54:53 xxx kdumpctl: Starting kdump: [FAILED]

Comment 11 Vivek Goyal 2014-02-13 17:32:49 UTC
Ok, so we are talking of 3 issues in this bugzilla so far.

1. If dump target has an encrypted device in the path, kdump fails to start.
2. If root is encrypted then dump should be saved to /boot by default.
3. If /boot is specified manually, dump was not available there after dump saving operation.

I guess failing the dump operation on encrypted targets is too strict an action. We can probably relax restrictions a bit and just warn user that encrypted device
is in dump path and second kernel will wait on console for password to be entered. If user does not want that, change dump target.

This should take care of problem 1. And this should also make problem 2 not a problem any more because by default one would be able to dump to encrypted root (as long as user is willing to enter root password). And there is no need to choose /boot as automatic destination.

Third problem Aruthr can't reproduce. Martin, can you still reproduce the issue of not being able to save to /boot. If this is a bug it needs to be fixed.

So I think we need to do following.

- Relax restrictions on encrypted dump target. Warn user about it but continue to build initramfs.

- Make sure saving to /boot is not a problem when rootfs disk is encrypted.

Comment 12 Vivek Goyal 2014-02-13 17:56:26 UTC
Arthur, can you look into the first problem and post a patch for that.

Comment 19 Ludek Smid 2014-06-13 12:05:53 UTC
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.


Note You need to log in before you can comment on or make changes to this bug.