Bug 1028397 - kdump asks for luks password, when encrypted fs mountable
kdump asks for luks password, when encrypted fs mountable
Status: CLOSED INSUFFICIENT_DATA
Product: Fedora
Classification: Fedora
Component: kexec-tools (Show other bugs)
rawhide
x86_64 Linux
unspecified Severity low
: ---
: ---
Assigned To: Dangyi Liu
Fedora Extras Quality Assurance
: FutureFeature, Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-11-08 06:06 EST by Jeremy Harris
Modified: 2015-10-16 00:43 EDT (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-10-16 00:43:05 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
"kdumpctl restart" shell trace (75.80 KB, text/plain)
2014-06-03 04:14 EDT, Jeremy Harris
no flags Details
output from "lsblk" (2.52 KB, text/plain)
2014-06-03 04:14 EDT, Jeremy Harris
no flags Details
kdumpctl/mkdumprd trace (75.51 KB, text/plain)
2014-06-03 04:59 EDT, Jeremy Harris
no flags Details
kdumpctl/mkdumprd trace (188.58 KB, text/plain)
2014-06-03 09:20 EDT, Jeremy Harris
no flags Details

  None (edit)
Description Jeremy Harris 2013-11-08 06:06:53 EST
Description of problem:
  On dumping to the default place, on a system that has in fstab an encrypted volume, the luks password is requested.  While usable this is needlessly difficult in a high-stress situation; the volume was presumably operational under the mainline kernel.  Requiring interaction makes remote-crashdumps harder.


Version-Release number of selected component (if applicable):
 3.11.7-300.fc20.x86_64

How reproducible:
 100%

Steps to Reproduce:
1. Add luks-encryted filesystem to fstab, and mount.
2. Configure kdump
3. Sysrq-c

Actual results:
  Dump kernel startup pauses with request for luks password (at least, running on an alternate text console.  I didn't attempt running from the desktop).


Expected results:
  Mainline kernel should pass volume credentials to kdump kernel for use without further interaction.

Additional info:
Comment 1 Jeremy Harris 2013-11-08 06:08:35 EST
For clarity, the encrypted filesystem was not the root FS, and not the FS where the dump was directed.
Comment 2 Josh Boyer 2013-11-08 09:36:03 EST
This RFE should be taken upstream.

Personally, I'd be hesitant to trust any credentials passed from the kernel that just crashed.  There might be a way to do this on kdump load, but it has some security implications that probably need to be looked into first.
Comment 3 Morgan Leijström 2014-06-02 08:48:32 EDT
This mayhave been improved, can you verify?

My experience last month is that it asked for encryption key on my install where everything except /boot was in a LUKS encrypted LVM.  (which was a big problem as th euser did not see it asig for key as the DE is stil shown on screen, frozen)

So, reinstall, and it did not ask when i had / in an unencrypted LVM and /home and /swap in a LUKS LVM.

(that said it most often fail to work for some reason, but it catched one of many resume crashes, and manually triggering always works)

kernel 3.14.4-200.fc20.x86_64
Comment 4 Jeremy Harris 2014-06-02 09:10:01 EDT
3.14.4-200.fc20.x86_64 worse, if anything:  two blank screens on sysrq-c.  Possible interaction with bug 1100193.
Comment 5 Vivek Goyal 2014-06-02 09:21:44 EDT
(In reply to Jeremy Harris from comment #1)
> For clarity, the encrypted filesystem was not the root FS, and not the FS
> where the dump was directed.

I think this is core of the problem. We should not be trying to mount an fs (except root and fs where dump is directed to).
Comment 6 Vivek Goyal 2014-06-02 09:25:26 EDT
> 
> Expected results:
>   Mainline kernel should pass volume credentials to kdump kernel for use
> without further interaction.

Storing credentials in kdump initramfs is security risk. Anybody with permissions can open initramfs and can steal the password.

So it is same issue as encrypted lvm during boot. Everytime you have to enter your password.

Problem here is why we are trying to mount an encrypted file system which does not fall into dump path.

It could be a dracut/systemd issue too. But investigation starts from kexec-tools makding sure we are not asking dracut to mount that fs (using --mount option).
Comment 7 Vivek Goyal 2014-06-02 09:31:17 EDT
(In reply to Morgan Leijström from comment #3)
> This mayhave been improved, can you verify?
> 
> My experience last month is that it asked for encryption key on my install
> where everything except /boot was in a LUKS encrypted LVM.  (which was a big
> problem as th euser did not see it asig for key as the DE is stil shown on
> screen, frozen)

If entering password for encrypted for root is not a problem when booting your
laptop, why is it a problem when same root is mounted again in second kernel?

> 
> So, reinstall, and it did not ask when i had / in an unencrypted LVM and
> /home and /swap in a LUKS LVM.

That's expected. Now your / is unencrypted and kdump will not try to mount 
/home (assuming your dump destination is root in this case).
Comment 8 Jeremy Harris 2014-06-02 09:34:38 EDT
Agreed on the #6 security point.  Addressing #5 would fix the immediate issue.
Asking for a password is an issue because a kdump is a) a high-stress situation and b) possibly unattended.

Going beyond that, what is the situation for encrpyted rootfs or dumpfs?  Is a password asked for then, and if so could an attempt to retrieve it from the to-be-dumped kernel be made?
Comment 9 Vivek Goyal 2014-06-02 09:43:17 EDT
(In reply to Jeremy Harris from comment #8)
> Agreed on the #6 security point.  Addressing #5 would fix the immediate
> issue.
> Asking for a password is an issue because a kdump is a) a high-stress
> situation and b) possibly unattended.

I think one should setup a non-encrypted fs for saving dump. In fact for a long time we denied dumping to encrypted fs just for this reason. People opened bug
that they want to dump to encrypted fs and don't mind entering credentials.

So for those who don't want to enter credentials in second kernel, they should
setup a separate unencrypted partition for saving dump.

> 
> Going beyond that, what is the situation for encrpyted rootfs or dumpfs?  Is
> a password asked for then, and if so could an attempt to retrieve it from
> the to-be-dumped kernel be made?

Original kernel is crashed and we can't ask for password after the crash.

One could argue that password could be passed to second kernel in bootparams
during load time. But this is very unconventional and first requires the work
I am doing to implement a new kexec syscall which prepares bootparam in kernel (as opposed to user space).

So in long term may be there is a case that pass credentials from old kernel to new kernel using bootparams. But don't expect anything soon.
Comment 10 Morgan Leijström 2014-06-02 09:50:29 EDT
> If entering password for encrypted for root is not a problem when booting
> your laptop, why is it a problem when same root is mounted again in second kernel?

There is no visible dialog to enter the key.
A normal user do not know to press Ctrl-Alt-Fx and expect to find a key dialog there when his screen froze.

Idea: It would be an improvement if the screen would switch to text mode and that encryption key dialog was visible.
Should we open an issue on that?

> > So, reinstall, and it did not ask when i had / in an unencrypted LVM and
> > /home and /swap in a LUKS LVM.
> 
> That's expected. Now your / is unencrypted and kdump will not try to mount 
> /home (assuming your dump destination is root in this case).

Exactly.  So what i meant to ask is if everybody agree *this* bug is solved?

Meanwhile, i found this very similar bug 1077625 is waiting for the same answer.
Comment 11 Vivek Goyal 2014-06-02 10:01:46 EDT
(In reply to Jeremy Harris from comment #1)
> For clarity, the encrypted filesystem was not the root FS, and not the FS
> where the dump was directed.

James,

Can you paste your /etc/fstab entry here. I am wondering if by any chance you are using mount option x-initrd. I think that option forces mounting of fs from initramfs.
Comment 12 Vivek Goyal 2014-06-02 10:02:33 EDT
(In reply to Vivek Goyal from comment #11)
> (In reply to Jeremy Harris from comment #1)
> > For clarity, the encrypted filesystem was not the root FS, and not the FS
> > where the dump was directed.
> 
> James,

Sorry, It should have been Jeremy and not James.
Comment 13 Vivek Goyal 2014-06-02 10:06:16 EDT
(In reply to Morgan Leijström from comment #10)
> 
> There is no visible dialog to enter the key.
> A normal user do not know to press Ctrl-Alt-Fx and expect to find a key
> dialog there when his screen froze.
> 
> Idea: It would be an improvement if the screen would switch to text mode and
> that encryption key dialog was visible.
> Should we open an issue on that?

I think this should be a separate but tracked in separate issue. So how does it look. I thought during reboot, when relevant graphics driver loads, it will reset the display, drop in text mode and then one would see the dialog for entering password on same screen. Is that not the case?

> 
> > > So, reinstall, and it did not ask when i had / in an unencrypted LVM and
> > > /home and /swap in a LUKS LVM.
> > 
> > That's expected. Now your / is unencrypted and kdump will not try to mount 
> > /home (assuming your dump destination is root in this case).
> 
> Exactly.  So what i meant to ask is if everybody agree *this* bug is solved?

I think this particular issue needs to first fix that why just by putting an
entry in /etc/fstab, we are faced with a password dialog for that fs in the second kernel, That requires fixing.
Comment 14 Jeremy Harris 2014-06-02 10:14:51 EDT
fstab:
=========
/dev/mapper/fedora-root /                       ext4    defaults        1 1
UUID=4291489d-a4fc-46b3-b5ee-0ff880c008da /boot ext4    defaults        1 2
/dev/mapper/fedora-swap swap                    swap    defaults        0 0
/dev/mapper/HDp2-home 	/hd			ext4	nofail		1 2
=========

The last line is the encrypted fs.
Comment 15 Morgan Leijström 2014-06-02 10:40:55 EDT
(In reply to Vivek Goyal from comment #13)
> I thought during reboot, when relevant graphics driver loads, it
> will reset the display, drop in text mode and then one would see the dialog
> for entering password on same screen. Is that not the case?

For unencrypted / that is how it works here, yes, good.

But when I had / in encrypted LVM, it just still displayed the image of the frozen desktop while it was waiting for key input;  For encrypted / , graphics driver do not load until after key is entered...

So screen need to be switched to text mode by some other means


Regarding the discussed option to have a separate partition to save dump to, also / must be unencrypted, i presume.  I guess dump may contain sensitive information, so the file should optimally be encrypted, if on an unencrypted partition?


> I think this particular issue needs to first fix that why just by putting an
> entry in /etc/fstab, we are faced with a password dialog for that fs in the
> second kernel, That requires fixing.

yes
Comment 16 Vivek Goyal 2014-06-02 16:34:22 EDT
(In reply to Jeremy Harris from comment #14)
> fstab:
> =========
> /dev/mapper/fedora-root /                       ext4    defaults        1 1
> UUID=4291489d-a4fc-46b3-b5ee-0ff880c008da /boot ext4    defaults        1 2
> /dev/mapper/fedora-swap swap                    swap    defaults        0 0
> /dev/mapper/HDp2-home 	/hd			ext4	nofail		1 2
> =========
> 
> The last line is the encrypted fs.

Thanks Jermey. So /etc/fstab entry looks fine. Next step would to be figure out how kdumpctl called dracut. What --mount options were passed.

Can you please enable debugging in /usr/bin/kdumpctl.  Add "set -x" at the top and run "kdumpctl restart". All the debug info will be thrown on console. Store it in a file and attach with the bug.

I am also interested in /etc/fstab file of generated kdump initramfs. So if you can open that initramfs and paste the contents here it would help. I want to see if somehow that encrypted device entry made into that file and is that the one is trying to mount it.
Comment 17 Vivek Goyal 2014-06-02 16:55:22 EDT
Jermey,

Also can you provide lsblk output of your system.
Comment 18 Jeremy Harris 2014-06-03 04:14:19 EDT
Created attachment 901631 [details]
"kdumpctl restart"  shell trace
Comment 19 Jeremy Harris 2014-06-03 04:14:58 EDT
Created attachment 901633 [details]
output from "lsblk"
Comment 20 Jeremy Harris 2014-06-03 04:21:36 EDT
The fstab in the newly-generated initramfs is empty:

[root@lap tmp]# cpio -itcv <foo | grep fstab
-rw-r--r--   1 root     root            0 Jun  3 09:09 etc/fstab.empty
-rwxr-xr-x   1 root     root        57664 Jun  3 09:09 usr/lib/systemd/system-generators/systemd-fstab-generator
81339 blocks
[root@lap tmp]#


That for my running kernel is similar.
Comment 21 WANG Chao 2014-06-03 04:48:21 EDT
(In reply to Jeremy Harris from comment #18)
> Created attachment 901631 [details]
> "kdumpctl restart"  shell trace

Hi, Jeremy

Can you also set -x in /sbin/mkdumprd and provide us the trace along with kdumpctl? That'd much helpful!
Comment 22 Jeremy Harris 2014-06-03 04:59:10 EDT
Created attachment 901676 [details]
kdumpctl/mkdumprd trace
Comment 23 WANG Chao 2014-06-03 06:10:37 EDT
(In reply to Jeremy Harris from comment #22)
> Created attachment 901676 [details]
> kdumpctl/mkdumprd trace

Emm. You don't reserve crashkernel:

"Memory for crashkernel is not reserved
 Please reserve memory by passing "crashkernel=X@Y" parameter to the kernel"
Comment 24 WANG Chao 2014-06-03 06:12:02 EDT
(In reply to WANG Chao from comment #23)
> (In reply to Jeremy Harris from comment #22)
> > Created attachment 901676 [details]
> > kdumpctl/mkdumprd trace
> 
> Emm. You don't reserve crashkernel:
> 
> "Memory for crashkernel is not reserved
>  Please reserve memory by passing "crashkernel=X@Y" parameter to the kernel"

And also please `touch /etc/kdump.conf` before `kdumpctl restart` for triggering a re-generating of kdump initramfs.
Comment 25 Jeremy Harris 2014-06-03 09:20:20 EDT
Created attachment 901783 [details]
kdumpctl/mkdumprd trace
Comment 26 Vivek Goyal 2014-06-03 10:59:04 EDT
Ok, so we call dracut as follows.

dracut --hostonly -o 'plymouth dash resume' -f /boot/initramfs-3.14.4-200.fc20.x86_64kdump.img 3.14.4-200.fc20.x86_64

So nowhere we are telling dracut to mount the encrypted fs.
Comment 27 Vivek Goyal 2014-06-03 11:06:20 EDT
I will cc harald to see if he has any ideas what might be going on . In the mean time it is worth to enable debugging of dracut during second kernel boot.

I think pass rd.debug to second kernel command line parameter. Modify /etc/sysconfig/kdump file and append rd.debug to KDUMP_COMMANDLINE_APPEND= and
restart kdump service. That will make sure this parameter is passed to second kernel.

I am not sure if rd.debug will output everything on console or will save it to a file. If it saves it to a file, then you will have to drop into shell, save log file to some destination and then reboot.
Comment 28 arthur 2014-06-04 01:26:00 EDT
Hmm, That is really strange. I do the same steps and can't reproduce the issue.
My /etc/fstab is:
/dev/mapper/fedora-root /                       ext4    defaults        1 1
UUID=a4d25820-5205-42d0-93e2-326f432720d1 /boot                   ext4    defaults        1 2
/dev/mapper/luks-0dd04f85-6523-4807-9d26-a970e67202a6 /tmp                    ext4    nofail 1 2
/dev/mapper/fedora-swap swap                    swap    defaults        0 0


kernerl version is 3.11.10-301.fc20.x86_64
kdump.conf is default

after crash, dump succeed without requesting a password , and then do the final action to reboot the system. At this time, a password is requseted and it is normal because /tmp will be mount automatically and it is encrypted.

Anyway, Let's follow up vivek's suggestion to cc harald and enable debugging of dracut to see what is going on.

Thanks
zzou
Comment 29 Morgan Leijström 2014-06-04 18:09:41 EDT
for comment #15, i submitted Bug 1104893

As you are "online" here i want to ask if I for the following problem should open a new issue, add to one (i see many kdump related) or just wait while some other issues get resolved and this is maybe solved then... :

On this system kdump works when manually triggered, but not when the system crash and reboots on resume from suspend.  It also have happened the system just reboots during use without kdump doing its thing.  (there is not even a delay like when it is saving vmcore)  System have / in regular unencrypted LVM, but /home and /swap in encrypted lvm.
Comment 30 Dave Young 2014-06-05 03:40:27 EDT
Morgan, please open bugs for the issues, not necessary to depend on this on if you think they are other issues.
Comment 31 Morgan Leijström 2014-06-05 05:36:27 EDT
OK. comment #29 -> Bug 1105047
Comment 32 Dangyi Liu 2015-10-13 06:29:04 EDT
I've tried but cannot reproduce the bug. It won't ask for password if the encrypted disk is not the dump target.

I'll close this bug if no more information provided.

Note You need to log in before you can comment on or make changes to this bug.