Bug 1028397

For clarity, the encrypted filesystem was not the root FS, and not the FS where the dump was directed.

This RFE should be taken upstream.

Personally, I'd be hesitant to trust any credentials passed from the kernel that just crashed.  There might be a way to do this on kdump load, but it has some security implications that probably need to be looked into first.

This mayhave been improved, can you verify?

My experience last month is that it asked for encryption key on my install where everything except /boot was in a LUKS encrypted LVM.  (which was a big problem as th euser did not see it asig for key as the DE is stil shown on screen, frozen)

So, reinstall, and it did not ask when i had / in an unencrypted LVM and /home and /swap in a LUKS LVM.

(that said it most often fail to work for some reason, but it catched one of many resume crashes, and manually triggering always works)

kernel 3.14.4-200.fc20.x86_64

3.14.4-200.fc20.x86_64 worse, if anything:  two blank screens on sysrq-c.  Possible interaction with bug 1100193.

(In reply to Jeremy Harris from comment #1)
> For clarity, the encrypted filesystem was not the root FS, and not the FS
> where the dump was directed.

I think this is core of the problem. We should not be trying to mount an fs (except root and fs where dump is directed to).

> 
> Expected results:
>   Mainline kernel should pass volume credentials to kdump kernel for use
> without further interaction.

Storing credentials in kdump initramfs is security risk. Anybody with permissions can open initramfs and can steal the password.

So it is same issue as encrypted lvm during boot. Everytime you have to enter your password.

Problem here is why we are trying to mount an encrypted file system which does not fall into dump path.

It could be a dracut/systemd issue too. But investigation starts from kexec-tools makding sure we are not asking dracut to mount that fs (using --mount option).

(In reply to Morgan Leijström from comment #3)
> This mayhave been improved, can you verify?
> 
> My experience last month is that it asked for encryption key on my install
> where everything except /boot was in a LUKS encrypted LVM.  (which was a big
> problem as th euser did not see it asig for key as the DE is stil shown on
> screen, frozen)

If entering password for encrypted for root is not a problem when booting your
laptop, why is it a problem when same root is mounted again in second kernel?

> 
> So, reinstall, and it did not ask when i had / in an unencrypted LVM and
> /home and /swap in a LUKS LVM.

That's expected. Now your / is unencrypted and kdump will not try to mount 
/home (assuming your dump destination is root in this case).

Agreed on the #6 security point.  Addressing #5 would fix the immediate issue.
Asking for a password is an issue because a kdump is a) a high-stress situation and b) possibly unattended.

Going beyond that, what is the situation for encrpyted rootfs or dumpfs?  Is a password asked for then, and if so could an attempt to retrieve it from the to-be-dumped kernel be made?

(In reply to Jeremy Harris from comment #8)
> Agreed on the #6 security point.  Addressing #5 would fix the immediate
> issue.
> Asking for a password is an issue because a kdump is a) a high-stress
> situation and b) possibly unattended.

I think one should setup a non-encrypted fs for saving dump. In fact for a long time we denied dumping to encrypted fs just for this reason. People opened bug
that they want to dump to encrypted fs and don't mind entering credentials.

So for those who don't want to enter credentials in second kernel, they should
setup a separate unencrypted partition for saving dump.

> 
> Going beyond that, what is the situation for encrpyted rootfs or dumpfs?  Is
> a password asked for then, and if so could an attempt to retrieve it from
> the to-be-dumped kernel be made?

Original kernel is crashed and we can't ask for password after the crash.

One could argue that password could be passed to second kernel in bootparams
during load time. But this is very unconventional and first requires the work
I am doing to implement a new kexec syscall which prepares bootparam in kernel (as opposed to user space).

So in long term may be there is a case that pass credentials from old kernel to new kernel using bootparams. But don't expect anything soon.

> If entering password for encrypted for root is not a problem when booting
> your laptop, why is it a problem when same root is mounted again in second kernel?

There is no visible dialog to enter the key.
A normal user do not know to press Ctrl-Alt-Fx and expect to find a key dialog there when his screen froze.

Idea: It would be an improvement if the screen would switch to text mode and that encryption key dialog was visible.
Should we open an issue on that?

> > So, reinstall, and it did not ask when i had / in an unencrypted LVM and
> > /home and /swap in a LUKS LVM.
> 
> That's expected. Now your / is unencrypted and kdump will not try to mount 
> /home (assuming your dump destination is root in this case).

Exactly.  So what i meant to ask is if everybody agree *this* bug is solved?

Meanwhile, i found this very similar bug 1077625 is waiting for the same answer.

(In reply to Jeremy Harris from comment #1)
> For clarity, the encrypted filesystem was not the root FS, and not the FS
> where the dump was directed.

James,

Can you paste your /etc/fstab entry here. I am wondering if by any chance you are using mount option x-initrd. I think that option forces mounting of fs from initramfs.

(In reply to Vivek Goyal from comment #11)
> (In reply to Jeremy Harris from comment #1)
> > For clarity, the encrypted filesystem was not the root FS, and not the FS
> > where the dump was directed.
> 
> James,

Sorry, It should have been Jeremy and not James.

(In reply to Morgan Leijström from comment #10)
> 
> There is no visible dialog to enter the key.
> A normal user do not know to press Ctrl-Alt-Fx and expect to find a key
> dialog there when his screen froze.
> 
> Idea: It would be an improvement if the screen would switch to text mode and
> that encryption key dialog was visible.
> Should we open an issue on that?

I think this should be a separate but tracked in separate issue. So how does it look. I thought during reboot, when relevant graphics driver loads, it will reset the display, drop in text mode and then one would see the dialog for entering password on same screen. Is that not the case?

> 
> > > So, reinstall, and it did not ask when i had / in an unencrypted LVM and
> > > /home and /swap in a LUKS LVM.
> > 
> > That's expected. Now your / is unencrypted and kdump will not try to mount 
> > /home (assuming your dump destination is root in this case).
> 
> Exactly.  So what i meant to ask is if everybody agree *this* bug is solved?

I think this particular issue needs to first fix that why just by putting an
entry in /etc/fstab, we are faced with a password dialog for that fs in the second kernel, That requires fixing.

fstab:
=========
/dev/mapper/fedora-root /                       ext4    defaults        1 1
UUID=4291489d-a4fc-46b3-b5ee-0ff880c008da /boot ext4    defaults        1 2
/dev/mapper/fedora-swap swap                    swap    defaults        0 0
/dev/mapper/HDp2-home 	/hd			ext4	nofail		1 2
=========

The last line is the encrypted fs.

(In reply to Vivek Goyal from comment #13)
> I thought during reboot, when relevant graphics driver loads, it
> will reset the display, drop in text mode and then one would see the dialog
> for entering password on same screen. Is that not the case?

For unencrypted / that is how it works here, yes, good.

But when I had / in encrypted LVM, it just still displayed the image of the frozen desktop while it was waiting for key input;  For encrypted / , graphics driver do not load until after key is entered...

So screen need to be switched to text mode by some other means


Regarding the discussed option to have a separate partition to save dump to, also / must be unencrypted, i presume.  I guess dump may contain sensitive information, so the file should optimally be encrypted, if on an unencrypted partition?


> I think this particular issue needs to first fix that why just by putting an
> entry in /etc/fstab, we are faced with a password dialog for that fs in the
> second kernel, That requires fixing.

yes

(In reply to Jeremy Harris from comment #14)
> fstab:
> =========
> /dev/mapper/fedora-root /                       ext4    defaults        1 1
> UUID=4291489d-a4fc-46b3-b5ee-0ff880c008da /boot ext4    defaults        1 2
> /dev/mapper/fedora-swap swap                    swap    defaults        0 0
> /dev/mapper/HDp2-home 	/hd			ext4	nofail		1 2
> =========
> 
> The last line is the encrypted fs.

Thanks Jermey. So /etc/fstab entry looks fine. Next step would to be figure out how kdumpctl called dracut. What --mount options were passed.

Can you please enable debugging in /usr/bin/kdumpctl.  Add "set -x" at the top and run "kdumpctl restart". All the debug info will be thrown on console. Store it in a file and attach with the bug.

I am also interested in /etc/fstab file of generated kdump initramfs. So if you can open that initramfs and paste the contents here it would help. I want to see if somehow that encrypted device entry made into that file and is that the one is trying to mount it.

Jermey,

Also can you provide lsblk output of your system.

Created attachment 901631 [details]
"kdumpctl restart"  shell trace

Created attachment 901633 [details]
output from "lsblk"

The fstab in the newly-generated initramfs is empty:

[root@lap tmp]# cpio -itcv <foo | grep fstab
-rw-r--r--   1 root     root            0 Jun  3 09:09 etc/fstab.empty
-rwxr-xr-x   1 root     root        57664 Jun  3 09:09 usr/lib/systemd/system-generators/systemd-fstab-generator
81339 blocks
[root@lap tmp]#


That for my running kernel is similar.

(In reply to Jeremy Harris from comment #18)
> Created attachment 901631 [details]
> "kdumpctl restart"  shell trace

Hi, Jeremy

Can you also set -x in /sbin/mkdumprd and provide us the trace along with kdumpctl? That'd much helpful!

Created attachment 901676 [details]
kdumpctl/mkdumprd trace

(In reply to Jeremy Harris from comment #22)
> Created attachment 901676 [details]
> kdumpctl/mkdumprd trace

Emm. You don't reserve crashkernel:

"Memory for crashkernel is not reserved
 Please reserve memory by passing "crashkernel=X@Y" parameter to the kernel"

(In reply to WANG Chao from comment #23)
> (In reply to Jeremy Harris from comment #22)
> > Created attachment 901676 [details]
> > kdumpctl/mkdumprd trace
> 
> Emm. You don't reserve crashkernel:
> 
> "Memory for crashkernel is not reserved
>  Please reserve memory by passing "crashkernel=X@Y" parameter to the kernel"

And also please `touch /etc/kdump.conf` before `kdumpctl restart` for triggering a re-generating of kdump initramfs.

Created attachment 901783 [details]
kdumpctl/mkdumprd trace

Ok, so we call dracut as follows.

dracut --hostonly -o 'plymouth dash resume' -f /boot/initramfs-3.14.4-200.fc20.x86_64kdump.img 3.14.4-200.fc20.x86_64

So nowhere we are telling dracut to mount the encrypted fs.

I will cc harald to see if he has any ideas what might be going on . In the mean time it is worth to enable debugging of dracut during second kernel boot.

I think pass rd.debug to second kernel command line parameter. Modify /etc/sysconfig/kdump file and append rd.debug to KDUMP_COMMANDLINE_APPEND= and
restart kdump service. That will make sure this parameter is passed to second kernel.

I am not sure if rd.debug will output everything on console or will save it to a file. If it saves it to a file, then you will have to drop into shell, save log file to some destination and then reboot.

Hmm, That is really strange. I do the same steps and can't reproduce the issue.
My /etc/fstab is:
/dev/mapper/fedora-root /                       ext4    defaults        1 1
UUID=a4d25820-5205-42d0-93e2-326f432720d1 /boot                   ext4    defaults        1 2
/dev/mapper/luks-0dd04f85-6523-4807-9d26-a970e67202a6 /tmp                    ext4    nofail 1 2
/dev/mapper/fedora-swap swap                    swap    defaults        0 0


kernerl version is 3.11.10-301.fc20.x86_64
kdump.conf is default

after crash, dump succeed without requesting a password , and then do the final action to reboot the system. At this time, a password is requseted and it is normal because /tmp will be mount automatically and it is encrypted.

Anyway, Let's follow up vivek's suggestion to cc harald and enable debugging of dracut to see what is going on.

Thanks
zzou

for comment #15, i submitted Bug 1104893

As you are "online" here i want to ask if I for the following problem should open a new issue, add to one (i see many kdump related) or just wait while some other issues get resolved and this is maybe solved then... :

On this system kdump works when manually triggered, but not when the system crash and reboots on resume from suspend.  It also have happened the system just reboots during use without kdump doing its thing.  (there is not even a delay like when it is saving vmcore)  System have / in regular unencrypted LVM, but /home and /swap in encrypted lvm.

Morgan, please open bugs for the issues, not necessary to depend on this on if you think they are other issues.

OK. comment #29 -> Bug 1105047

I've tried but cannot reproduce the bug. It won't ask for password if the encrypted disk is not the dump target.

I'll close this bug if no more information provided.