Bug 1297526

Summary:	LUKS password refused during grub2 BIOS boot on LVM-on-LUKS system following update on 2016-01-11, including updates from roughly prior 48 hours
Product:	[Fedora] Fedora	Reporter:	reescf
Component:	dracut	Assignee:	dracut-maint-list
Status:	CLOSED NOTABUG	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	23	CC:	bcl, dracut-maint-list, harald, jonathan, lkundrak, mads, pjones, reescf, zbyszek
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-01-12 20:03:02 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description reescf 2016-01-11 19:01:14 UTC

Description of problem:

Following update of system on 2016-01-11, boot fails because my LUKS password is no longer accepted as correct. The system was previously updated on 2016-01-09 and rebooted multiple times last week. The system uses LVM over LUKS with unencrypted boot partition. root, home etc. are all logical volumes over the encrypted LUKS partition. grub.cfg is generated manually by running grub2-mkconfig -o grub.cfg in /boot/grub2 after each kernel update, since the system has never auto-updated the configuration file on this system. grub2-mkconfig uses some custom options from /etc which result in menu entries of the form shown below.

The system uses GPT partitioning with BIOS booting. It has an EFI partition because anaconda would not install Fedora without one when the system was originally installed. However, this is not used for booting. Only the /boot partition is used to boot.

The password GUI *is* displayed and I don't believe that the problem is due to a kernel update because I am 95% certain I've booted with the latest kernel successfully before. I rebooted today because the updates included systemd.

Moreover, the same problem occurs with the 4.2.8 kernel, the 4.2.7 kernel and an old kernel from Fedora 21 which lists as belonging to Fedora 23 (probably this was the kernel DNF used during the upgrade), as well as attempting to boot using the rescue option. These are the only options in the grub menu, although older kernels from Fedora 21 exist on the system.

I can get a grub prompt, but that's it.

grub entry for current kernel looks like this:

insmod all_video
set gfxpayload=keep
insmod gzio
insmod part_gpt
insmod ext2
set root='hd0.gpt2'

Then there is a standard if-else-fi block of this form:

if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy ...
else
search ...
fi

Then the command line and initial ram disk:

linux16 /vmlinuz-4.2.8-300.fc23.x86_64 root=/dev/mapper/f101-fedora ro rd.md=0 rd.dm=0 rd.lvm.lv=f101/swap rd.luks.uuid=luks-<long number giving ID> vconsole.font=latarcyrheb-sun16 vconsole.keymap=uk rd.lvm.lv=f101/fedora rd.lvm.lv=f101/lleol rd.lvm.lv=f101/between rd.lvm.lv=f101/cartref rhgb quiet
initrd16 /initramfs-4.2.8-300.fc23.x86_64.img

Version-Release number of selected component (if applicable):

Sorry, but I'm unsure how to provide this information from the grub prompt. Kernel is 4.2.8 as above.

How reproducible:

100%. I cannot boot beyond the password request no matter what I try.

Steps to Reproduce:
1. Configure system as above.
2. Install latest software updates, including systemd.
3. Reboot.
4. Cry.

Actual results:

System refuses LUKS password as it does if I mistype it i.e. as if I am giving an incorrect password. (But the password is not incorrect: I've been successfully booting with this password for well over a year, including following upgrade from Fedora 21 to 23 just before Christmas.)

Expected results:

Password is accepted, LUKS partition is decrypted, LVM lvs are found and mounted, system continues booting.

Additional info:

I'm more than happy to diagnose further. (Indeed, I'm desperate to do this since I can't get any work done right now.) However, I'm not terribly familiar with the grub prompt which is all I currently have access to, so I apologise if I'm omitted essential information which I could easily have provided.

Comment 1 reescf 2016-01-11 19:28:13 UTC

Just to clarify: /boot is unencrypted on my system. 

But /, /home, /usr/local, /mnt/between etc. are encrypted and are normally mounted when the encrypted LUKS partition is unlocked on boot. This makes the logical volume group f101 available and the various logical volumes used for /, /home, /usr/local etc. are then available and mounted. 

In the f101 lvm group:

fedora -> /
cartref -> /home
lleol -> /usr/local
between -> /mnt/between
 
What seems odd is that booting with the same kernel/initial ram disk worked last week and doesn't now, even though all of this stuff is the same. The only thing I can think is the systemd update. But I can't understand how that could matter since systemd isn't available until after the LUKS container is decrypted, is it?

Comment 2 Brian Lane 2016-01-11 22:18:32 UTC

dracut is what handles unlocking and mounting the partitions, not grub. If you're getting to the point where it prompts for the password it has loaded the kernel and initrd.

You may want to edit out the vconsole.keymap=uk entry and see if that makes a difference.

Comment 3 reescf 2016-01-12 16:10:08 UTC

I'm very sorry, but I don't think this is a bug at all. Yesterday, I repeated this many times and it always failed. Today, I rebooted again and everything worked fine. I did disconnect the mouse during boot, but I tried that yesterday as well. My suspicion is that I somehow repeatedly reversed two parts of my password, although I have no idea why I would have done that. If so, I'm very sorry for reporting this spuriously.