Red Hat Bugzilla – Bug 896010
SELinux gets in the way of systemd-ask-password trying to use cached passwords during system upgrade
Last modified: 2013-06-23 01:56:15 EDT
Description of problem:
This was reported by 'nyloc' on #fedora-qa, I reproduced it. I installed clean Fedora 17 on this disk layout:
vda2: / (encrypted)
vda3: swap (encrypted)
vdb1: /data (encrypted)
All standard partitions, no LVM. The password is the same for both systems.
After I finished running fedup-cli and rebooted, the system asks for disk password and then hangs. nyloc reported he received a maintenance shell after a timeout, I didn't, but that probably doesn't matter much.
This is the System Upgrade boot line from grub:
linux /vmlinuz-fedup root=/dev/mapper/luks-f1cab7e0-fae8-4a00-94b9-c5d3976d6031 ro rd.luks.uuid=luks-80aa76ad-d5a3-4bfa-b8cd-7f70fe8379a0 rd.lvm=0 rd.dm=0 SYSFONT=True rd.luks.uuid=luks-f1cab7e0-fae8-4a00-94b9-c5d3976d6031 KEYTABLE=us rd.md=0 LANG=en_US.UTF-8 rhgb quiet upgrade systemd.unit=system-upgrade.target plymouth.splash=fedup
It looks fine.
This is the System Upgrade boot output:
> dracut-cmdline: Warning: Kernel command line option 'KEYTABLE' is deprecated, use 'vconsole.keymap' instead.
> dracut-cmdline: Warning: Kernel command line option 'SYSFONT' is deprecated, use 'vconsole.font' instead.
> [ 3.114503] systemd-cryptsetup-generator: Failed to create unit file: File exists
> [ OK ] Found device /dev/disk/by-uuid/80aa76ad-d5a3-4bfa-b8cd-7f70fe8379a0.
> Starting Cryptography Setup for luks-80aa76ad-d5a3-4...7f70fe8379a0...
> [ OK ] Found device /dev/disk/by-uuid/f1cab7e0-fae8-4a00-94b9-c5d3976d6031.
> Starting Cryptography Setup for luks-f1cab7e0-fae8-4...c5d3976d6031...
> Expecting device dev-mapper-luks\x2d80aa76ad\x2dd5a3...379a0.device...
> Expecting device dev-mapper-luks\x2df1cab7e0\x2dfae8...d6031.device...
> [ OK ] Started Show Plymouth Boot Screen.
> Starting Forward Password Requests to Plymouth...
> [ OK ] Started Forward Password Requests to Plymouth.
> Please enter passphrase for disk luks-f1cab7e0-fae8-4a00-94b9-c5d3976d6031!:**********
> [ OK ] Started Cryptography Setup for luks-f1cab7e0-fae8-4a...9-c5d3976d6031.
> [ OK ] Found device /dev/mapper/luks-f1cab7e0-fae8-4a00-94b9-c5d3976d6031.
> [ OK ] Started Cryptography Setup for luks-80aa76ad-d5a3-4b...d-7f70fe8379a0.
> [ OK ] Reached target Local File Systems.
> [ OK ] Reached target Encrypted Volumes.
> [ OK ] Reached target System Initialization.
> [ OK ] Reached target Basic System. <------ HANGS HERE
Boot with rd.debug is attached.
System reboots if Ctrl+Alt+Del is hit, so it's not dead.
Version-Release number of selected component (if applicable):
fedup-dracut of unknown version taken from development/18 tree. (Hey Will, why it is not mentioned anywhere, like in --debuglog?)
with nyloc and my confirmation, seems like every time
Steps to Reproduce:
1. follow description
Created attachment 679569 [details]
Created attachment 679570 [details]
Created attachment 679571 [details]
Same issue when rebooting after running fedup-cli from F17.
As reported by Kamil, but with four encrypted partitions on a non-virtual system.
Duplicate issue with my /home partition after fedup reboot.
Upgrading from Fedora 17 to 18.
Does this upgrade effect LVM?
This has also blown up my LVM?
[root@pegasus ~]# vgdisplay
No volume groups found
[root@pegasus ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 852K 3.9G 1% /dev/shm
tmpfs 3.9G 1.2M 3.9G 1% /run
/dev/dm-0 15G 211M 15G 2% /
/dev/sda6 15G 6.3G 7.5G 46% /usr
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
tmpfs 3.9G 0 3.9G 0% /media
/dev/sda3 985M 139M 797M 15% /boot
/dev/sda7 2.0G 100M 1.8G 6% /usr/local
/dev/sdb6 4.9G 2.8G 1.8G 62% /var
/dev/sdb8 2.0G 266M 1.6G 15% /opt
/dev/sdb7 985M 18M 917M 2% /tmp
/dev/dm-1 385G 289G 77G 80% /home
[root@pegasus ~]# pvdisplay
This didn't occur in my case. I have / and /home partitions encrypted with luks. Swap is not encrypted. I upgraded with fedup and it worked. Could the swap be the problem?
I have the same problem - LVM - encrypted /, /home, swap
Same issue here. Bare metal, update F17->18, all the fedup-cli steps were ok, after reboot fedup-upgrade fails to start and drops to the maintenance shell.
No LVM plain GPT partitions:
- sda1: pre-boot
- sda2: boot (ext4)
- sda3: home (LUKS/ext4)
- sda4: root (LUKS/ext4)
- sda5: tmp (LUKS/ext4)
- sda6: swap (unencrypted)
Under the maint. shell only / was mounted, no tmp nor home.
I have a similar problem. When upgrading from f17 to f18 using fedup, after reboot I got the logo for a second and then it disappears and I get 2 warnings about deprecated flags, one about FONT and another about KEYTABLE (iirc).
After that the boot process gets stuck for a minute or so and I got dropped to rescue shell. I have only one encrypted partition (/home) and I don't have LVM.
On my netbook with Fedora 17 x86_64 with encrypted /, /home, and swap, FedUp completed with no problem. The netbook has an SSD drive. On my desktop with the same setup, FedUp hangs as described.
Could it be a timing issue for the boot process and password entry?
I think I've managed to overcome this. Mostly thanks to denials from #fedora IRC channel.
Denials told me to remove "rhgb quiet" from grub.cfg "upgrade system" entry (after running fedup on old system, before rebooting).
After rebooting the password prompt got lost somewhere in the output, so I waited a few seconds until the screen stopped popping up new messages and I started entering the password for my encrypted partition. The encryption prompt reappeared and after making sure I typed the correct password I hit enter and the upgrade process continued (it's still in progress as I'm typing this message).
Thanx a bunch denials!!! :)
I hope this is helpful
Created attachment 680590 [details]
Screenshot of point where FedUp fails
Rather than try to type all of this, I have taken a picture of the screen and scaled it down to reduce the file size. This is what happens every time. It looks like the password I entered when prompted is not being passed on so all the encrypted partitions can be unlocked.
Could this be related to Red Hat Bugzilla – Bug 749027, which was what happened when Fedora 15 started to change the boot process. If it is, the same solution of adding the line for debug did not work.
I have a setup with encrypted /home and / . I was able to complete the update by adding "selinux=0" to the kernel parameters.
There are probably several issues combined in this report. Adding selinux=0 didn't help me (nothing changed) with the original problem described in comment 0.
Will, we really need you to look into this, it affects a lot of people. I believe you'll have no problems to reproduce the issue using steps that I described above.
Another attempt to upgrade, removing "rhgb quiet" parameters didn't work.
What worked is commenting out my otwer ecrypted partiitions (home and tmp) in fstab. Then despite selinux warning upgrade succeeded (with some small glitches I had to correct manually after upgrade).
I can confirm that selinux=0 in boot parameters allows the upgrade to proceed without dropping to maintenance shell for me.
Just to add that this appears to be part of a wider issue with selinux and the upgrade process, since when the upgrade is finished I can't login either through the GUI or through a VT. Once again, disabling selinux as a boot param resolves this.
Same here. Setting selinux=0 and commenting out /home and swap in /etc/fstab allowed fedup to proceed (I did not try those two individually, so I'm not sure which fixed the problem). Once the update was complete, I was unable to fully boot my system into F18 unless I set selinux=0. After a 'yum update' and a reboot, SELinux relabeling was automatically invoked and the system works fine now.
I ended up booting into F17 and disabling SELinux from there. After that, the update proceeded. This led to a second issue in which I had problems with SELinux after the update, the main symptom being error messages and failure to launch when I tried to invoke system-config-selinux. I ended up ripping out all the parts of SELinux that I could without having it delete everything in the system and then reinstalling them. I then went to /etc/selinux and changed the config file to have the enforcing parameter. After a cople of reboots in the process and a "touch /.autorelabel" and another reboot, SELinux started to behave again and system-config-selinux now launches properly.
I think this is another bit of confirmation that the problem is not as much with encrypted partitions, but with SELinux since once that is disabled, FedUp does complete the upgrade, even with multiple encrypted partitions.
I had the same issue, and followed the comments here. I was able to successfully install by doing this:
1. Edit fed-up entry in grub, adding 'selinux=0' to kernel line
2. Install completed just fine.
3. Edit fedora 18 kernel boot line to boot to runlevel 1 (i.e. append "1" to kernel line)
4. touch /.autorelabel && reboot
5. After autorelabel completed, I was able to log in to the newly upgraded F18 with no issues.
autorelabel did not work for me. Booted with "selinux=0", changed mode from enforcing to permissive in /etc/selinux/config , the rebooted without "selinux=0" parameter. I was then able to run "fixfiles relabel" as root.
All working normally now with selinux enabled and upgrade complete.
successful upgrade with 4 encrypted hard disk
1. /etc/selinux/config SELINUX=disabled
2. /etc/fstab comment out all encrypted disks except file system
3. run fed-up upgrade
4. edit grub before boot for some reason was left fed-up entry
5. boot and re-added in fstab my other 3 disks
My configuration is:
#2 disks, both encrypted
LVM on the first:
/, /var, /var/lib/libvirt, /opt
LVM on the second:
/lv1, /lv2, /home
I have successful upgraded my system following the steps below:
1. boot F17
2. As root disable selinux:
# edit /etc/selinux/config and set SELINUX=disabled
4. As root execute the fedup preaupgrade to F18:
# fedup --network 18 --debuglog fedupdebug.log
5. Wait the end of the fedup upgrade.
6. Login in F18 and as root enable selinux:
# edit /etc/selinux/config and set SELINUX=permissive
7. Login and check the SELinux messages:
# grep "SELinux is preventing" /var/log/messages
8. As root, fix the "denial messages" :
# fixfiles relabel
9. Login and verify that there are no new SELinux messages, repeat step 7.
10. If there are no new denial messages, as root enable selinux in enforcing:
# edit /etc/selinux/config and set SELINUX=enforcing
11. Login and verify the selinux mode:
1. Upgrade with Fedup FAILED.
2. Edit fedup in grub, adding 'selinux=0' to kernel line - Upgrade OK
1. Start Fedora 18 FAILED.
2. Edit Kernel in grub, adding 'selinux=0' to kernel line - Startup Failed (> [ OK ] Reached target Basic System. <------ HANGS HERE)
3. Edit Kernel in grub, adding 'selinux=0'and single to kernel line
a) startup OK to single mod
b) init 3
c) login user
Every reboot I must repeat the procedure.
OKAY HI. SHORT EXPLANATION:
- This is a SELinux problem.
- Use 'enforcing=0' as a workaround
- The graphical progress meter probably won't work
- Hit 'Esc' to get text progress
- 'selinux=0' is *never* the right workaround for a SELinux problem
- Because then you need to relabel afterward
- And it takes a long time and that sucks
- Use 'enforcing=0' instead
Having examined a system that hit this bug (THANX IAN) my analysis is:
The policy in F18 (which is used during the upgrade) doesn't allow systemd-ask-password to read the temporary file that gets written when you unlock your first partition (e.g. / or swap).
So: when it tries to unlock the second partition (e.g. /home), systemd-ask-password fails, and then (I'm guessing) falls back to trying to ask on /dev/console. Except plymouth is up, so the text doesn't display anywhere useful. So it looks like the system has hung... if you're impatient.
If you wait 30 seconds, the system will drop to a systemd emergency shell, because it can't mount /home.
(Interestingly, if you hit 'Esc' before the shell happens, a *hell* of a lot of asterisks appear onscreen. I'm pretty sure that plymouth is trying to display all the console text but its output gets dumped into systemd-ask-password... which displays an asterisk as each character is 'typed'..)
The progress meter also seems to be kind of screwed up - it reports a pretty random completion percentage. I'm guessing this is because the number keeps increasing while you're typing your passphrase (even though I've told it to pause!).
So: it's very likely that it will hit the end of the progress bar and show the Beefy Miracle *before* the upgrade starts, so once the upgrade starts and tries to set the meter to 1%, there's no meter onscreen.. so it just sits there showing Beefy and looking hung.
The solution here is.. tricky. Something needs to happen with SELinux. Possibly (since we have to set it permissive for the upgrade anyway) we may just run the upgrade in permissive mode by default.
There's also maybe a plymouth bug to be fixed, to make pause-progress actually pause the progress so you don't get Early Beefy.
I'll update this bug when I have patches to test.
Will, I hate to spoil the party, but the original issue I reported is not fixed with selinux=0. It is also not affected by the timeout. So it's most probably something else.
But it seems that selinux is the culprit for majority of people here, so it's definitely good to concentrate on that and deal with the original issue later.
I have a single SSD with:
sda1 - /boot no encryption
sda2 - Encrypted with a single volume group and multiple logical volumes (root, home, swap)
SELinux is disabled (at least for my F17 install).
This seems same issue seems to affect me as well.
When I boot into fedup I get a second or so of the progress bar followed by a screen full of asterisks (stars) even though I DID not hit ESC. This continues for roughly an hour until it reboots. At that point nothing has been updated. I've tried setting enforcing=0 for fedup but it has no affect. I'm also unable to re-run fedup after that first attempt without manually deleting the fedup stuff in /boot and rerunning fedup-cli to recreate AND grub2-mkconfig. Although I can't see what's going on with whatever the plymouth issue is it appears fedup thinks it's doing sometime as it writes to /boot.
Fedup deletes its boot entry before the upgrade starts in order to avoid infinite reboot loops.
You can re-add the fedup boot entry by re-running fedup. (You don't need to delete anything in /boot.)
If you're sure there's no new updates you can even speed it up by adding '--skippkgs', so it won't bother looking for packages.
(In reply to comment #28)
> I have a single SSD with:
> sda1 - /boot no encryption
> sda2 - Encrypted with a single volume group and multiple logical volumes
> (root, home, swap)
> SELinux is disabled (at least for my F17 install).
How is it disabled? If SELinux is disabled in F17, fedup should add 'selinux=0' to the boot arguments automatically:
if not is_selinux_enabled():
Is 'selinux=0' in the fedup boot args? If not, what happens if you run:
selinuxenabled; echo $?
if SELinux is enabled that should return '0'. And 'selinux=0' should be in the fedup boot args. If it isn't, something went wrong.
> This seems same issue seems to affect me as well.
> When I boot into fedup I get a second or so of the progress bar followed by
> a screen full of asterisks (stars) even though I DID not hit ESC.
This could also happen if there's some other error output, which causes plymouth to try to switch to the text terminal, which is the same as hitting Esc.
> This continues for roughly an hour until it reboots. At that point
> nothing has been updated.
Can you add this to the boot args:
rd.upgrade.debug rd.upgrade.break=pre enforcing=0
If you boot like that, do you get a dracut debug shell (eventually)?
If not, the upgrade never actually started. If so, exit the shell and the upgrade should start; after the reboot, grab /var/log/upgrade.log and attach it here.
something similar has hit my already running luks protected F18 box.
I am not able to boot due to systemd-cryptsetup-generator complaining about some unexpected file, see https://bugzilla.redhat.com/show_bug.cgi?id=904332
I don't know how related my issue is to this one, but I am however able to boot the box now by adding "enforcing=0" as a boot param ...
(In reply to comment #30)
> (In reply to comment #28)
> > I have a single SSD with:
> > sda1 - /boot no encryption
> > sda2 - Encrypted with a single volume group and multiple logical volumes
> > (root, home, swap)
> > SELinux is disabled (at least for my F17 install).
> How is it disabled? If SELinux is disabled in F17, fedup should add
> 'selinux=0' to the boot arguments automatically:
> if not is_selinux_enabled():
> Is 'selinux=0' in the fedup boot args? If not, what happens if you run:
> selinuxenabled; echo $?
> if SELinux is enabled that should return '0'. And 'selinux=0' should be in
> the fedup boot args. If it isn't, something went wrong.
> > This seems same issue seems to affect me as well.
> > When I boot into fedup I get a second or so of the progress bar followed by
> > a screen full of asterisks (stars) even though I DID not hit ESC.
> This could also happen if there's some other error output, which causes
> plymouth to try to switch to the text terminal, which is the same as hitting
> > This continues for roughly an hour until it reboots. At that point
> > nothing has been updated.
> Can you add this to the boot args:
> rd.upgrade.debug rd.upgrade.break=pre enforcing=0
> If you boot like that, do you get a dracut debug shell (eventually)?
> If not, the upgrade never actually started. If so, exit the shell and the
> upgrade should start; after the reboot, grab /var/log/upgrade.log and attach
> it here.
Turns out my F17 grub2 install was slightly messed up and the fedup boot args we'ren't making it into the config. Once I corrected that everything worked as expected.
I suffer the same issue.
Having multiple encrypted partitions fails upgrading with FedUp. I am also not able to pass through disk partitioning using the LiveCd installation.
On machines with a single encrypted partition, moving selinux to permissive and disabling plymouth (remove rhgb/quiet), fedup works just fine.
On machines with multiple encrypted drivers (or worse, multiple encrypted drivers running on LVM/MD-raid), even manually mounting the encrypted partitions (within the systemd emergency console) doesn't seem to help.
As it stands, between the new Anaconda and fedup, its seems that Fedora 18 will be skipped on most of my machines :(
fedup-0.7.3-0.git20130128.fc17 has been submitted as an update for Fedora 17.
fedup-0.7.3-0.git20130128.fc17 doesn't fix the problem mentioned in comment 0.
fedup-0.7.3-0.git20130128.fc17 fixed the issue for me.
fedup-0.7.3-0.git20130128.fc17 fixed my issue too.
set up an fc17 with a regular partitions ( not lvm ) of encrypted root and encrypted swap
ran yum distro-sync, yum update, yum clean all, got fedup-0.7.3-0.git20130128.fc17.x86_64 from koji and installed it,
rebooted and ran:
fedup-cli --network 18
which downloaded packages, rebooted to upgrade, it asked for encryption password and continued to boot and sucessfully performed the upgrade, rebooted, it asked for encryption password and booted fine into fc18 :')
Let me try to clear things up a little. There's two LUKS-specific problems that we know of:
1) SELinux gets in the way of systemd-ask-password trying to use cached passwords
This is what happens when you have multiple LUKS partitions (all with the same passphrase) and encrypted root. That should be fixed in fedup-0.7.3-0.git20130128.fc17, and that's what I'm using this bug for.
2) plymouthd crashes when trying to ask for a password
This only happens if you *don't* have root encrypted (bug 894242, among others). The workaround is to remove 'rhgb' from the boot arguments; I'm working on a fix for that.
If adding 'enforcing=0' and removing 'rhgb' doesn't fix your problem, please file a new bug so it doesn't get lost!
I reported bug 910326 about the problem in comment 0.
fedup-0.7.3-1.fc17 has been submitted as an update for Fedora 17.
* should fix your issue,
* was pushed to the Fedora 17 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing fedup-0.7.3-1.fc17'
as soon as you are able to.
Please go to the following url:
then log in and leave karma (feedback).
fedup-0.7.3-3.fc18 has been submitted as an update for Fedora 18.
fedup-0.7.3-3.fc17 has been submitted as an update for Fedora 17.
fedup-0.7.3-4.fc17 has been submitted as an update for Fedora 17.
fedup-0.7.3-4.fc18 has been submitted as an update for Fedora 18.
fedup-0.7.3-4.fc19 has been submitted as an update for Fedora 19.
fedup-0.7.3-4.fc18 has been pushed to the Fedora 18 stable repository. If problems still persist, please make note of it in this bug report.
*** Bug 904253 has been marked as a duplicate of this bug. ***
fedup-0.7.3-5.fc17 has been submitted as an update for Fedora 17.
fedup-0.7.3-4.fc19 has been pushed to the Fedora 19 stable repository. If problems still persist, please make note of it in this bug report.
fedup-0.7.3-5.fc17 has been pushed to the Fedora 17 stable repository. If problems still persist, please make note of it in this bug report.