Bug 677438

Summary: systemd does not support plymouth password caching
Product: [Fedora] Fedora Reporter: Thomas Meyer <thomas.mey>
Component: systemdAssignee: Lennart Poettering <lpoetter>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 15CC: lpoetter, metherid, mschmidt, notting, plautrba, tomek
Target Milestone: ---   
Target Release: ---   
Hardware: i686   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-08-17 21:17:27 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 679907    
Bug Blocks:    

Description Thomas Meyer 2011-02-14 19:35:11 UTC
Description of problem:
I have two encrypted partitons:
$ cat /etc/crypttab
luks-home /dev/sda3
luks-swap /dev/sda4
(both partitions use the same key).

in the rawhide boot process I need to enter the password three times (in Fedora 14 I only need to enter the password once).

I guess this is what happens:
1.) dracut asks for password and uses this password for both partition

then systemd kicks in and ask for each entry in the crypttab file for a password.

the funny thing about systemd is, that when it asks for the password, it prints the name of my harddrive (something like TOSHIBA MKxxxx), which is not very useful when you want to distinguish what partiton is actually asked for.

AND after entering the password in systemd two times (for each partition), the password is ask a third time, and you can enter as many times the password as you want (the boot process continues after entering the password for each partiton mentioned in /etc/crypttab - i.e. 2 times).

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

Comment 1 Thomas Meyer 2011-02-20 19:05:03 UTC
situation got worse after upgrading to systemd-18-1.fc15. Now I get the dracut password question, and systemd seems to not ask for the password anymore. I get to the shell but /home isn't mounted. This is what I see in /var/log/messages (grep for systemd):

Feb 20 19:44:59 localhost kernel: [    0.000000] Kernel command line: root=UUID=5764c1f2-f119-4f97-b8e2-d827b8655edb LANG=de_DE.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=de-latin1-nodeadkeys kgdboc=kms,kbd resume=/dev/mapper/luks-swap rhgb quiet systemd.unit=multi-user.target
Feb 20 19:44:59 localhost kernel: [   13.819790] systemd[1]: systemd 18 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX +SYSVINIT +LIBCRYPTSETUP; fedora)
Feb 20 19:44:59 localhost kernel: [   14.123952] systemd[1]: Set hostname to <localhost.localdomain>.
Feb 20 19:44:59 localhost kernel: [   16.688607] systemd[1]: Found ordering cycle on nfslock.service/start
Feb 20 19:44:59 localhost kernel: [   16.688617] systemd[1]: Walked on cycle path to network.target/start
Feb 20 19:44:59 localhost kernel: [   16.688624] systemd[1]: Walked on cycle path to NetworkManager.service/start
Feb 20 19:44:59 localhost kernel: [   16.688632] systemd[1]: Walked on cycle path to lm_sensors.service/start
Feb 20 19:44:59 localhost kernel: [   16.688639] systemd[1]: Walked on cycle path to nfslock.service/start
Feb 20 19:44:59 localhost kernel: [   16.688646] systemd[1]: Breaking ordering cycle by deleting job network.target/start
Feb 20 19:44:59 localhost kernel: [   21.612546] type=1400 audit(1298227440.453:5): avc:  denied  { write } for  pid=755 comm="systemd-tty-ask" name="sck.3301086953241499494" dev=tmpfs ino=11636 scontext=system_u:system_r:systemd_passwd_agent_t:s0 tcontext=system_u:object_r:device_t:s0 tclass=sock_file
Feb 20 19:44:59 localhost kernel: [   21.755719] systemd-fsck[735]: /dev/sda1: sauber, 69/122400 Dateien, 155405/488248 Blöcke
Feb 20 19:44:59 localhost kernel: [   76.689044] systemd[1]: Job dev-mapper-luks\x2dhome.device/start timed out.
Feb 20 19:44:59 localhost kernel: [   77.058484] systemd[1]: Job dev-mapper-luks\x2dswap.device/start timed out.
Feb 20 19:45:00 localhost systemd[1]: kdump.service: control process exited, code=exited status=1
Feb 20 19:45:00 localhost systemd[1]: Unit kdump.service entered failed state.
Feb 20 19:45:56 localhost systemd[1]: Job dev-mapper-luks\x2dhome.device/start timed out.
Feb 20 19:45:56 localhost systemd[1]: Job dev-mapper-luks\x2dswap.device/start timed out.
Feb 20 19:45:56 localhost systemd[1]: Startup finished in 2s 662ms 439us (kernel) + 11s 484ms 888us (initrd) + 2min 3s 486ms 933us (userspace) = 2min 17s 634ms 260us.
Feb 20 19:47:22 localhost systemd[1]: Job dev-mapper-luks\x2dswap.device/start timed out.

Any ideas?

Comment 2 Lennart Poettering 2011-02-23 02:52:54 UTC
systemd git now supports dealing with passwords cached by plymouth. That should have the effect that you no longer need to enter the same password for all disks.

dracut is not involved with mounting non root disks.

The issue described in #1 is probably https://bugzilla.redhat.com/show_bug.cgi?id=679321 -- a bug in lvm.

Comment 3 Thomas Meyer 2011-02-23 17:52:23 UTC
A few remarks from my side:

1.) I don't use LVM
2.) dracut is not responsible for mounting the non-root fs, but I takes care of the setup of the LUKS mappings for non-root-fs (and swap devices). This function may should be removed. dracut should only try to cryptsetup the root fs (and maybe the resume (swap) device as systemd takes care of the non-root-LUKS devices (when a correspondig /etc/crypttab exists)
3.) I still don't get the plymouth password screen with systemd-18! I cannot enter a password. This is also true for booting without the "rghb" option. I don't have a chance to enter the cryptsetup password, so I guess this is why I hit a timeout and the mounting of /home (in /etc/fstab) fails, as you can see in above log grep. did something brake in plymouth? koji lists the package plymouth-0.8.4-0.20110209.2.fc15 build on 10.02.2011. my bug report is from 14.02.2011 and there the plymouth password ask screen used to work. maybe the package did take a few days to get into update-testing and the all mirrors? I could try to downgrade to a previous version and see if this works.

Comment 4 Lennart Poettering 2011-02-23 20:44:55 UTC
(In reply to comment #3)

> 2.) dracut is not responsible for mounting the non-root fs, but I takes care of
> the setup of the LUKS mappings for non-root-fs (and swap devices). This
> function may should be removed. dracut should only try to cryptsetup the root
> fs (and maybe the resume (swap) device as systemd takes care of the
> non-root-LUKS devices (when a correspondig /etc/crypttab exists)

It normally does that. But for that to work it needs information passed via the kernel cmdline which on which luks device it needs to look at (rd_LUKS_UUID=). Anaconda normally configures things like this, If this didn't happen for you this might be because your installation is very old or was later on converted to a encryption?

There's generally the problem that you have no clue what is stored on the disk before you have decrypted. So how would you know that the disk with the fs label "foo" is on that luks device if you can't read the label because it is encrypted on disk?

> 3.) I still don't get the plymouth password screen with systemd-18! I cannot
> enter a password. This is also true for booting without the "rghb" option. I
> don't have a chance to enter the cryptsetup password, so I guess this is why I
> hit a timeout and the mounting of /home (in /etc/fstab) fails, as you can see
> in above log grep. did something brake in plymouth? koji lists the package
> plymouth-0.8.4-0.20110209.2.fc15 build on 10.02.2011. my bug report is from
> 14.02.2011 and there the plymouth password ask screen used to work. maybe the
> package did take a few days to get into update-testing and the all mirrors? I
> could try to downgrade to a previous version and see if this works.

Hmm? in your initial bug report you mentioned that both dracut and systemd ask for the password, and now you don't get any password screens at all anymore?

Could you please write down exactly what your problems are, I have trouble understanding what works and what doesn't.

Comment 5 Lennart Poettering 2011-02-23 20:45:41 UTC
BTW, the systemd fix for cached passwords cannot actually work without https://bugzilla.redhat.com/show_bug.cgi?id=679907 fixed first.

Comment 6 Thomas Meyer 2011-02-23 21:53:06 UTC
(In reply to comment #4)

> Hmm? in your initial bug report you mentioned that both dracut and systemd ask
> for the password, and now you don't get any password screens at all anymore?
> 
> Could you please write down exactly what your problems are, I have trouble
> understanding what works and what doesn't.

yes. my initial bug report was done against systemd-17.5.fc15. a few days later systemd-18.1.fc15 was available on updates-testing. In version 18 the password-ask thing is broken. I neither get a graphical question, nor a text only question from plymouth (without rhgb boot option).

Currently I use systemd-17.5 (after a yum downgrade; the working version). This is what I wanted to say in comment #1.

in version 18 the jobs seems to get started "Feb 20 19:45:56 localhost systemd[1]: Job dev-mapper-luks\x2dswap.device/start timed out." but I never get to enter a password as there is no password question/password screen anymore!

Comment 7 Michal Schmidt 2011-02-24 09:12:14 UTC
(In reply to comment #6)
> in version 18 the jobs seems to get started "Feb 20 19:45:56 localhost
> systemd[1]: Job dev-mapper-luks\x2dswap.device/start timed out."

This really looks like bug 679321. Though it has LVM in the title, it's in fact an issue with udev and it affects all device-mapper devices (even non LVM). If it's it, a known workaround is to rebuild your initramfs (in order to have the same version of udev in the initramfs and the root fs).

Comment 8 Lennart Poettering 2011-02-24 15:13:41 UTC
Thomas, let's forget about the bugs in systemd 17, let's focus on systemd 18 now.

So, I agree with Michal, the issue you have with devices never being seen is most likely a DM problem, and for that there is bug 679321.

Let's keep this bug about systemd not supporting password caching and asking you a couple of times even if you use the same password everywhere.

Comment 9 Thomas Meyer 2011-02-24 17:25:22 UTC
1.) I upgraded everything to the current versions in update-testing, i.e. systemd-18
2.) I ran udevadm info --update-db (just to be sure)
3.) udev version in initramfs and in / (rootfs) are the same (166-1.fc15)
4.) dracut password entry works always (w/rhgb + w/quiet, w/rhgb + wo/quiet, wo/rhgb + wo/quiet)
4.) Many reboots later:

Booting

1) with "rhgb" and with "quiet": I only saw the password entry screen once in many tries. The normal result is a boot screen, that switches in some point of time into a black screen (I assume this is the time I should enter the password, but I cannot see anything (black screen), blindly typing the password doesn't seem to change anything. password job times out, boot continues then).

2.) with "rhgb" and without "quiet" option: same behaviour as in 1) but I never saw the password entry screen at all

3.) without "rhgb" and without "quiet" option: I see the "enter password" text. I start to enter the password the first few characters are echoed to the screen, then the echoing stops. When I continue to enter the password nothing seem to happen. The enter key doesn't seem to have any effect.

Something in the call to ask-password (i.e. plymouth) got broken between systemd-17 and systemd-18 (maybe a tty problem?).

PS: The one boot try w/rhgb and w/quiet that I saw the plymouth screen I could enter the password twice (as I have to /etc/crypttab entries). But the password entry was somehow strange: I typed the password and press enter. No black dots were echoed while typing. After pressing enter, for each char in the password a black dot was painted on the screen (very quickly), but only after pressing the enter key.

Comment 10 Lennart Poettering 2011-02-25 01:01:38 UTC
seems like there's something wrong with plymouth the way you explain this. Hmm, I wonder if this is triggered because in some situations we start getty too early.

Do you boot into gdm or is this a getty only systemd?

Comment 11 Thomas Meyer 2011-02-25 19:19:48 UTC
I boot into the multi-user.target.

Comment 12 Thomas Meyer 2011-02-26 20:36:10 UTC
renaming both luks devices in /etc/crypttab from

luks-home to luksHome
luks-swap to luksSwap

and change the corresponding mount entries in /etc/fstab I get this status after booting to the graphical.target:

$ systemctl --full | grep cryptsetup
cryptsetup loaded active     exited        Cryptography Setup for luksHome
cryptsetup loaded activating start         Cryptography Setup for luksSwap
cryptsetup.target         loaded active     active        Encrypted Volumes

I need to enter the luks password just once in dracut! (that still is the only password dialog I see in the boot process).

But only home seems to get setup via crypt:

$ ll /dev/mapper/lu*
lrwxrwxrwx. 1 root root 7 26. Feb 21:17 /dev/mapper/luksHome -> ../dm-0

luksHome is setup correctly in cryptsetup and /home is mounted!
But I get no swap device:
$ swapon -s
Filename				Type		Size	Used	Priority

So the name of the luks device in /etc/crypttab seems change the behaviour of systemd.

the name "luks-home" results in the service file "cryptsetup@luks\x2dhome.service", so the string "\x2d" in the service name seems to cause some problem?!

Comment 13 Thomas Meyer 2011-03-02 20:24:14 UTC
After upgrading to systemd-19 I don't even have any gettys anymore!
Luckily I can use the gdm to log into the system!

systemctl tells me:
$ systemctl --full | grep waiting
systemd-ask-password-wall.path loaded active     waiting         Forward Password Requests to Wall Directory Watch
systemd-tmpfiles-clean.timer loaded active     waiting         Daily Cleanup of Temporary Directories

$ systemctl --full | grep dead
getty        loaded inactive   dead      start Getty on tty2
getty        loaded inactive   dead      start Getty on tty3
getty        loaded inactive   dead      start Getty on tty4
getty        loaded inactive   dead      start Getty on tty5
getty        loaded inactive   dead      start Getty on tty6
systemd-update-utmp-runlevel.service loaded inactive   dead      start Notify Audit System and Update UTMP about System Runlevel Changes
getty.target              loaded inactive   dead      start Login Prompts
graphical.target          loaded inactive   dead      start Graphical Interface
multi-user.target         loaded inactive   dead      start Multi-User

$ systemctl --full | grep fail
cryptsetup loaded failed     failed          Cryptography Setup for luksSwap

I still don't see a plymouth password prompt in systemd. I only get the one from dracut.

Comment 14 Thomas Meyer 2011-03-02 20:27:44 UTC
by the way: should I care about this message:

[   22.304374] type=1400 audit(1299096305.522:10): avc:  denied  { write } for  pid=675 comm="systemd-tty-ask" name="sck.11215492513668470035" dev=tmpfs ino=10904 scontext=system_u:system_r:systemd_passwd_agent_t:s0 tcontext=system_u:object_r:device_t:s0 tclass=sock_file

Comment 15 Thomas Meyer 2011-03-02 20:32:46 UTC
there are many more of this "denied" messages:

[   17.491987] type=1400 audit(1299099900.706:3): avc:  denied  { add_name } for  pid=384 comm="mount" name=".mount" scontext=system_u:system_r:mount_t:s0 tcontext=system_u:object_r:tmpfs_t:s0 tclass=dir

[   17.492011] type=1400 audit(1299099900.706:4): avc:  denied  { create } for  pid=384 comm="mount" name=".mount" scontext=system_u:system_r:mount_t:s0 tcontext=system_u:object_r:tmpfs_t:s0 tclass=dir

[   17.492110] type=1400 audit(1299099900.706:5): avc:  denied  { create } for  pid=384 comm="mount" name="utab" scontext=system_u:system_r:mount_t:s0 tcontext=system_u:object_r:tmpfs_t:s0 tclass=file

[   17.615836] type=1400 audit(1299099900.829:6): avc:  denied  { read write } for  pid=386 comm="loadkeys" name="tty" dev=tmpfs ino=5955 scontext=system_u:system_r:loadkeys_t:s0 tcontext=system_u:object_r:device_t:s0 tclass=chr_file

[   17.615853] type=1400 audit(1299099900.829:7): avc:  denied  { open } for  pid=386 comm="loadkeys" name="tty" dev=tmpfs ino=5955 scontext=system_u:system_r:loadkeys_t:s0 tcontext=system_u:object_r:device_t:s0 tclass=chr_file

[   17.615885] type=1400 audit(1299099900.829:8): avc:  denied  { ioctl } for  pid=386 comm="loadkeys" path="/dev/tty0" dev=tmpfs ino=5958 scontext=system_u:system_r:loadkeys_t:s0 tcontext=system_u:object_r:device_t:s0 tclass=chr_file

[   19.892522] type=1400 audit(1299096303.108:9): avc:  denied  { mmap_zero } for  pid=422 comm="vbetool" scontext=system_u:system_r:vbetool_t:s0-s0:c0.c1023 tcontext=system_u:system_r:vbetool_t:s0-s0:c0.c1023 tclass=memprotect

[   22.304374] type=1400 audit(1299096305.522:10): avc:  denied  { write } for  pid=675 comm="systemd-tty-ask" name="sck.11215492513668470035" dev=tmpfs ino=10904 scontext=system_u:system_r:systemd_passwd_agent_t:s0 tcontext=system_u:object_r:device_t:s0 tclass=sock_file

[   23.366421] type=1400 audit(1299096306.582:11): avc:  denied  { write } for  pid=702 comm="systemd-tmpfile" name="log" dev=tmpfs ino=7644 scontext=system_u:system_r:systemd_tmpfiles_t:s0 tcontext=system_u:object_r:tmpfs_t:s0 tclass=sock_file

[   82.216296] type=1400 audit(1299096365.430:12): avc:  denied  { write } for  pid=722 comm="rtkit-daemon" name="log" dev=tmpfs ino=7644 scontext=system_u:system_r:rtkit_daemon_t:s0 tcontext=system_u:object_r:tmpfs_t:s0 tclass=sock_file

[   82.533220] type=1400 audit(1299096365.745:13): avc:  denied  { write } for  pid=675 comm="systemd-tty-ask" name="sck.11419201820819665340" dev=tmpfs ino=12895 scontext=system_u:system_r:systemd_passwd_agent_t:s0 tcontext=system_u:object_r:device_t:s0 tclass=sock_file

[   83.942209] type=1400 audit(1299096367.155:14): avc:  denied  { write } for  pid=799 comm="auditd" name="log" dev=tmpfs ino=7644 scontext=system_u:system_r:auditd_t:s0 tcontext=system_u:object_r:tmpfs_t:s0 tclass=sock_file

Comment 16 Thomas Meyer 2011-03-02 20:35:35 UTC
plymouth seems still to run:
$ pidof plymouth
915

after killing it via "sudo kill 915", I get my gettys back!

Comment 17 Michal Schmidt 2011-03-03 09:38:17 UTC
Whenever you're in doubt whether a problem is related to SELinux, retry in permissive mode (set it in /etc/selinux/config, or boot with "enforcing=0").

Comment 18 Lennart Poettering 2011-03-09 14:00:15 UTC
The plymouth getting stuck issue is tracked in bug 679503. Let's keep this bug about the missing password caching.

Comment 19 Thomas Meyer 2011-06-20 21:32:29 UTC
This password caching doesn't seem to work on my laptop (updated to Fedora 15 via yum update). I need to enter the password twice: One password question is asked by dracut, and then systemd asks again for the password. Any ideas how to debug this?

Shouldn't the password asked by dracut be cached by plymouth and reused in systemd?

Comment 20 Thomas Meyer 2011-08-17 21:17:27 UTC
Hello,

password caching works now for me, the problem is that dracut only seems to work correctly when *every* entry in /etc/crypttab has an newline as last character. I had a newline missing in the second and last line of my /etc/crypttab:

$ cat /etc/crypttab 
luksHome /dev/sda3
luksSwap /dev/sda4 -> newline was missing

I guess the missing newline did confuse dracut and so only the luksHome device was setup via cryptsetup.

"man crypttab" doesn't explicitly stated that a newline is needed for every new entry in this file.

the script "sbin/cryptroot-ask" in the initramfs seems to do a "while read" on /etc/crypttab.
I guess this while loop misses the second line as EOF is encountered!?

So the luksSwap device was not marked via file "/tmp/cryptroot-asked-luksSwap"(?) as already asked for.

I guess this is why systemd did ask for the luksSwap password again!

maybe a check could be added somewhere that every line in /etc/crypttab ends with a newline character.

interestingly systemd had no problem with the missing newline and did correctly setup the crypt device.