Red Hat Bugzilla – Bug 1405539
changing the default keyboard layout changes also disk decryption in plymouth, but only after kernel update, long after
Last modified: 2017-09-06 10:42:09 EDT
Description of problem:
This is based on real issues where users encountered seemingly random inability to unlock their disks during boot anymore. I've traced the issue to keyboard layout changes which affect disk decryption at a much later time, thus being not attributed to this change by users at all.
Imagine the following scenario:
1. Your system is installed with keymap X. The system has its disks encrypted, you're used to decrypt them using keymap X.
2. Some time later, you decide to add keymap Y in the gnome-control-center. For any arbitrary reason you set it as the first keymap in the list (maybe you like it much better, maybe you're really serious about learning new language, maybe you have no idea what the ordering is for, maybe you do it by accident).
3. Everything works, your disks are still being unlocked with the same keyboard sequence (using keymap X), the default keymap in the booted system is keymap Y (but you might not even notice this, because your GNOME session remembers the last used keymap, so the default keymap doesn't really affect it in any way).
4. As a responsible user, you update your system regularly using "Install pending software updates" checkbox when powering off. Everything works splendid.
5. After one such update, you can't boot your system anymore. The system keeps asking you for a disk decryption password during boot, but no matter how many times you provide it (correctly!) it asks again and does not proceed further.
6. You panic. Your system is bricked. You can't access your data. The last update completely broke your computer. Bad Fedora!
Point 5 can happen weeks or months after point 2. There's no way you connect the inability to boot to the fact you changed your default keyboard layout weeks or months earlier. Anyone debugging your system (if you don't give up on Fedora right away) will be looking for completely different issues. And even if you realize what the problem is, you might not be able to input the original password with the new keyboard layout, because not all characters might be there.
The most significant issues and questions are:
* The keymap in plymouth is not changed immediately with the rest of the system, but only after kernel update (which might happen long after the action). That seems to be a fault of gnome-control-center, which is in charge of this (or perhaps some low level tools and libraries which it uses).
* gnome-control-center changes the default *system* keymap without being clear about it. This is fairly recent change (a year or so); in the past there used to be a button where you'd change between "login screen" (meaning system) settings and your personal settings. That continues to be present on multi-user systems, but for single-user systems it presents a single interface which changes both system and personal preferences, without informing you about it. It's no longer possible to change only your personal settings on a single-user system, even if you wanted.
* The user is not informed at all that changing system keymap affects disk decryption process. It is especially unclear that only the first keymap in the list is the one that is used to unlock the disks.
* Do we want default keymap change to affect plymouth at all? I guess we do, but I'm not certain.
I believe we should either:
a) make sure plymouth is not affected by the changed system keymap (if we decide this is a good idea). This would need changes in plymouth/dracut or perhaps setting appropriate kernel cmdline args (we currently do set LANG, but we don't set vconsole)
- or -
b) improve how gnome-control-center works. This involves:
* only change personal preferences by default, and make changing the system preferences a separate, distinguishable action (a simple solution would be to revert the simplification change that happened some time ago)
* clearly warn the user that changing system keymaps affects the disk decryption process and that the user might no longer be able to unlock it. The issue is further complicated by the fact that console keymaps are not always the same as their matching X11 keymaps, which we might include in the warning.
* if system keymaps are changed, make sure the changes affect plymouth immediately (and not just after next kernel update)
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. install clean Fedora 25 Workstation using keymap X (I'll use US in this example). Encrypt your disk with a password (I used "qwerty").
2. go to GNOME control center -> Region & Language -> Input sources and add keymap Y. The new keymap should have at least some of your password's characters differently placed than the original keymap. I added French (because they have Q swapped with A and W swapped with Z).
3. Move the new keymap to the top of the list, possibly because you think it'll make it the default when you log in. This is how keymaps look like in my case:
System Locale: LANG=en_US.UTF-8
VC Keymap: fr
X11 Layout: fr,us
X11 Variant: ,
3. reboot and make sure you still input the same physical buttons when unlocking the disk (so still "qwerty", as seen on the US keyboard). It should work.
4. update the kernel, or run "sudo dracut -f". That will regenerate initramfs.
5. reboot and find out that the usual sequence "qwerty" does no longer unlock the disk. Instead, you need to input the password using the new keymap you added (in this case, "azerty", as seen on the US keyboard). In case you added a keymap which can't input those characters (in this case no support for latin characters; or maybe you encrypted your disks with "koníček" password originally and then added a standard US keyboard), well, bad luck.
I'm tagging this for a blocker discussion for F26 Final. The reproduction steps are very simple to trigger for an unsuspecting user, and the consequences are extremely harsh and hard to fix. This might be classified under "basic functionality of default apps" criterion  where gnome-control-center is a default app and managing keyboard layouts is a basic functionality, which is neither working properly (the changes are not immediately effective in plymouth) nor as expected (the expected outcome might very often be just a change of keymaps in your current session and not a change in disk decryption process).
"The keymap in plymouth is not changed immediately with the rest of the system, but only after kernel update (which might happen long after the action). That seems to be a fault of gnome-control-center, which is in charge of this (or perhaps some low level tools and libraries which it uses)."
This is likely because the host system keyboard layout config is copied into the initramfs when it's generated. Sounds like GNOME is set up (likely through localectl) to change /etc/vconsole.conf to place the 'matching' console layout for whatever layout is placed at the top of GNOME's list in /etc/vconsole.conf ? Have a look at that file when changing layouts and see what happens to it.
This may in fact vary depending on whether the user is an admin and whether there are multiple users on the system.
Another note: it *shouldn't* ever be possible to get a console layout that can't input ASCII characters, if you can achieve that, you found a different bug, and anaconda warns about (but does not prohibit) passphrases with non-ASCII characters.
(In reply to Adam Williamson from comment #2)
> This is likely because the host system keyboard layout config is copied into
> the initramfs when it's generated. Sounds like GNOME is set up (likely
> through localectl) to change /etc/vconsole.conf to place the 'matching'
> console layout for whatever layout is placed at the top of GNOME's list in
> /etc/vconsole.conf ?
Yes, that's exactly what happens. With just "us" layout, vconsole.conf looks like this:
When I add "fr" layout and place it on top, vconsole.conf looks like this:
It's clear that this change affects plymouth only after initramfs is regenerated (on next kernel update).
> This may in fact vary depending on whether the user is an admin and whether
> there are multiple users on the system.
Yes, as I described in comment 0, this is mainly an issue for single-user systems. For multi-user systems, you change only your personal session settings by default and there's a "Login screen" button to switch to system mode. (However, that still suffers from the problem that you're not informed this will affect encrypted disk unlocking, and still takes affect only after a kernel update).
I don't see what gnome-control-center could do more here. Sounds like a problem with plymouth only ever having updated keyboard layout settings when a new kernel is installed or the initramfs is regenerated.
(In reply to Adam Williamson from comment #3)
> Another note: it *shouldn't* ever be possible to get a console layout that
> can't input ASCII characters, if you can achieve that, you found a different
> bug, and anaconda warns about (but does not prohibit) passphrases with
> non-ASCII characters.
That's good to know. But since we don't see the current keymap nor can test it in plymouth during disk unlocking, this help is very limited (it's good that you can always input ASCII, but if you don't know which keymap is active and don't remember the layout fully, you can still easily be unable to do it - keys are often swapped from their usual us-like position). Also, console keymaps are different from X11 keymaps (e.g. "ru" defaults to cyrillic in X11, but to ascii in console), so you might be completely unaware of this. That's why I believe we should very visibly warn the user that changing the default system keymap has significant consequences if the disk encryption is present, and require this change to be clearly intentional.
(In reply to Bastien Nocera from comment #5)
> I don't see what gnome-control-center could do more here.
As described above, it could default to changing just user settings by default, even for single-user systems. Also, it should visibly warn the user that changing default system keymap has important consequences for encrypted systems.
Btw, I don't think this is plymouth's fault, because it can't access /etc/vconsole.conf when the disks are locked. Initramfs regeneration must be triggered from gnome-control-center or a tool that gnome-control-center calls to change the layout configuration (localectl?).
(In reply to Kamil Páral from comment #7)
> (In reply to Bastien Nocera from comment #5)
> > I don't see what gnome-control-center could do more here.
> As described above, it could default to changing just user settings by
> default, even for single-user systems. Also, it should visibly warn the user
> that changing default system keymap has important consequences for encrypted
How would it know?
> Btw, I don't think this is plymouth's fault, because it can't access
> /etc/vconsole.conf when the disks are locked. Initramfs regeneration must be
> triggered from gnome-control-center or a tool that gnome-control-center
> calls to change the layout configuration (localectl?).
gnome-control-center just calls locale1. If setting the system layout doesn't actually work because of the way things are implemented underneath, then nothing much that gnome-control-center can do.
(I think plymouth should use libxkbcommon. I mean, we had a boot splash using a full Xorg and it wasn't that bad).
(In reply to Bastien Nocera from comment #8)
> How would it know?
I assume it should be possible to figure out if any of the mounted partitions have an underlying device of the crypto_LUKS fstype (as seen in lsblk). If that is not possible or too error-prone, then at least warning the user unconditionally would do the job ("The first keyboard layout in the list is used to unlock encrypted disks during boot, in case you use this feature. The during-boot layout might differ from the layout used in a graphical session. Be careful when configuring this.").
Of course it would help tremendously if the system configuration was not edited by default for single user sessions, that would reduce the number of problems. Currently it's not even possible to set a session keymap without affecting global keymaps on single-user systems.
All of the above is gnome-control-center UI issues, that's why I set this to g-c-c initially. The UI is one part of the problem, initramfs regeneration is the second part.
> gnome-control-center just calls locale1. If setting the system layout
> doesn't actually work because of the way things are implemented underneath,
> then nothing much that gnome-control-center can do.
Yes, there are multiple projects involved in this. Maybe we need to ask locale1 (not sure what that is, doesn't seem to be a package or a binary, maybe a dbus service?) to provide an option to regenerate initramfs after editing vconsole.conf?
> (I think plymouth should use libxkbcommon. I mean, we had a boot splash
> using a full Xorg and it wasn't that bad).
That's out of my area of expertise, but even if it used a full Xorg stack, the config files still need to be present in initramfs, right? Which they currently aren't (until you install a new kernel). That's why I commented that plymouth is probably not right component to assign this to, but instead it should be the tool that edits vconsole.conf (and should regenerate initramfs immediately).
"(I think plymouth should use libxkbcommon. I mean, we had a boot splash using a full Xorg and it wasn't that bad)."
Well, IIRC, three years ago, we were going to have kmscon next year (which was supposed to fix this, along with every other problem caused by kernel consoles being kinda crappy). Then it became systemd-consoled , then it sort of died.
This is really just another in the very large category of "problems caused by kernel console character input and display not working anything like graphical desktop character input and display", but no-one seems inclined to invest the necessary resources to really fix that properly :/
This could match the purpose of PrioritizedBug tracker, adding a keyword. I can't make a tldr version really, please refer to comment 0, thanks.
Discussed during the 2017-01-09 blocker review meeting: 
The decision to delay the classification of this as a bug was made as a unanimous decision regarding this bug could not be made. Time is being appointed in which the bug can be considered and different viewpoints can be gathered.
The bug has been approved as a Fedora Priority Bug: https://fedoraproject.org/wiki/Fedora_Program_Management/Prioritized_bugs_and_issues
See also: https://bugzilla.redhat.com/show_bug.cgi?id=1059485
Discussed during the 2017-02-13 blocker review meeting: 
The decision was made to delay the classification of this as a blocker and get more clarification on this bug from the g11n and desktop teams before voting.
Discussed during the 2017-02-27 blocker review meeting: 
The decision to delay the classification of this as a bug was made as we did not get the information we needed from the g11n team. We will look at this bug again next meeting.
It's a major data loss bug that will affect anybody who changes the system keyboard layout after installing. It should be a blocker.
Discussed during the 2017-03-06 blocker review meeting: 
The decision was made to delay the classification of this as a bug as we still don't have the information we need to make a decision.
What further information is requested...? The situation is really clear: if you change your system keyboard layout, you're going to be locked out of your computer after the next kernel update. This violates our data loss criterion.
But IIRC others on the desktop team argued against this being a blocker, so we already have a disagreement just within the desktop team.
We're not actually waiting for *information*: we're waiting for *opinions*. We delegated someone (Kohane) to ask the g11n teams for their opinions on this, since it seemed reasonable that they might be in the best position to know or be able to guess how big of a problem this is in the real world (i.e. how often do people change keyboard layouts post-install without understanding this consequence of the change). We're waiting for the outcome of that.
Here is an opinion, my personal: This is an ugly bug that may be really be annoying on a production system. I am changing my keyboard layout (adding two more to the standard en_US) after every install.
Please make it a blocker bug!
This sounds pretty bad to me. I agree that something should be done about this.
So what should be the right/expected behaviour here?
That part is not completely clear to me yet. systemd?
(In reply to Jens Petersen from comment #22)
> This sounds pretty bad to me. I agree that something should be done about
> So what should be the right/expected behaviour here?
> That part is not completely clear to me yet. systemd?
That's the main problem. We all know the current behavior is unacceptable, but nobody seems to know what the expected behavior should be. It doesn't seem correct for the keyboard layout used for plymouth to be different from the system keyboard layout, for example, but that would be the most obvious way to fix this issue.
The heart of the issue is that plymouth does not display the current keyboard layout, or allow changing it. But that would be difficult to change.
= regenerating the initramfs =
I don't think regenerating the initramfs is something that we'd want to do, even if we had the machinery to do it. There are many components which influence the initramfs, and we do not recreate the initramfs automatically after updating any of them, except for the kernel. This is certainly debatable, since there are cases where it'd be nice to have the initramfs automatically updated, but there are good reasons not to do it, and at least current rules are clear: new kernel, new initramfs, other changes, old initramfs.
Also, even if we updated the initramfs, that'd be just the latest kernel (or the current kernel?), and the user could subsequently boot into a different kernel, and see the old keyboard settings. Updating all kernels would imho be a very bad, because it's slow and is incompatible with the idea of keeping old kernel+initramfs pairs as backup in case of regressions. If we updated multiple kernels automatically, people would be very unhappy.
Also, implementation would be a bitch ;) Right now the initramfs is automatically updated from within rpm. Doing it automatically from within systemd-localed means a completely different permission model, selinux configuration, and it'd be a break from systemd-localed being a simple distro-agnostic program. It doesn't seem like a good idea at all.
= not setting the system mapping =
> Of course it would help tremendously if the system configuration was not edited by default for single user sessions, that would reduce the number of problems. Currently it's not even possible to set a session keymap without affecting global keymaps on single-user systems.
I'd lean this way too. I think explicit is better than implicit here, and people in general should be fine with keeping the keymap that the system was installed with unless explicitly changed. Not my area of expertise though.
= other options =
I think we should show they keymap in the password prompt. Possibly only if the user mistypes the password and the prompt is shown for the second time. We have (had?) something like this with CapsLock, where a little icon would be shown, and it helps a lot. This would apply for both the graphical prompt under plymouth and the text prompt under systemd-ask-password. In case it is decided that this is the way to go, I'll work on the patch for systemd-ask-password-console.
Yet another useful feature would be an option to reveal the password. Like the little eye icon that the competition has, which you click and you see the actual letters. I don't think we can do this on the console (unless through some magic key combo which would be hard to discover), but it would be doable under plymouth.
(In reply to Zbigniew Jędrzejewski-Szmek from comment #24)
> = not setting the system mapping =
> > Of course it would help tremendously if the system configuration was not edited by default for single user sessions, that would reduce the number of problems. Currently it's not even possible to set a session keymap without affecting global keymaps on single-user systems.
On a single-user system, it's really confusing to change your locale and then discover that the login screen didn't change. You basically never want to have separate system/user settings if there is only one user.
I like the idea of Plymouth showing which keyboard layout is being used.
(In reply to Michael Catanzaro from comment #26)
> On a single-user system, it's really confusing to change your locale and
> then discover that the login screen didn't change. You basically never want
> to have separate system/user settings if there is only one user.
I understand the intention here and it makes sense from this POV (even though I consider the 'login screen' button quite clear and not confusing). However, I had the opposite experience with the current dialog design - I was extremely confused why the dialog seems to do different things on different systems, and just after 1-2 hours I discovered it depends on the number of users in your system (something that wouldn't have ever occurred to me). So talk about confusing... maybe we can improve the dialog design then, if not functionality? When changing system settings, make it clearer that you're doing so, and for keymap order, make it clear that it affects both login screen and disk decryption screen (if LUKS is detected)?
(In reply to Zbigniew Jędrzejewski-Szmek from comment #24)
> I think we should show they keymap in the password prompt.
(In reply to Zbigniew Jędrzejewski-Szmek from comment #25)
> Yet another useful feature would be an option to reveal the password.
Both of these would be seriously awesome.
My opinion is that is hard enough to fix that it's unlikely to get traction this release.
To do it right, we need an indicator in plymouth. We'd have to start with adding a kernel api for querying the console keymap. We don't currently show fonts in the initramfs, so the indicator would have to be image based. And at the moment, switching console keymaps requires knowing a keymap specific key combination, which isn't shared with X. So having the indicator still leaves the user somewhat trapped, if they don't know the combo.
We could make it plymouth use /dev/input instead of the console, but that would require extensive changes to the code base.
Rebuilding the initramfs from systemd or control-center doesn't really solve anything. User may still reboot weeks after changing it and not realize the change affects boot screen.
Not changing the keymap for single user case sucks for that user too, unless we started using autologin for single users, so they don't see the login screen, but that's a big change, and doesn't help precisely in the case of encrypted disk.
Also, the person who was going to dive into this switched teams and doesn't work on plymouth anymore.
I think the behavior we have now is suboptimal, but we've had it ~forever, and fixing it is hard enough, that I don't see how this could be a blocker, personally.
(In reply to Zbigniew Jędrzejewski-Szmek from comment #24)
> I think we should show they keymap in the password prompt.
you'd need a way to query it first...which the kernel doesn't provide.
(In reply to Zbigniew Jędrzejewski-Szmek from comment #25)
> Yet another useful feature would be an option to reveal the password. Like
> the little eye icon that the competition has, which you click and you see
> the actual letters. I don't think we can do this on the console (unless
> through some magic key combo which would be hard to discover), but it would
> be doable under plymouth.
hard to do under plymouth too. plymouth doesn't currently support mice. or text from the initramfs...
(In reply to Ray Strode [halfline] from comment #29)
> I think the behavior we have now is suboptimal, but we've had it ~forever,
From low-level POV, yes. From general user POV, it got more likely to hit since Fedora 19, when GNOME started changing system keymaps on single user systems (I just checked). But that's also long time ago (4 years), I admit.
For the record, here's a discussion of the Workstation WG regarding this bug (starting with "#topic BZ 1405539" line):
(In reply to Ray Strode [halfline] from comment #29)
> Rebuilding the initramfs from systemd or control-center doesn't really solve
> anything. User may still reboot weeks after changing it and not realize the
> change affects boot screen.
It's not sufficient, but it is necessary. If changing the system keyboard layout is going to affect the initramfs, then the change has to take effect immediately. No question about that. If we don't want to do this then we need to never change the keyboard layout in the initramfs at all.
With that fixed, we could then add some message in gnome-control-center to warn about the layout change and say the issue has been papered over well enough for the time being... I suppose....
Discussed in today's Blocker Review meeting. While this is a severe bug we'd love to have a fix for, the problem has existed for multiple releases and is exceptionally non-trivial to fix so it is Rejected as a Blocker for F26 Alpha.
(In reply to Mike Ruckman from comment #34)
> it is Rejected as a Blocker for F26 Alpha.
Correction, rejected as a Final blocker.
Note that we *are* tracking this as a high-priority bug, despite the non-blocker status. https://fedoraproject.org/wiki/Fedora_Program_Management/Prioritized_bugs_and_issues
I think an acceptable short-term workaround would be to ensure that the keyboard layout used on the unlock screen just never changes ever. That's not great, but it would at least avoid users getting locked out with no indication as to why.
The hint that anaconda is giving when entering a passphrase for encrypted volumes is not intuitive for end-users that are not familiar with the context.
"Warning: You won't be able to switch between keyboard layouts (from the default one) when you decrypt your disks after install"
The wiki page https://fedoraproject.org/wiki/Fedora_Program_Management/Prioritized_bugs_and_issues lists this bug as both Approved Prioritized and Already solved
@FioH: That was a mistake. The bug was listed twice on the page; once as resolved and once as unresolved. I am sorry for this confusion.
@mcatanzaro: Thanks for fixing the page.
Some, less experienced Fedora users may be confused by the Warning that anaconda is stating. (see example here: http://pasteboard.co/6Y8H4KYrW.png). One may think that this applies to the general OS or even the Desktop Environment. Maybe the warning could be more descriptive.
Removing this bug from the Prioritized list as it does not affect many users.
A similar problem is plaguing Debian Stretch. Gnome changes the console keyboard map to match the Gnome's one, but does it more or less unpredictably. Then, the changes are passed to initramdisk, but only when the kernel is updated.
Sounds like a some kind of design flaw. Non-root users aren't supposed to be able to change system-wide settings, right?
And to add to my comment, when Gnome has changed console keyboard layouts, "VC Keymap: n/a" shows on the output of "localectl" command. I am using the Finnish (DAS) keyboard layout.
This does not seem to be a plymouth bug, but something else.
Older duplicate of this bug report: https://bugzilla.redhat.com/show_bug.cgi?id=1151651
> This does not seem to be a plymouth bug, but something else.
This is NOT a bug in plymouth, as booting without plymouth is also affected. I'd prefer to blame localectl or whichever tool is changing the system keyboard layout. A similar issue has been reported a few times before:
Possible solutions from here:
* automatically trigger rebuilding the initramfs every time the system-wide keyboard layout changed. Sounds like a bad idea because it might lock out people from their own systems on reboot, so we need to educate the user about the fact that this also changes the initramfs layout.
* (from localectl binary): When changing the system layout, tell the user to rebuild their initramfs manually to apply these changes.
(In reply to Iiro Laiho from comment #43)
> Sounds like a some kind of design flaw. Non-root users aren't supposed to be
> able to change system-wide settings, right?
Yes, with PolicyKit (or sudo) they are, that's what is going on behind gnome-control-center asking for your password.(In reply to Iiro Laiho from comment #44)
> And to add to my comment, when Gnome has changed console keyboard layouts,
> "VC Keymap: n/a" shows on the output of "localectl" command. I am using the
> Finnish (DAS) keyboard layout.
Also, please have a look at `man localectl`:
> Note that the changes performed using this tool might require the initramfs to be rebuilt to take effect during early system boot. The initramfs is not rebuilt automatically by localectl.
The reason this is against Plymouth, IIRC, is that one of the proposed solutions / ameliorations is to have Plymouth display the active keyboard layout.