Bug 889562 - Console keymap set to "us" if you install with a keymap not provided by systemd-localed
Console keymap set to "us" if you install with a keymap not provided by syste...
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: systemd (Show other bugs)
18
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: systemd-maint
Fedora Extras Quality Assurance
AcceptedBlocker
: i18n
Depends On:
Blocks: F18Blocker/F18FinalBlocker
  Show dependency treegraph
 
Reported: 2012-12-21 13:04 EST by Adam Williamson
Modified: 2013-05-24 12:08 EDT (History)
17 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-01-08 16:30:18 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Adam Williamson 2012-12-21 13:04:15 EST
This is split off from https://bugzilla.redhat.com/show_bug.cgi?id=859867#c14 . As described there:

Testing just the way I did for "French (French)", I did the install in English, added the keyboard layout in Keyboard spoke, moved it to the top with ^, then completed the install. The completed spoke shows "French (French (Canada))", the install completes, but after reboot, I have US keyboard layout at the console, and vconsole.conf says KEYMAP="us".

It works if you use French (French).

According to vpodzime:

"The problem here is that for the 'ca' X layout ("French (French (Canada))") systemd-localed doesn't set matching VConsole keymap when asked to. Thus Anaconda can't get the matching VConsole keymap from systemd-localed and defaults to 'us'. "French (French)" ('fr') works as expected because systemd-localed converts it correctly to the 'fr' VConsole keymap.

There is nothing Anaconda can do about this. It is a separate systemd(-localed) bug."

This seems like it might be a big problem? If I read it right, for an install with any keymap not 'provided' by systemd-localed, console keymap will get set to "us".
Comment 1 Adam Williamson 2012-12-21 13:04:48 EST
Proposing as a blocker to make sure we don't miss something serious here, will continue to investigate.
Comment 2 Adam Williamson 2012-12-21 13:10:29 EST
Oh, the other part of the puzzle here is a passing remark by vpodzime in https://bugzilla.redhat.com/show_bug.cgi?id=875567:

"It is not a blocker because it affects only some installations -- with a LUKS password set using an X layout different from 'us' which has matching VConsole keymap correctly provided by systemd-localed (there are not much of them"

implying that systemd-localed doesn't provide a vconsole keymap correctly very often, hence this would affect a lot of keymap choices. I can't figure out how to tell when systemd-localed 'works' and when it doesn't from the code, can anyone help with that?
Comment 3 Vratislav Podzimek 2012-12-21 13:59:12 EST
(In reply to comment #2)
> Oh, the other part of the puzzle here is a passing remark by vpodzime in
> https://bugzilla.redhat.com/show_bug.cgi?id=875567:
> 
> "It is not a blocker because it affects only some installations -- with a
> LUKS password set using an X layout different from 'us' which has matching
> VConsole keymap correctly provided by systemd-localed (there are not much of
> them"
> 
> implying that systemd-localed doesn't provide a vconsole keymap correctly
> very often, hence this would affect a lot of keymap choices. I can't figure
> out how to tell when systemd-localed 'works' and when it doesn't from the
> code, can anyone help with that?
It's not that bad. There are two "not-working-as-expected cases":
1) systemd-localed provides a matching keymap, but this keymap is not the best matching one -- e.g. 'cz-lat2' VC keymap for the 'cz' X layout where 'cz-lat2' starts to work as 'cz' only after you hit the PauseBreak key (which almost nobody knows about).

2) systemd-localed doesn't provide any matching keymap. I don't know how often this happens.
Comment 4 Adam Williamson 2012-12-21 14:46:09 EST
So we need to quantify 2), really.

A big difference between F17 and F18 is that F17 didn't offer this giant list of keymaps that F18 does. You only got to pick the language from the same fairly small list F18 uses on its first screen, and we then tried to do some magic to pick a keyboard layout from that. That process has other problems, but it at least keeps the list of possible keyboard layouts anaconda has to deal with quite small.

systemd folks, do you have a ballpark guess of how many of the keyboard layouts listed on F18 installer's 'Keyboard' spoke might be subject to this bug?
Comment 5 Adam Williamson 2012-12-21 15:02:58 EST
sigh, I'm wrong again, F17 does offer a big keymap list.
Comment 6 Adam Williamson 2012-12-21 16:21:50 EST
Discussed at 2012-12-21 blocker review meeting: http://meetbot.fedoraproject.org/fedora-bugzappers/2012-12-21/f18final-blocker-review-7.2012-12-21-18.33.log.txt .

This is provisionally accepted as a blocker, but we could really do with input from systemd folks and i18n folks.

according to 'localectl list-keymaps', systemd appears to understand 209 keymaps:

localectl list-keymaps | wc -l

we're still working on replicating the list anaconda uses exactly - we think it reads from xkb via xklavier or something - but doing an exceedingly dumb test, if you scroll the list in the Keyboard spoke down through 200 entries, the scrollbar is still noticeably above halfway, so anaconda has *at least* 400 keymaps.

Even in the best possible case - every keymap systemd knows is also one that anaconda knows - that means anaconda is listing out at least 211 keymaps which systemd doesn't understand, and which will consequently hit this bug: if you pick any of those 211 keymaps in anaconda, you'll get en-US on the console after installation.

I guess what we need to know now is, is systemd's list too small? Is anaconda's too big? Is systemd missing some significant layouts? Notably all of Arabic seems to be missing, for instance, and systemd only seems to know one Japanese layout (anaconda knows 6).

i18n folks and systemd folks, can you give any input as to how big of a problem this is, and what we can do about it? Thanks.
Comment 7 Adam Williamson 2012-12-21 16:23:23 EST
er, 191, not 209. I fail arithmetic, apparently.
Comment 8 Adam Williamson 2012-12-21 19:11:13 EST
Chinese seems to be affected by this bug. That seems like a big one.
Comment 9 Adam Williamson 2012-12-21 19:44:56 EST
on the comparison to F17 - I have no idea where F17 got its keyboard mapping list, but it seems to be some kind of hand-picked thing. It has 70 keyboard layouts, less than either systemd or anaconda in F18, but it *does* include French Canadian. It only has one Japanese layout and no Chinese.
Comment 10 Adam Williamson 2012-12-21 20:35:10 EST
So systemd's map is /usr/share/systemd/kbd-model-map . Poking at it, I see a few things:

I think French (Canada) doesn't work because systemd thinks the X keyboard layout name for this is "ca(fr)". Maybe it used to be. Now, according to /usr/share/X11/xkb/rules/base.xml and 'man xkeyboard-config', I believe it's simply "ca".

I think the arabic layouts may be screwed by "ara" vs. "ara,us", but I may be missing some subtlety in kbd-model-map there.
Comment 11 Lennart Poettering 2012-12-22 04:20:37 EST
Hmm, can you summarize for me what the X11 layout/model/variant/options are that are translated incorrectly, and what they are translated to, and what you'd expect instead?

The translation logic is actually kinda simple: we pick the closest match, and if two entries match equally well, we take the earlier one. Also, of course, layout matters more than model, which matters more than variant, which matters more than options.
Comment 12 Adam Williamson 2012-12-22 10:27:17 EST
If it's 'fuzzy' like that it may be less broken than I thought, I'll have to do more testing. Unfortunately this is a PITA to test.
Comment 13 Leslie Satenstein 2012-12-23 18:59:34 EST
I believe it is a blocker.
In the past, with my System, purchased in France, it was delivered with the AZERTY keyboard.  AZERTY is not QWERTY.  I have no other keyboard.

I begin the installation, I chose English as my installation language, and in keyboard selection, I remove all keyboard definitions but add back azerty.
Before clicking to continue, Anaconda confirms it is azerty.
Virtual terminal interface and keyboard support after reboot is US English with the octothorpe #  on the 3 key. I have no «accented characters»éÉ etc in virtual terminal mode.but gnome,and kde terminal emulation works with the proper layout.
The initial problem after a reboot is that the password field is going to be analyzed, not as input from azerty, but from US PC105 layout. It became a problem because I used the # and the é as two characters of my password.
Comment 14 Adam Williamson 2012-12-23 19:34:45 EST
Leslie: the regular French layout works fine for me. It's only French (Canada) that doesn't work. Looking at the GNOME keyboard layout thingy, French (Canada) doesn't appear to be an AZERTY layout, but QWERTY with different modifiers and stuff. French (France) is AZERTY, as are several of the other French layouts, but not French (Canada).
Comment 15 Akira TAGOH 2012-12-26 22:26:41 EST
(In reply to comment #8)
> Chinese seems to be affected by this bug. That seems like a big one.

No Chinese keymaps on console on even f17. so it shouldn't be a problem.
Comment 16 Leslie Satenstein 2012-12-29 15:58:12 EST
Will I be able to test this before go live, as TC4?  

 

Please see another more serious bug.... 
Bug  890343 It may be related..

I can pick up the next TCx image from the usual Folder where I found TC3
Comment 17 Adam Williamson 2013-01-02 21:42:52 EST
So, I'm getting some more of a handle on this. 

1. Lots of the entries in the Xkb list are XkbVariant of some XkbLayout. e.g. there are 8 Arabic layouts, all but one of which are XkbVariants of the same XkbLayout - the XkbLayout is "ara" and there are a ton of XkbVariants .

kbd-model-map has a few Arabic entries:

ar-digits		ara,us	pc105		digits
ar-qwerty		ara,us	pc105		qwerty	
ar-azerty-digits	ara,us	pc105		azerty_digits
ar-azerty		ara,us	pc105		azerty	
ar-qwerty-digits	ara,us	pc105		qwerty_digits

the variant mapping doesn't seem to work, though. All the Arabic layouts wind up with 'ar-digits' - the first entry in the list - in vconsole.conf .

So it looks like if the XkbLayout is something that's roughly in kbd-model-map - the mapping seems to be a bit fuzzy, so 'ara' vs 'ara,us' is ok, apparently - you will get the first 'matching' entry in kbd-model-map in vconsole.conf . Variant matching doesn't seem to be working. This also affects some Indian layouts: anything which is a variant of 'in' is going to get 'guj' (presumably Gujarati) for a console layout. I just did an install with Bengali (India, Probhat) as the layout - which in Xkb terms is XkbLayout 'in', XkbVariant 'ben_probhat' - and it gets 'guj' in /etc/vconsole.conf .

Note that both ar-digits and guj seem to *behave* in a very qwerty way when used as console layouts. They may need some kind of dead key to switch to Arabic / Indian characters. I just don't have the knowledge there. As X layouts, they're very very not-qwerty (you can check this in GNOME control center).

2. Rather more straightforward - any valid XkbLayout which does not have a mapping (even fuzzy) in kbd-model-map is going to wind up as "us" in vconsole.conf. There seem to be a *lot* of these, just a whole lot. So far I found that any variant of the following is going to fail:

Armenian "am"
Azerbaijani "az"
Bambara "ml"
Belarusian "by"
Bengali "bd"
Berber "ma"
Bosnian "ba"
Burmese "mm"
Catalan "ad"
(Central) Khmer "kh"
Chinese "cn"

and at that point I'm kind of losing the will to live. I also found one or two oddball cases - like Cherokee, which in Xkb terms is a 'variant' of English (it's XkbLayout "us" XkbVariant "chr") but if you look at it, it just looks nothing like English. So vconsole.conf of "us" is technically a 'correct mapping' there, but probably not what someone with a Cherokee keyboard wants. I guess. Who the hell knows? I ain't Cherokee.

Bottom line, I think we can conclude that a _lot_ of the mappings offered in anaconda are going to fail (get "us" as the console layout), as I thought originally, just because Xkb has grown a whole chunk of XkbLayouts which kbd-model-map doesn't cover. I think Lennart underestimated how much change has gone on in xkb and hence how many layouts there are available which systemd's list just cannot handle. Then there are the variant mapping and 'oddball' problems which just pile up on top. An executive summary of 'stuff be broken' is pretty valid here. I can generate a close-to-complete list of broken families if it's *really* desired, but it's incredibly boring work and I'm not sure we need it.

The method I've been using is to look through anaconda's layout list in a VM and have GNOME's keyboard layout thingy open in the host: find the matching layout (the two lists are similar but not identical, and GNOME's is even bigger, for some reason) in GNOME, 'add' it, and run:

gsettings get org.gnome.desktop.input-sources sources

this will tell you the actual XkbLayout and XkbVariant for the selected layout, it's much faster than actually completing an install and seeing what gets written to 00-anaconda-keyboard.conf . You can then search kbd-model-map and see if it has any kind of match. I did a bunch of test installs to develop and confirm the above theories - I tested every Arabic layout, and various others (like the Bengali/Gujarati case).

So I think we need to decide simply whether 'anaconda is offering this huge list of layouts, and many of them will fail to be translated properly to console input' is a blocker bug.

The other data that might be useful here are:

1. How many of the 'default layouts' for languages listed on the very first page of install fail? This is something I can poke. 

When anaconda starts, it asks you first for a *language*, not a keyboard layout, and there's a 'Set keyboard to default layout for selected language' checkbox which more or less does what it says on the tin: if you check that, and pick a language, anaconda will try to pick an appropriate keyboard layout for said language. We could draw a line here and say 'if all or most of those options work, we're okay', for instance. The list of languages is much smaller than the list of layouts. To get the full range of layouts, you have to go into the actual layout spoke.

2. How many of the layouts that were offered in F17 fail? Again, I can try and provide this info.

In F17 anaconda offered what looks to have been a kind of 'hand-weeded' list of layouts - far fewer than it offers in F18, and far fewer than even system-config-keyboard (used to) offer, by the looks of things. I could just check out each of those, and see how many (if any) of them are broken in F18. Again this is somewhere we could at least theoretically draw a line, and say that even though F18 offers way more layouts, we're okay as long as the smaller range that was in F17 works.

I'll try and provide the above data in further comments, if I can dredge up the will to poke this bug any further today.
Comment 18 Adam Williamson 2013-01-02 21:44:29 EST
It might _really_ help to have some kind of input from i18n/l10n communities on whether their users really expect console input to be native or US, how badly it's going to throw them if the console input is US, I guess.
Comment 19 Adam Williamson 2013-01-03 00:05:21 EST
So, I went through data point 1) above. I went through every language offered on the first screen of Anaconda, and saw what happened when I asked for the 'default keyboard' for that language.

Here are the languages which display this bug, with their translation completion percentage:

Amharic - "et" / "us" (6%)
Belarusian - "by" / "us" (4%)
Bengali (Bangladesh) - "bd" / "us" (33%)
Bengali (India) - "in+ben" / "guj" (33%)
Bosnian - "ba" / "us" (10%)
Catalan (Spain) - this results in U.S. in anaconda even though Catalan layouts are available. Manually selected Catalan "ad" would not work (25%)
Persian (Iran) - "ir" / "us" (0%)
Hebrew (Israel) - "il" / "us" (22%)
Hindi (India) - "in+bolnagri" / "guj" (92%)
Armenian (Armenia) - "am" / "us" (2%)
Georgian (Georgia) - "ge" / "us" (24%)
Kazakh (Kazakhstan) - "kz" / "us" (100%)
Kannada (India) - "in+kan" / "guj" (33%)
Lithuanian (Lithuania) - "lt" / "us" (100%)
Latvian (Latvia) - "lv" / "us" (20%)
Macedonian (Macedonia) - "mk" / "us" (10%)
Malayalam (India) - "in+mal" / "guj" (20%)
Malay (Malaysia) - see Catalan (20%)
Nepali (Nepal) - "np" / "us" (10%)
Oriya (India) - "in+ori" / "guj" (27%)
Punjabi (India) - see Assamese (100%)
Sinhala (Sri Lanka) - see Assamese (27%). Manually selected layouts will not work
Albanian (Albania) - "al" / "us" (16%)
Serbian (Serbia) - "rs" / "sr-latin" (should be sr-cy, X default is Cyrillic not Latin) (23%)
Serbian (Latin, Serbia) - gives Serbian (Montenegrin) "me", which seems wrong, I'd expect Serbian (Latin) "rs+latin". me is not going to work (23%)
Tamil (India) - "in+tam_unicode" / "guj" (100%)
Telugu (India) - "in+tel" / "guj" (32%)
Thai (Thailand) - "th" / "us" (18%)
Urdu (Pakistan) - "pk" / "us" (7%)
Vietnamese (Vietnam) - "vn" / "us" (6%)
Chinese (China) - "cn" / "us" (99%)
Chinese (Taiwan) - "cn" / "us" (100%)

Quite a lot of those are very incomplete translations, but several are complete and fairly important languages.

I also found several cases where Anaconda's 'default keyboard' mapping seems to be broken - where you get English (U.S.) in anaconda even though you'd expect a 'native' layout - and some where there are native layouts present in GNOME's keyboard configuration applet, but they are not available through anaconda. I will file those separately, as they are separate bugs, but they contribute to an overall impression of brokenness here, unfortunately.
Comment 20 Adam Williamson 2013-01-03 01:36:15 EST
Filed https://bugzilla.redhat.com/show_bug.cgi?id=891489 and https://bugzilla.redhat.com/show_bug.cgi?id=891487 for the above noted separate bugs. The overall impression given by this bug, those bugs, the clunkiness of the keyboard selection functionality, the fact that rather a lot of the translations we offer are very incomplete, and a few other misc. bugs is that the support for non-U.S. English in Fedora 18 is extremely flaky, and this might be something we need to consider as a whole.
Comment 21 Vratislav Podzimek 2013-01-03 06:11:17 EST
To get the list of all X layouts Anaconda knows/understands, the [1] script referenced from [2] can be used.

[1] http://vpodzime.fedorapeople.org/layouts_list.py
[2] http://fedoraproject.org/wiki/Anaconda/Kickstart#keyboard
Comment 22 Adam Williamson 2013-01-03 17:11:34 EST
well, THAT would've saved me a lot of work. :)

filed upstream bug https://bugs.freedesktop.org/show_bug.cgi?id=59000 with several patches to kbd-model-map that fix the following layouts:

French (Canadian)
Belarusian
Hebrew (Israel)
Kazakh (Kazakhstan)
Lithuanian (Lithuania)
Macedonian (Macedonia)

those are all the ones from the list that can be reasonably straightforwardly fixed. Most of the others just plain don't have matching console layouts. There are probably console layouts it'd be 'sensible' to pick for several of them, but that's getting into tricky territory I don't want to stumble into right now. The other thing that it would help to fix is the variant mapping: right now, when mapping from an X config to a console config, systemd-localed seems to be just ignoring the xvariant and going with the first line that matches the xlayout. So even if your X config file lists xlayout 'rs' with no xvariant (which is the Cyrillic X layout), it gets matched against this line in kbd-model-map:

sr-latin		rs	pc105		latin		terminate:ctrl_alt_bksp

not this one lower down:

sr-cy			rs	pc105		-		terminate:ctrl_alt_bksp

and so you get a Latin console layout, not a Cyrillic one. I'll file a separate bug for that, for clarity.
Comment 23 Adam Williamson 2013-01-03 17:13:14 EST
Kazakh and Lithuanian are 100% translations, so it'd be especially nice to fix those.
Comment 24 Fedora Update System 2013-01-03 19:25:37 EST
systemd-195-15.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/systemd-195-15.fc18
Comment 25 Adam Williamson 2013-01-03 23:47:05 EST
I checked and the fix seems to work. Be good for someone else to sanity check, though, since I wrote it. The six layouts listed in c#22 ought to be fixed. If you do an install which includes systemd-195-15 and pick one of those layouts, you should get a correct console layout.
Comment 26 Fedora Update System 2013-01-04 15:27:39 EST
Package systemd-195-15.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing systemd-195-15.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-0190/systemd-195-15.fc18
then log in and leave karma (feedback).
Comment 27 Adam Williamson 2013-01-04 17:15:49 EST
Since it hasn't been recorded here: we decided there's only a limited amount of 'fixing' that can be done to this bug. I may re-open it but no longer as a blocker after the update goes through, as there are probably a few other mappings that could be improved, which I might do post-release if I have nothing else to do. But it's not a huge priority. In most cases where this is broken there simply isn't a 'corresponding' kbd layout at all, and I don't really want to go around making guesses about which 'not-really-the-same-but-similar' kbd layout the users of any given xkb layout 'would want', that doesn't sound like much fun.

So the long-term 'fix' for this is going to be https://bugzilla.redhat.com/show_bug.cgi?id=680990 and https://bugzilla.redhat.com/show_bug.cgi?id=837292 - the work to simply use xkb layouts on the console. That's the sensible way to fix this, avoid the whole ridiculous 'two sets of keyboard layouts' thing entirely.

So far as blocker status goes, the 'fix' above - fixing the layouts that are associated with offered locales and that can actually be fixed - is considered sufficient to resolve blocker status, per the go/no-go meeting yesterday.
Comment 28 Leslie Satenstein 2013-01-06 19:14:09 EST
(In reply to comment #0)
> This is split off from
> https://bugzilla.redhat.com/show_bug.cgi?id=859867#c14 . As described there:
> 
> Testing just the way I did for "French (French)", I did the install in
> English, added the keyboard layout in Keyboard spoke, moved it to the top
> with ^, then completed the install. The completed spoke shows "French
> (French (Canada))", the install completes, but after reboot, I have US
> keyboard layout at the console, and vconsole.conf says KEYMAP="us".
> 
> It works if you use French (French).
> 
> According to vpodzime:
> 
> "The problem here is that for the 'ca' X layout ("French (French (Canada))")
> systemd-localed doesn't set matching VConsole keymap when asked to. Thus
> Anaconda can't get the matching VConsole keymap from systemd-localed and
> defaults to 'us'. "French (French)" ('fr') works as expected because
> systemd-localed converts it correctly to the 'fr' VConsole keymap.
> 
> There is nothing Anaconda can do about this. It is a separate
> systemd(-localed) bug."
> 
> This seems like it might be a big problem? If I read it right, for an
> install with any keymap not 'provided' by systemd-localed, console keymap
> will get set to "us".


I just tested with CA as the keyboard layout and there is no vconsole.conf in /etc

Using man vconsole.conf I created the appropriate one with
KEYMAP="ca" 
as a single one line entry.  This had no effect after a reboot.
Comment 29 Leslie Satenstein 2013-01-06 19:20:48 EST
I also retried with KEYMAP="cf" after listing 

localectl list-keymaps

Is there actually a ca or cf keymap?
Comment 30 Adam Williamson 2013-01-06 19:33:52 EST
Yes. It is very similar to U.S., though, you may just not notice the difference. It is not an AZERTY layout. If you want an AZERTY layout, pick French (French).

"I just tested with CA as the keyboard layout and there is no vconsole.conf in /etc"

You didn't say what you tested *with*. This fix is only in RC1, I think.
Comment 31 Fedora Update System 2013-01-08 16:30:21 EST
systemd-195-15.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 32 Leslie Satenstein 2013-01-09 10:33:33 EST
Is the patch in RC2 Live Desktop or RC2 live KDE?

Testing takes time..., and as I retired gentleman, who is glad to test new patches, but his wife gets upset if her linux system is broken.

Or do I wait to end of day today for RC3?
Comment 33 Jens Petersen 2013-01-09 23:04:32 EST
(In reply to comment #32)
> Is the patch in RC2 Live Desktop or RC2 live KDE?

I think RC2 was released before systemd-195-15.fc18 went stable.

> Or do I wait to end of day today for RC3?

I see it in RC3 Live yes.
Comment 34 Adam Williamson 2013-05-24 12:08:51 EDT
Note: we will be moving to kbd layouts generated from the xkb layouts for F20, as announced at https://lists.fedoraproject.org/pipermail/devel/2013-May/183099.html . We need to make sure systemd does the right thing with the new kbd layouts.

Note You need to log in before you can comment on or make changes to this bug.