Bug 1031848 - xkb-converted layouts do not include multiple mappings where they are commonly expected
xkb-converted layouts do not include multiple mappings where they are commonl...
Product: Fedora
Classification: Fedora
Component: kbd (Show other bugs)
All All
unspecified Severity urgent
: ---
: ---
Assigned To: Vitezslav Crhonek
Fedora Extras Quality Assurance
: i18n
Depends On:
Blocks: F20FinalBlocker
  Show dependency treegraph
Reported: 2013-11-18 19:38 EST by Adam Williamson
Modified: 2013-11-25 22:57 EST (History)
7 users (show)

See Also:
Fixed In Version: kbd-1.15.5-11.fc20
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2013-11-25 22:57:58 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Adam Williamson 2013-11-18 19:38:00 EST
So this seems like an important enough issue to track it explicitly. The issue is as I explained in https://bugzilla.redhat.com/show_bug.cgi?id=837292#c46 .

In F20 we're kind of trying to replace the 'legacy' layouts in the kbd package with layouts converted from xkb. However, there's a major issue caused by a difference in how kbd and xkb handle layout switching.

When typing in some languages, it's customary to have both a layout for entering the characters for the language in question and a layout for entering Roman characters, and to switch between them frequently. Russian is the obvious example. A Russian keyboard looks like this:


Russian users expect to always have both a 'Russian' and a 'US' layout defined, and to switch frequently, so they can enter both Cyrillic and Roman characters. With the 'Russian' layout selected, pressing the 'a/Ф' key will result in a Ф character; with the 'US' layout selected, it'll result in an 'a' character.

xkb's implementation of layout switching is really just that - you define each possible mapping of the keyboard as a single layout, and then xkb allows you to configure as many layouts as you want to be 'active', and a switcher key to switch between them (desktops may also offer you a menu or whatever to do this). So a Russian user would expect to configure both the 'Russian' and 'English (US)' xkb layouts and switch between them frequently.

kbd doesn't work this way, though. There isn't a framework like xkb for console text entry which would handle this kind of switching between multiple kbd layouts: really there's just a simple mechanism for requesting that a given layout be loaded, which is what 'loadkeys' does, and we have a systemd service that runs 'loadkeys' and loads the single layout defined in /etc/vconsole.conf at startup. You can't have multiple 'active' kbd layouts, you can only load one at a time.

Instead, in the kbd world, a single layout *itself* can define multiple possible sets of mappings and a switcher key. If you look at the legacy 'ru.map.gz', for instance, it defines both Cyrillic and Roman character mappings for the letter keys, and the key combination ctrl+shift to switch between them. If you do 'loadkeys /usr/lib/kbd/keymaps/legacy/i386/qwerty/ru.map.gz' you can test this: hit 'a', then hit ctrl+shift and hit 'a' again. Your Ф may show up as a Ф or a little unicode box depending on other factors, but you can reliably switch between the two character sets.

The actual problem we have here is that the xkb-converted kbd layouts for keyboards whose users would generally expect to this kind of switching do not include the multiple mappings and switcher key definitions. If you just do 'loadkeys ru' then what will actually get loaded is /lib/kbd/keymaps/xkb/ru.map.gz , the xkb-converted Russian layout. Try that, and you'll note that switching just doesn't work: all you have is the mapping for entering Cyrillic characters. The upshot is that you're completely unable to enter Roman characters at the console when using our current 'ru' kbd layout.

Unfortunately I can't think of a very awesome solution for this. I suppose we could implement an xkb-alike for the console, but that seems like a lot of work. A not-very-awesome solution would be to preserve the 'legacy' kbd layouts for the cases where we know 'native/Roman switching' of this kind is desired and expected. Going from kbd-model-map, those cases would appear to be:

mk-utf			mk,us
guj			in,us
gur			gur,us
tml-inscript		in,us
ben-probhat		in,us
tml-uni			in,us
ua-utf			ua,us
bg_pho-utf8		bg,us
cz-us-qwertz		cz,us
ar-digits		ara,us
ar-qwerty		ara,us
ar-azerty-digits	ara,us
ben			in,us
bg_bds-utf8		bg,us
ru			ru,us
ar-azerty		ara,us
ar-qwerty-digits	ara,us
dev			dev,us
gr			gr,us
by			by,us
kazakh			kz,us

We'd have to come up with some kind of refinement to the localed stuff which caught any of those cases coming out of anaconda and did the correct thing, I guess.

A more comprehensive fix, however it looked, would I think require langtable to know about these cases - cases where it is standard, expected practice that both 'native' and US layouts will be defined. Right now I don't see that langtable really expresses this. If we had that data in langtable, we might be able to do something ugly like have the script that generates kbd layouts from xkb layouts generate a combined native+us kbd layout for those cases?

Going to tentatively propose this as a release blocker and seek input from i18n folks on whether my understanding here is correct and how big of an issue this is for them.
Comment 1 Adam Williamson 2013-11-18 20:47:11 EST
I did an actual install test just to verify my reading of this issue is correct. It seems to be. If you install in Russian, then anaconda does something sane: it configures both Russian and English (US) keyboard layouts. In anaconda and then in X in the installed system, everything works great - both layouts are available everywhere, and you can switch between them. But at the console in the installed system, you have the xkb-converted 'ru' layout loaded, and you cannot input anything but Cyrillic characters.
Comment 2 Mike FABIAN 2013-11-18 23:40:24 EST
(In reply to Adam Williamson from comment #0)

> A more comprehensive fix, however it looked, would I think require langtable
> to know about these cases - cases where it is standard, expected practice
> that both 'native' and US layouts will be defined. Right now I don't see
> that langtable really expresses this. If we had that data in langtable, we
> might be able to do something ugly like have the script that generates kbd
> layouts from xkb layouts generate a combined native+us kbd layout for those
> cases?

langtable has this information. It can be queried
with the supports_ascii() function:

    def supports_ascii(keyboardId=None):
        Returns True if the keyboard layout with that id can be used to
        type ASCII, returns false if the keyboard layout can not be used
        to type ASCII or if typing ASCII with that keyboard layout is

        >>> supports_ascii("jp")
        >>> supports_ascii("ru")
        if keyboardId in _keyboards_db:
            return _keyboards_db[keyboardId].ascii
        return True

Anaconda uses this to offer US English layout as a second layout by
default during the installation if the layout for the chosen language
does not support ASCII. I.e. when the language chosen is Russian,
anaconda will offer both Russian and US English layout by default, if
the language chosen is Japanese, it will only offer Japanese layout by
default (because the Japanese layout can be used to input ASCII
without problems).
Comment 3 Adam Williamson 2013-11-19 00:10:56 EST
ahh, I saw that key but didn't catch what it was for. Cool. So we do have the info to try and do something with it at the kbd level, whatever that is.
Comment 4 Vitezslav Crhonek 2013-11-19 09:48:18 EST
I wouldn't put that logic into kbd...

I think that easiest way is this one - when during install when users selects one of switchable layout, appropriate keymap variant should be written to /etc/vconsole.conf for console by anaconda/systemd (or whatever is doing it now).

For example, for

cz-us-qwertz		cz,us

it is cz-lat2.

In case that kbd would be without current "old keymaps names to xkb converted keymaps" symlinks, correct keymap cz-lat2 from legacy directory will be selected and switching will work.

The only disadvantage I see is that cz-lat2 is probably not exactly same as cz-us-qwertz. But I don't think that this is problem, because virtual console users are used to having just these "old" keymaps.
Comment 5 Adam Williamson 2013-11-19 12:44:46 EST
Well, couple of problems. One, I don't think your example is right. The table I gave is the first two columns of kbd-model-map for affected layouts. The first entry is the kbd layout name, the second is the matching set of xkb layouts: so what "cz-us-qwertz     cz,us" means is "if the user picks cz-us-qwertz kbd layout, select the cz,us X11 layouts".

But more significantly, we have another problem with the approach: name collisions. In some cases the kbd and xkb names differ, but in at least three that I can see, they don't: ru, gr and by. If you pick Russian, Greek or...whatever the hell by is..., then how do we manage to get the legacy keymap loaded rather than the xkb-converted one? We can't do it by name: whether we're using symlinks or the path approach you just posted to https://bugzilla.redhat.com/show_bug.cgi?id=1028207 , the xkb-converted layout would 'win' as things stand, AFAICS.

So I guess the way to address that in the symlink approach would be to drop the symlinks for those three layouts and instead have the 'real' legacy layouts present in the /usr/lib/kbd/keymaps dir. The way to address it in the path approach would be to...simply not include the xkb-converted versions of those three layouts at all, I suppose? Then the legacy ones would be loaded...
Comment 6 Adam Williamson 2013-11-19 14:15:54 EST
One thing Kay points out is that this should become moot whenever we get around to switching to kmscon: http://www.freedesktop.org/wiki/Software/kmscon/ . It sounds like, to a rough approximation, in a kmscon world your console keyboard configuration is exactly the same as your xkb keyboard config - it uses libxkbcommon to pretty much implement Xkb entirely. As long as that includes layout switching, then once we have kmscon we can ditch all the kbd crap entirely and simply write an xkb configuration from anaconda and everything will work.

If we could, say, switch to kmscon for F21, then we'd really just need to try and implement whatever quick fixes and hacks we can safely do to make F20 work as decently as possible, without worrying about 'longer-term' fixes.
Comment 7 Adam Williamson 2013-11-19 15:08:15 EST
OK, so I think I have a good idea for the best way to hack this for F20.

Where kbd.spec actually does the xkb conversion, we should add a check against langtable and don't convert layouts which have ASCII support set as 'false'.

I think the result of that would be that, where kbd-model-map has a mapping to a 'good' legacy layout, that legacy layout would be used, and where it doesn't, a US layout would be used. Which I think is a sensible place to be.

It should be relatively trivial to add this logic for someone who knows what they're doing, I think - it should just be a single query against langtable-python. I don't know what I'm doing, so it'd take me longer :) But I can try.

This seems saner than hacking in a static list of 'rm -f's after the conversion occurs.
Comment 8 Adam Williamson 2013-11-19 17:38:41 EST
If we can't get it done 'nicely' via langtable we could also do this hack I just came up with:

zgrep -L "U+0041" * | xargs rm -f

run against the set of generated keymaps would delete any which do not contain a mapping for a capital "A", which seems a reasonable proxy for 'layouts that can't input ASCII'. Ugly, but probably effective.
Comment 9 Adam Williamson 2013-11-19 18:20:36 EST
http://koji.fedoraproject.org/koji/taskinfo?taskID=6203682 is a kbd built with the change from https://bugzilla.redhat.com/show_bug.cgi?id=1028207#c61 and *also* with the hack I just described added - all the converted layouts which fail that grep test are deleted right after the conversion runs. I'm planning to build a test live image with this kbd and the new systemd build that should be showing up soon, and see how that behaves.
Comment 10 Adam Williamson 2013-11-20 01:01:13 EST
OK, I built a live image with my kbd scratch build and https://admin.fedoraproject.org/updates/systemd-208-6.fc20 and did installs in Russian, Arabic, and UK English. Behaviour in all cases was as I'd expect: the Russian install loaded the 'legacy' Russian kbd layout and switching worked, the Arabic install wrote 'ar-digits' into vconsole.conf (indicating that localed is still 'converting' layout names when appropriate) but actually wound up with a US console layout because ar-digits doesn't actually *exist* (indicating that fallback mechanisms work as expected), and the UK install loaded the xkb-converted 'gb' layout, not the legacy 'uk' one. So unless anyone can see a flaw in my Awesome Plan somewhere, I'd suggest we just go with it. I'm actually pretty happy with the 'grep hack'; it's simple, and it seems to achieve what we need to achieve.
Comment 12 Adam Williamson 2013-11-20 13:03:45 EST
Discussed at 2013-11-20 blocker review meeting: http://meetbot.fedoraproject.org/fedora-blocker-review/2013-11-20/f20-blocker-review.2013-11-20-17.00.log.txt . This was accepted as a release blocker per Alpha criterion "A system installed without a graphical package set must boot to a state where it is possible to log in through at least one of the default virtual consoles." in the case of installing with an affected keyboard mapping - you won't be able to log in if you entered a username/password using Roman characters (which is very likely, people usually don't use native characters for those). It also likely affects unlocking of encrypted partitions.
Comment 13 Vitezslav Crhonek 2013-11-21 05:06:20 EST
The solution looks fine to me.

Adam, could you please attach the patch explicitly? I'm not able to grab it from the scratch build.
Comment 14 Adam Williamson 2013-11-21 12:14:35 EST
vitezslav: simply add these lines right above find_lang:

# wipe converted layouts which cannot input ASCII (#1031848)
zgrep -L "U+0041" $RPM_BUILD_ROOT/lib/kbd/keymaps/xkb/* | xargs rm -f

that's all it takes.
Comment 15 Fedora Update System 2013-11-21 12:56:43 EST
kbd-1.15.5-11.fc20 has been submitted as an update for Fedora 20.
Comment 16 Fedora Update System 2013-11-23 23:06:28 EST
Package kbd-1.15.5-11.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing kbd-1.15.5-11.fc20'
as soon as you are able to.
Please go to the following url:
then log in and leave karma (feedback).
Comment 17 Fedora Update System 2013-11-25 22:57:58 EST
kbd-1.15.5-11.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.