Bug 448704 - non-UTF console support is broken
non-UTF console support is broken
Status: CLOSED WONTFIX
Product: Fedora
Classification: Fedora
Component: systemd (Show other bugs)
17
All Linux
low Severity medium
: ---
: ---
Assigned To: systemd-maint
Fedora Extras Quality Assurance
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-05-28 07:04 EDT by Vsevolod Volkov
Modified: 2017-06-14 19:17 EDT (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-08-01 14:34:25 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
console_init patch (1.30 KB, patch)
2008-05-28 07:04 EDT, Vsevolod Volkov
no flags Details | Diff
Set language after starting udev (787 bytes, patch)
2009-01-01 13:00 EST, Vsevolod Volkov
no flags Details | Diff
/etc/profile.d/lang.* fix (1.46 KB, patch)
2009-01-01 13:15 EST, Vsevolod Volkov
no flags Details | Diff

  None (edit)
Description Vsevolod Volkov 2008-05-28 07:04:33 EDT
Description of problem:
Non-UTF concole support is fully broken:
- console is not initialized properly;
- localized messages are not dispalyed properly at boot time;
- localized messages are not dispalyed properly at shutdown.

Version-Release number of selected component (if applicable):
initscripts-8.76.2-1 (probably, from 8.64-1).

How reproducible:
Always.

Steps to Reproduce:
1. Setup console for non-UTF encoding.
Example of /etc/sysconfig/i18n:
-------------------------------
LANG="ru_RU.KOI8-R"
SUPPORTED="ru_RU.KOI8-R:ru_RU:ru"
SYSFONT="ruscii_8x16"
SYSFONTACM="koi8-u_to_uni.trans"
UNIMAP="ruscii.uni"

Example of /etc/sysconfig/keyboard:
-----------------------------------
KEYBOARDTYPE="pc"
KEYTABLE="ru3"

2. Reboot.
3. Login and try to print or type cyrillic letters. Use Capslock to toggle
keyboard layout.
4. Logout or switch to unused console and shutdown by Ctrl-Alt-Del or power off
button.

Actual results:
Boot time: all localized messages are not readable (garbage). But in first
localized message (Starting udev) all cyrillic letters look like squares.

Shutdown time: all cyrillic letters look like squares.

Console: cyrillic messages are not displayed properly. Garbage is printed if use
cyrillic keyboard layout. Keyboard is working in UTF mode.

Expected results:
Cyrillic letters should be displayed properly from boot till shutdown.

Additional info:
The main problem, that /lib/udev/console_init does not initialize console
properly because
- it uses SYSFONTACM (instead of UNIMAP) as unicode font map (-u switch of
/bin/setfont);
- it does not initialize unicode console map (-m switch of /bin/setfont).

Included patch will fix it. Additionally, /bin/setfont is called with -C and
given console device. Fixed console_init causes proper displaying boot messages.
Only first localized message (Starting udev) still displayed as squares, because
it's printed before starting udev and console_init. To fix this, console
initialization should be moved from udev to the beginning of rc.sysinit, as
before removing /sbin/setsysfont from rc.sysinit.

Second problem, that after login keyboard is working in UTF mode and user
console map is not selected by Esc(K. Appropriate code was removed from
/etc/profile.d/lang.* recently. Script unicode_stop should be invoked if LANG is
not UTF-8 and Esc(K should be printed if SYSFONTACM is defined. It will be
enough to restore lang.* from Fedora8 initscripts.

Shutdown messages can be fixed by selecting console map by unicode_stop and echo
Esc(K as in old version of lang.sh. It should be done when shutdown is called by
control-alt-delete event or power off button.
Comment 1 Vsevolod Volkov 2008-05-28 07:04:33 EDT
Created attachment 306920 [details]
console_init patch
Comment 2 Bill Nottingham 2008-05-29 16:22:20 EDT
I'm not seeing why you need the -C option to setfont.
Comment 3 Vsevolod Volkov 2008-05-30 07:53:02 EDT
Really, -C option is not needed if execute 'console_init /dev/tty0' from udev on
boot. But setfont prints Esc(K to stdout while console_init works with console
given as args[1]. So, it will work incorrectly if run 'console_init /dev/tty2'
from tty1, for example. I'm not needed the -C option, but it allows to get
expected result at any case.
Comment 4 Bill Nottingham 2008-11-03 16:40:06 EST
Does this work with the most recent initscripts update for you?
Comment 5 Vsevolod Volkov 2008-11-04 15:01:37 EST
It does not work. But there are some changes.

1. All messages are readable on boot. But all cyrillic letters look like squares in first localized message (Starting udev). You should init console before starting /sbin/start_udev or change

STRING=$"Starting $prog: "

to

STRING="Starting $prog: "

in /sbin/start_udev.

2. Console: cyrillic messages are not displayed properly (letters look like squares). Cyrillic letters are displayed properly when typing keyboard. But keyboard still working in UTF mode.

The patch attached in previous message works fine for non-UTF console and does not break UTF console. Please, try it.
Comment 6 Bill Nottingham 2008-11-04 15:08:16 EST
> 1. All messages are readable on boot. But all cyrillic letters look like
> squares in first localized message (Starting udev). You should init console
> before starting /sbin/start_udev or change
> 
> STRING=$"Starting $prog: "
> 
> to
> 
> STRING="Starting $prog: "
> 
> in /sbin/start_udev.

Can you file the latter idea as a udev bug? We can't init the console before printing the message, as it's actually done as a udev rule.

> 2. Console: cyrillic messages are not displayed properly (letters look like
> squares). Cyrillic letters are displayed properly when typing keyboard. But
> keyboard still working in UTF mode.
> 
> The patch attached in previous message works fine for non-UTF console and does
> not break UTF console. Please, try it.

... the prior patch is very much obsolete with current initscripts - most all of those changes are already in in one form or another.
Comment 7 Vsevolod Volkov 2008-11-04 16:55:32 EST
1. Another way to fix "Starting udev: " message is replacing

/sbin/start_udev

with

LC_MESSAGES=C /sbin/start_udev

in rc.sysinit.

2. Current version of console_init does all that it should do. Now we need some changes in /etc/profile.d/lang.* for non-UTF console:

- execute '/bin/kbd_mode -a' to switch keyboard to ascii mode (console_init swithes only tty1);

- echo -ne '\e%@' - select default charset;

- echo -ne '\e(K' - select user mapping.

I suppose, also it's a good idea to add

/bin/kbd_mode -u
echo -ne '\e%G'

for UTF console in /etc/profile.d/lang.*.

That's all.
Comment 8 Bill Nottingham 2008-11-05 15:01:43 EST
(In reply to comment #7)
> 2. Current version of console_init does all that it should do. Now we need some
> changes in /etc/profile.d/lang.* for non-UTF console:
> 
> - execute '/bin/kbd_mode -a' to switch keyboard to ascii mode (console_init
> swithes only tty1);

No, it switches /dev/console. If that gets mangled along the way so that it only affects /dev/tty1, that's a kernel bug.

> - echo -ne '\e%@' - select default charset;
> 
> - echo -ne '\e(K' - select user mapping.

This is not something we've ever done before, so I'm not seeing why it would be needed now.
Comment 9 Vsevolod Volkov 2008-11-06 10:20:17 EST
>> - execute '/bin/kbd_mode -a' to switch keyboard to ascii mode (console_init
>> swithes only tty1);
>
> No, it switches /dev/console. If that gets mangled along the way so that it
> only affects /dev/tty1, that's a kernel bug.

If you are sure, please submit bug report.

There was no such problem with Fedora8 initscripts because kbd_mode was called from /etc/profile.d/lang.* via unicode_stop/unicode_start. That worked fine.

>> - echo -ne '\e%@' - select default charset;
>> 
>> - echo -ne '\e(K' - select user mapping.
>
> This is not something we've ever done before, so I'm not seeing why it would be
> needed now.

It is not enough to send the escape sequences only once. It must be sent after every login because virtual console drops the state after logout.
Comment 10 Vsevolod Volkov 2009-01-01 12:47:17 EST
I would like to suggest two patches and close the ticket.
Comment 11 Vsevolod Volkov 2009-01-01 13:00:02 EST
Created attachment 328038 [details]
Set language after starting udev

The patch removes setting language from /sbin/start_udev and from the beginning of /etc/rc.d/rc.sysinit and sets language after starting udev.
Comment 12 Vsevolod Volkov 2009-01-01 13:15:27 EST
Created attachment 328039 [details]
/etc/profile.d/lang.* fix

The patch sets utf/ascii mode of keyboard and screen. Also selects user mapping if defined SYSFONTACM variable.
Comment 13 Bug Zapper 2009-06-09 21:15:33 EDT
This message is a reminder that Fedora 9 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 9.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '9'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 9's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 9 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 14 Bug Zapper 2010-11-04 07:54:07 EDT
This message is a reminder that Fedora 12 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 12.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '12'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 12's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 12 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 15 Vsevolod Volkov 2010-11-04 08:14:29 EDT
/etc/profile.d/lang.* patch is still actual in Fedora 14.
Comment 16 Peter Backes 2011-06-17 17:15:31 EDT
Same problem actually for setting LANG=en_US (which means ISO8859-1 instead of UTF-8). Problem persists in Fedora 15.
Comment 17 Fedora End Of Life 2012-08-07 15:37:18 EDT
This message is a notice that Fedora 15 is now at end of life. Fedora
has stopped maintaining and issuing updates for Fedora 15. It is
Fedora's policy to close all bug reports from releases that are no
longer maintained. At this time, all open bugs with a Fedora 'version'
of '15' have been closed as WONTFIX.

(Please note: Our normal process is to give advanced warning of this
occurring, but we forgot to do that. A thousand apologies.)

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, feel free to reopen
this bug and simply change the 'version' to a later Fedora version.

Bug Reporter: Thank you for reporting this issue and we are sorry that
we were unable to fix it before Fedora 15 reached end of life. If you
would still like to see this bug fixed and are able to reproduce it
against a later version of Fedora, you are encouraged to click on
"Clone This Bug" (top right of this page) and open it against that
version of Fedora.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

The process we are following is described here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 18 Vsevolod Volkov 2012-08-08 03:58:10 EDT
Please, remove completely non-UFT console support if you don't want to fix it for a long time. Now console support of uk_UA.KOI8-U, ru_RU.KOI8-R and other non-UTF locales requires patching init scripts.
Comment 19 Bill Nottingham 2012-08-08 16:59:04 EDT
vconsole setup is done in systemd these days - lang.sh is only for setting locale settings; it doesn't change other settings.
Comment 20 Lennart Poettering 2012-08-08 21:37:35 EDT
What is this about? What precisely is there to fix in systemd-vconsole? Can somebody summarize for me what this is about?
Comment 21 Vsevolod Volkov 2012-08-13 05:22:20 EDT
1. systemd does not switch keyboard to ASCII mode (kbd_mode -a) at all virtual consoles except tty1.

2. Every time just after login in every virtual console default (not UTF-8) character set should be selected by escape sequence Esc % @.

3. Every time just after login in every virtual console user mapping character set should be selected by escape sequence Esc ( K.

lang.sh did it before Fedora 9 and all worked fine.
Comment 22 Bill Nottingham 2012-08-13 13:08:21 EDT
Why is 2 and 3 needed? Where is the setting being lost on logout?
Comment 23 Vsevolod Volkov 2012-08-14 02:35:01 EDT
Yes. But 'kbd_mode -a' should be started at least once for every console.
Comment 24 Lennart Poettering 2012-08-14 11:15:06 EDT
Hmm, isn't the "kbd_mode -a" setting inherited by the other virtual consoles? We can't really run this on all consoles since consoles might get de-allocated/re-allocated all the time, hence we really need some intra-VC inheritance here to work reliably. Isn't that something to fix in the kernel?

Is there documentation available about Esc%@ and Esc(K? console_codes(4) does not list them, or am I blind?
Comment 25 Vsevolod Volkov 2012-08-27 04:54:45 EDT
"kbd_mode -a" really isn't inherited by all virtual consoles. I'm not sure it should be inherited. Probably kernel developers consider that every console can be switched up to different mode separately.

Esc%@ and Esc(K are both listed in console_codes(4):

ESC %              Start sequence selecting character set
ESC % @               Select default (ISO 646 / ISO 8859-1)
ESC % G               Select UTF-8
ESC % 8               Select UTF-8 (obsolete)
ESC (              Start sequence defining G0 character set
ESC ( B               Select default (ISO 8859-1 mapping)
ESC ( 0               Select VT100 graphics mapping
ESC ( U               Select null mapping - straight to character ROM
ESC ( K               Select user mapping - the map that is loaded by
                      the utility mapscrn(8).
Comment 26 Bill Nottingham 2012-08-27 12:07:09 EDT
Given that consoles can't really have separate fonts (it is highly dependent on the console driver in use), I'm not sure how having each console with a different keyboard mode makes much sense.
Comment 27 Fedora End Of Life 2013-07-04 02:54:17 EDT
This message is a reminder that Fedora 17 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 17. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '17'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 17's end of life.

Bug Reporter:  Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 17 is end of life. If you 
would still like  to see this bug fixed and are able to reproduce it 
against a later version  of Fedora, you are encouraged  change the 
'version' to a later Fedora version prior to Fedora 17's end of life.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.
Comment 28 Fedora End Of Life 2013-08-01 14:34:34 EDT
Fedora 17 changed to end-of-life (EOL) status on 2013-07-30. Fedora 17 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.
Comment 29 Peter Backes 2017-06-14 19:02:48 EDT
This is still a problem as of F25. A locale like en_US is not honored on the console. Please reopen.

echo -e '\e%@\c' seems to work fine for the output; but kbd_mode -a does not entirely fix the input. On my german keyboard, I then get working latin1 umlauts (äöüÄÖÜ), but hitting the sharp s key (ß) still produces the letter in UTF-8 encoding, not in latin1.
Comment 30 Peter Backes 2017-06-14 19:17:12 EDT
Further, stty reports that iutf8 is set, which shouldn't be for a non-UTF locale. The flag says that backspace in canonical mode deletes utf8 multi-byte-characters instead of single bytes. If iutf8 is set on a non-UTF-locale, the input is thus corrupted if the backspace key is used over a sequence of non-ASCII characters.

Note You need to log in before you can comment on or make changes to this bug.