Bug 186961 - UTF-8 terminal/console not in IUTF8 mode
UTF-8 terminal/console not in IUTF8 mode
Status: CLOSED WORKSFORME
Product: Fedora
Classification: Fedora
Component: util-linux-ng (Show other bugs)
9
All Linux
medium Severity medium
: ---
: ---
Assigned To: Karel Zak
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2006-03-27 13:22 EST by Leszek Matok
Modified: 2008-11-25 03:57 EST (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2008-11-25 03:57:18 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
patch for /etc/profile.d/lang.csh (330 bytes, patch)
2006-03-27 13:24 EST, Leszek Matok
no flags Details | Diff
patch for /etc/profile.d/lang.sh (421 bytes, patch)
2006-03-27 13:24 EST, Leszek Matok
no flags Details | Diff

  None (edit)
Description Leszek Matok 2006-03-27 13:22:40 EST
Description of problem:
It's in the summary. The most visible effect is that whenever I use shell with
LANG pl_PL.UTF-8, backspace works improperly.

Version-Release number of selected component (if applicable):
My solution will be a patch to current (FC5) initscripts-8.31.1-1

How reproducible:
Always since RHL 7.9x

Steps to Reproduce:
1. Login on the console or run gnome-terminal with (GDM_)LANG set to anything.UTF-8.
2. Run `cat`
3. Enter two multibyte characters, like "óó" ("ó" has two bytes - 0xc3 0xb3).
4. Press Backspace two times - both "ó"-s disappear.
5. Press Enter - cat shows one "ó", because two Backspaces removed two bytes
(the second "ó") and not two characters.
  
Expected results:
Backspace should erase characters, not bytes.

Additional info:
The problem until FC3 (I haven't upgraded to FC4) was not with the kernel, but
with coreutils 5.2.1. With coreutils 5.3.0 (FC5 has 5.93) there's a new option
to stty, "iutf8", enabling IUTF8 mode in the kernel.

This affects more programs than just "cat" - most troublesome for me (and some
of my MUD's players) is telnet.

I'll attach patches now. I don't check for stty presence because initscripts
require coreutils (precisely: fileutils and sh-utils provided by coreutils), so
it will be there. They work for me both under console (unicode_start doesn't set
IUTF8) and in gnome-terminal. It's strange it's not there.
Comment 1 Leszek Matok 2006-03-27 13:24:01 EST
Created attachment 126840 [details]
patch for /etc/profile.d/lang.csh
Comment 2 Leszek Matok 2006-03-27 13:24:54 EST
Created attachment 126841 [details]
patch for /etc/profile.d/lang.sh
Comment 3 Leszek Matok 2006-06-24 18:38:49 EDT
From looking at initscripts and kbd packages, FC6t1 still has broken UTF-8
consoles. It's a matter of only one simple line, with even patches included here
:) (or you can choose to hack start_unicode instead) If you don't care that
Fedora is broken even if so simple distribution-independant fix is available for
years, at least reject this bug with some reason, please :)
Comment 4 Bill Nottingham 2006-06-28 23:27:50 EDT
Actually, the kernel should be in utf-8 mode by default.
Comment 5 Leszek Matok 2006-06-29 05:27:51 EDT
But it still (as of 2.6.17-1.2139_FC5) isn't. If it's going to make iutf8 the
default in the future or even it is already done in rawhide kernels (which I
don't really believe - who would patch the kernel when all you need is one line
of shell script?), then non-UTF-8 locales should disable this mode, which isn't
done in lang.[c]sh either.
Comment 6 Bill Nottingham 2006-10-06 14:15:09 EDT
This was added in 8.44-1. However, it caused too many regressions.

- Calling stty from a background process will hang, with SIGTTOU. This broke
  startx (bug 209469), as the xinitrc process is a background process on the
  terminal (startx itself is the fg process)
- This can be worked around by doing 'trap "" SIGTTOU ; stty iutf8 ; trap - 
  SIGTTOU". But that only works for bash....
- which means pretty much any standard csh script will hang if backgrounded.

Reverted for 8.45-1, in search of a better solution.
Comment 7 Leszek Matok 2006-10-09 12:48:25 EDT
I have to admit my csh scripts running in the background all start with
"#!/bin/csh -f" (ignoring cshrc) and the ones without -f weren't tested by me on
a Unicode tty.

If there's no way to tell if the shell is in the background, maybe the solution
is to do it only for a login shell? That's sufficient for any interactive
program withing a login session, because stty needs to be run only once. Frankly
the only reason that I can think of for having lang.*sh executed per every shell
is that some terminal emulators can be set to spawn non-login, but interactive
shells and we can't detect interactive shells (I've searched for some sure way
for tcsh without luck). Am I right?

Quick Google search revealed that Solaris people fixed something similar with
stty by making the kernel not emit SIGTTOU for certain ioctls.


Also, in today's testing I've been able to spot another problem and fix it -
using scp to the machine resulted with "not a terminal" error message. It can be
simply fixed by doing something like this (version working for tcsh):

/usr/bin/tty -s
if ( ! $? ) then
    /bin/stty iutf8
endif
Comment 8 Bill Nottingham 2006-10-09 13:54:47 EDT
> If there's no way to tell if the shell is in the background, maybe the solution
> is to do it only for a login shell? That's sufficient for any interactive
> program withing a login session, because stty needs to be run only once. Frankly
> the only reason that I can think of for having lang.*sh executed per every shell
> is that some terminal emulators can be set to spawn non-login, but interactive
> shells and we can't detect interactive shells (I've searched for some sure way
> for tcsh without luck). Am I right?

Correct. For example, just running 'bash' by default isn't a login shell,
but it may be interactive.

> Also, in today's testing I've been able to spot another problem and fix it -
> using scp to the machine resulted with "not a terminal" error message. It can be
> simply fixed by doing something like this (version working for tcsh):
> 
> /usr/bin/tty -s
> if ( ! $? ) then
>     /bin/stty iutf8
> endif

This was fixed in the patch in 8.44-1.
Comment 9 Leszek Matok 2006-10-09 14:31:58 EDT
Interactive shells run from already "unicoded" terminal are fine, there's no
need to re-set terminal modes to the same values, no matter if I run bash from
tcsh or the other way. I was talking about a situation when a shell is run as a
first process controlling a new terminal (in xterm, screen etc.), but thinks
it's not a login shell. If only there was a way to check if the shell is
interactive...

If there's no way, should I bug Fedora kernel people or lkml about SIGTTOU from
tcsetattr( ) or whatever stty calls?
Comment 10 Bill Nottingham 2006-10-09 14:37:10 EDT
IIRC, things like xterm only start the shell as login shells with a specific
commandline/config option. Of course, gnome-terminal (well, vte) sets iutf8
itself, without worrying about the shell.

You could ask about SIGTTOU, but I'm not 100% sure the current behavior is wrong.
Comment 11 Leszek Matok 2006-10-09 14:52:41 EDT
With gnome-terminal-2.14.2-1 and vte-0.12.2-1.fc5.1 I still need stty iutf8 to
have it enabled. I'll try with FC6 when it's available.
Comment 12 petrosyan 2008-03-13 19:28:32 EDT
Fedora Core 5 and Fedora Core 6 are no longer maintained. Is this bug still
present in Fedora 7 or Fedora 8?
Comment 13 Leszek Matok 2008-03-16 08:02:57 EDT
It seems fixed in F8 and Rawhide, but I have no idea about when and how that got
fixed. It certainly doesn't use stty in lang.{c,}sh, maybe the kernel or glibc
sets the flag upon pty creation now?
Comment 14 Leszek Matok 2008-03-16 08:17:06 EDT
Scratch that - it's NOT fixed, or rather, it's only partly fixed. Previously I
tested gnome-terminal and xterm started from gnome-terminal - both had iutf8
mode enabled. Probably the terminal emulators set the flag when started from
UTF8 environment.

But logging in from tty1 still has the problem, both in F8 and Rawhide (which is
now at F9 beta). This problem is minor in todays GUI world, but I do use the
console, even on my desktop machines (esp. Rawhide ;)).
Comment 15 Bill Nottingham 2008-03-17 11:24:44 EDT
I suspect this really needs fixed in the kernel tty layer.
Comment 16 Chuck Ebbert 2008-03-19 12:13:57 EDT
Shouldn't /bin/login set the correct mode when the user logs on to a terminal?
What could the kernel do here?
Comment 17 Bug Zapper 2008-05-13 22:07:53 EDT
Changing version to '9' as part of upcoming Fedora 9 GA.
More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 18 Alan Cox 2008-07-28 12:01:11 EDT
Reassigning to the login owner, not a kernel problem
Comment 19 Karel Zak 2008-07-30 18:07:44 EDT
I have doubts that login(1) is the right place to set IUTF8. The login(1) does
not setup terminal, it's usually job for {a,min,...}getty program. I see in
mingetty.c:

  * TODO:
  * - Can UTF-8 setup be done within mingetty?

Comment 20 Karel Zak 2008-07-31 04:27:15 EDT
(In reply to comment #14)

> But logging in from tty1 still has the problem, both in F8 and Rawhide
> (which is now at F9 beta). This problem is minor in todays GUI world,
> but I do use the console, even on my desktop machines (esp. Rawhide ;)).

Leszek, I see IUTF8 flags enabled:

	$ tty
	/dev/tty1

	$ stty --all | grep iutf8
	-iuclc -ixany -imaxbel iutf8

[Fedora F9]

Comment 21 Leszek Matok 2008-11-18 17:20:34 EST
You're right - it works on the console as well (just checked with F10).

Anyone knows where the fix lays? :)
Comment 22 Karel Zak 2008-11-24 04:35:19 EST
(In reply to comment #21)
> You're right - it works on the console as well (just checked with F10).

 So we can close this issue, right?
Comment 23 Leszek Matok 2008-11-24 16:04:45 EST
I think so.

But can you say, when and where it got fixed?
Comment 24 Karel Zak 2008-11-25 03:57:18 EST
(In reply to comment #23)
> But can you say, when and where it got fixed?

 Sorry, I have no clue (maybe kernel, init scripts, getty, ...).

Note You need to log in before you can comment on or make changes to this bug.