Bug 329071 - Incorrect UTF-8 character encodings for Romanian language
Summary: Incorrect UTF-8 character encodings for Romanian language
Keywords:
Status: CLOSED DUPLICATE of bug 337271
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kbd
Version: 5.4
Hardware: All
OS: Linux
low
high
Target Milestone: ---
: ---
Assignee: Vitezslav Crhonek
QA Contact:
URL: http://www.secarica.ro
Whiteboard:
: 221777 444869 (view as bug list)
Depends On: 294461 294471 294531 337271 353721
Blocks:
TreeView+ depends on / blocked
 
Reported: 2007-10-12 06:20 UTC by Răzvan Sandu
Modified: 2008-05-13 10:58 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2008-05-13 09:57:30 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Răzvan Sandu 2007-10-12 06:20:46 UTC
Description of problem:

Hello,

When system is configured for Romanian language (LANG="ro_RO.UTF-8",
KEYBOARD="ro_win"), pressing keys on the keyboard insert wrong UTF-8 characters
in documents:

- "S with comma below" (Unicode 0218) - instead of "S with cedilla below"
(Unicode 015E)
- "s with comma below" (Unicode 0219) - instead of "s with cedilla below"
(Unicode 015F)
- "T with comma below" (Unicode 021A) - instead of "T with cedilla below"
(Unicode 0162)
- "t with comma below" (Unicode 021B) - instead of "t with cedilla below"
(Unicode 0163)

This is wrong, according to official Romanian National Standards SR 13411:1999
and SR 13392:2004 and International Standard ISO/IEC 8859-16 (Latin 10).

Please correct this in both console and X-based programs.

The implementation in Microsoft Windows systems is correct, starting with
Microsoft Windows Vista.

Version-Release number of selected component (if applicable):
kbd-1.12-25.fc8

How reproducible:
Always.

Steps to Reproduce:
1. Install a fresh system.
2. Configure it with system-config-language and system-config-keyboard for
Romanian language.
3. Open a text editor (say OpenOffice.org Writer or vi).
4. Insert Romanian-specific characters by pressing AltRight-s, AltRight-Shift-s,
AltRight-t, AltRight-Shift-t.
5. Note that incorrect cedilla-versions of characters are generated, instead of
the correct comma-versions (see Unicode codes above).
  
Actual results:
Cedilla-versions of characters are generated.

Expected results:

RedHat systems should use comma-versions of characters, in both X- or
console-based programs.

Additional info:

In Romanian:
http://www.secarica.ro
http://diacritice.sourceforge.net/tastaturi.html

Comment 1 Răzvan Sandu 2007-10-12 06:24:37 UTC
Sorry, I made a huge mistake in the description above, when listing Unicode codes:

The CORRECT versions of characters are the comma-ones, the incorrect versions
are the cedilla-ones.


Regards,
Razvan

Comment 2 Răzvan Sandu 2007-10-12 06:28:23 UTC
*** Bug 221777 has been marked as a duplicate of this bug. ***

Comment 3 Vitezslav Crhonek 2007-10-12 09:49:29 UTC
Thanks for reporting.

I've recetly inserted new ro.map from Alexandru Szasz to kbd (see #253892). If
you chceck the map file, you can see that cedilla-ones are used instead of
comma-ones because there is no font able to display them.

So what I'd like to do:

a) fix the ro.map file to right unicode values
b) fix the unicode map in LatArCyrHeb-16 font (now default for Romanian language
in console). Same glyph as before will be used, but with right unicode value.

Comment 4 Răzvan Sandu 2007-10-12 11:01:31 UTC
Thanks a lot !

If you do the above, things will start to get straight - it is a veeeeeeeeery
old bug on Romanian computers, due to some wrong, historical, Microsoft
implementations in the '80s... AND to the disinterest of the Romanian
authorities (Ceausescu's) in that time... ;-)


Regards,
Razvan


Comment 5 Răzvan Sandu 2007-10-12 11:17:29 UTC
Is there a way to notice the font "manufacturers" - so all future fonts will
come with the right maps by default ?

This problem is not limited to Red Hat/Fedora, but I'm pretty sure it's present
in other major distros too (Debian, Ubuntu, CentOS, SuSE, etc.)

Regards,
Răzvan

Comment 6 Răzvan Sandu 2007-10-12 11:27:44 UTC
At least in OpenOffice Writer's special character table (Insert -> Special
character) on Fedora 8, under "Latin Extended-B", I see glyphs which are able to
properly display the Romanian correct comma-versions.

Regards,
Răzvan

Comment 7 Vitezslav Crhonek 2007-10-15 14:38:35 UTC
Please try kbd-1.12-26.fc8.

Comment 8 Răzvan Sandu 2007-10-17 17:57:58 UTC
Hello and many thanks,

I will test with it, as soon kbd-1.12-26.fc8 shows up in the Fedora development
repository.

Regards,
Răzvan


Comment 9 Răzvan Sandu 2007-10-17 19:25:00 UTC
Hello,


Here is the test after installing kbd-1.12-26.fc8:

1. In text mode, I have no support for Romanian-specific characters at all (for
example, pressing AltRight+s) gives plain "s".

2. In X mode, the incorrect cedilla-versions of the Romanian characters are
still inserted/displayed when pressing AltRight+character.

3. Initialization scripts display cedilla-versions of Romanian characters, with
no garbled character.


Regards,
Răzvan



Comment 10 Vitezslav Crhonek 2007-10-18 11:22:55 UTC
Hi Razvan,

1. Do you have right keymap and font selected? As root do:

loadkeys /lib/kbd/keymaps/i386/qwerty/ro.map
setfont LatArCyrHeb-16

And then try press AltGr+s. (For further information see #294531.)

2. X mode is AFAIK something different and is not in scope of kbd component.


Comment 11 Răzvan Sandu 2007-10-18 12:50:01 UTC
Hello,

I did the following:

1. Exited X
2. Issued the two commands above (as root).
3. Pressed AltGr-s and AltGr-t.

Result:
When pressing AltGr-s, the incorrect cedilla-character is displayed.
When pressing AltGr-t, the simple t (with no comma or cedilla) is displayed.

However, what must be fixed is the way the programs work in the distro as a
whole, in both text and X mode. Could you please add in Cc: the right persons,
so the maintainers for the X part be kept informed as well ?


Many thanks,
Răzvan


Comment 12 Răzvan Sandu 2007-10-18 12:53:46 UTC
Please note that, before the test I did in comment #9, I just used the standard
GUI tools to configure language and keyboard: System -> Administration ->
Language and System -> Administration - Keyboard.

Did no other "hacking" at all...

Răzvan


Comment 13 Alexandru Szasz 2007-10-18 13:49:25 UTC
Hello Răzvan,

The keymaps from text mode and graphic mode are different.

As regarding the cedilla/comma, please note that there are still many fonts that
DO NOT have the characters with comma so it would be a frustrating thing to set
the default with comma now. First all the fonts should have the comma characters.

I strongly recomment to NOT set the variant with comma as default but rather
leave it as an option.

Windows XP is unable to display the comma characters with most of the fonts.
There is a fix on microsoft.com but regular users don't care and don't know usually.

In X, Fedora right now the comma is an option (setxkbmap ro comma), and the
sedila is the default.

For consistency, I recommend we leave the cedilla as default in text mode also,
otherwise there will be pain with samba share names that have cedilla characters
and other platforms won't be able to display them.

As for the „t” problem, it really is a problem.

kbd-1.12-27 has a defective keymap that differs from the one I submitted in bug
#253892

I'll reopen that bug.

Răzvan, in the meantime you can try downloading
https://bugzilla.redhat.com/attachment.cgi?id=162824 to
/lib/kbd/keymaps/i386/qwerty/ro.map.gz and try to write with diacritics.

I'll propose an alternative keymap for commas, that should be optional.

Comment 14 Alexandru Szasz 2007-10-18 13:51:10 UTC
I meant comma characters in the following sentence:

For consistency, I recommend we leave the cedilla as default in text mode also,
otherwise there will be pain with samba share names that have cedilla characters
and other platforms won't be able to display them.

Comment 15 Alexandru Szasz 2007-10-18 14:09:33 UTC
Right now there is NO text font that has the ş and ţ characters with comma.
I was lucky enough to find a font with all the rest of the romanian characters.
Sticking to sedilla is the only option until a font with commas exists. I'll see
what I can do regarding that.

Also ro_win can be removed safely, it's unusable anyway.
I'll prepare a variant with comma for the future and two more with sedilla/comma
but with the romanian layout instead of the altgr solution.

Note: there are very very few keyboards with romanian layout, that's why the
altgr solution is the default one.

Comment 16 Răzvan Sandu 2007-10-18 14:33:24 UTC
Hello,

Thanks a lot, Alex, about clarification !

However, I'm very unsatisfied about that state of facts.

Is there anything we can do - in a relatively short term - to get some fonts
with *proper* Romanian diacritics ? Who's the actual author of the fonts we use
*now* in Fedora/Red Hat, anyway ?

I'm afraid that continuing to use cedilla-characters for "compatibility"  sake
will propagate indefinitely this state of facts, due to the lack of interest
from the large mass of Windows users and the very slow acceptance of Windows
Vista (where this bug is fixed, as far as I know).

Please note also another bug I've opened, bug #337271 - the standardised
Romanian arrangements for keyboard are still not present in Red Hat/ Fedora.


Regards,
Răzvan

Comment 17 Alexandru Szasz 2007-10-18 14:51:56 UTC
Hello Răzvan,

There is something we can do, but please contact me by e-mail so we don't spam
this application.

Meantime, Vitezslav can close this after fixing
https://bugzilla.redhat.com/show_bug.cgi?id=253892#c3

As I said I'll provide arrangements for the romanian layout as well (not the alt
gr version, but the one with diacritics on the keyboard).

Comment 18 Alexandru Szasz 2007-11-16 13:24:49 UTC
For text mode, this was fixed in bug 253892.

Comment 19 Răzvan Sandu 2008-05-04 16:35:12 UTC
*** Bug 444869 has been marked as a duplicate of this bug. ***

Comment 20 Răzvan Sandu 2008-05-08 06:15:42 UTC
The issue about Romanian incorrectly-generated characters was recently corrected
in bug #13277 at freedesktop.org.

Please see comments #82 and #83, with the appropriate patch:

http://bugs.freedesktop.org/show_bug.cgi?id=13277

Would you please include this correction in Red Hat/Fedora ASAP, since it
affects *thousands* of documents and webpages generated in Romanian language ?

Thanks a lot,
Răzvan


Comment 21 Vitezslav Crhonek 2008-05-13 09:57:30 UTC
To make summary:

Bug is fixed in RHEL5 (kbd-1.12-20). Strictly speaking - console part of this
issue. There is bug #337271 to track X part of issue, so I'll close this bz as
duplicate of #337271. It's not in a scope of kbd package.

Console part is also fixed in Fedora, still it isn't in RHEL4 (bug #294461).

Am I right?

*** This bug has been marked as a duplicate of 337271 ***

Comment 22 Alexandru Szasz 2008-05-13 10:02:50 UTC
Yes, Fedora 9 has it fixed, can't say about RHEL, I'm not using it.

Comment 23 Răzvan Sandu 2008-05-13 10:31:53 UTC
Hello, Alex !


Can you please sketch a step-by-step procedure to find out if/how is that fixed
in Fedora 9 ?

I'm running Fedora 9 preview release with all updates (upgraded from Fedora 8
via preupgrade) and I have three problems:


- the graphical utility for setting the keyboard language
(system-config-keyboard) is nowhere in the Gnome menus;

- if invoked manually from a terminal, system-config-keyboard let me choose
"Romanian" from the list, but that's all. There is no way to choose from the
five possible settings, namely:

    a. Primary Romanian layout with comma-below characters
    b. Secondary Romanian layout with comma-below characters (should be DEFAULT
for Romanian language)
    c. Primary Romanian layout with cedilla-below characters (compatibility mode
still necessary)
    d. Secondary Romanian layout with cedilla-below characters (compatibility
mode still necessary)
    e. Romanian Legacy/Martin Poehler (compatibility mode still necessary)


- using AltGr+key in applications to generate diacritics, I'm still inserting
cedilla-below characters (Thunderbird, mutt, OpenOffice.org, etc.).


How do you suggest to fix all this ?


Thanks a lot,
Răzvan


Comment 24 Alexandru Szasz 2008-05-13 10:58:05 UTC
Please see comment https://bugzilla.redhat.com/show_bug.cgi?id=337271#c13


Note You need to log in before you can comment on or make changes to this bug.