Bug 528279 - Improperly renders UTF-8 characters
Summary: Improperly renders UTF-8 characters
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: aumix
Version: rawhide
Hardware: All
OS: Linux
low
medium
Target Milestone: ---
Assignee: Gabriel Somlo
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On: 529790
Blocks:
TreeView+ depends on / blocked
 
Reported: 2009-10-10 17:35 UTC by Alexey Torkhov
Modified: 2009-10-20 15:56 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2009-10-20 15:32:26 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
Screenshot (22.53 KB, image/png)
2009-10-10 17:36 UTC, Alexey Torkhov
no flags Details
Screenshot (26.52 KB, image/png)
2009-10-10 18:02 UTC, Alexey Torkhov
no flags Details
ru.po converted to utf8 (8.59 KB, application/octet-stream)
2009-10-19 17:01 UTC, Alexey Torkhov
no flags Details
ru.po patch from old to utf8 (8.49 KB, application/octet-stream)
2009-10-19 17:01 UTC, Alexey Torkhov
no flags Details
patch for all .po files, but better not to apply it, but convert in %prep or like (76.40 KB, application/octet-stream)
2009-10-19 17:07 UTC, Alexey Torkhov
no flags Details

Description Alexey Torkhov 2009-10-10 17:35:57 UTC
Description of problem:
aumix improperly renders UTF-8 characters.

Version-Release number of selected component (if applicable):
aumix-2.8-20.fc12.x86_64

How reproducible:
Always.

Steps to Reproduce:
1. Load OSS, as there is no one in F-12 now: modprobe snd-mixer-oss
2. LANG=ru_RU aumix
  
Actual results:
Cyrillic characters are rendered improperly - see screenshot.
Same if run with fr_FR locale - characters that should be with accents are wrong.

Expected results:
All is fine.

Additional info:

Comment 1 Alexey Torkhov 2009-10-10 17:36:56 UTC
Created attachment 364347 [details]
Screenshot

Comment 2 Alexey Torkhov 2009-10-10 18:02:56 UTC
Created attachment 364348 [details]
Screenshot

Right locale to start is
LANG=ru_RU.UTF-8 aumix

And it does work fine with fr_FR.UTF-8 (shows accents).
Doesn't work, for example, with:
LANG=zh_CN.UTF-8 aumix

Comment 3 Gabriel Somlo 2009-10-14 15:09:16 UTC
This is due to the language files supplied with aumix using charsets other than UTF-8. For instance, the ru.po file in aumix uses koi8-r, and the zh_CN.po one uses gb2312. Works fine if your LANG is set to ru_RU instead of ru_RU.UTF-8 (or zh_CN instead of zh_CN.UTF-8).

I'd be happy to replace the aumix ru.po file (or any other .po for that mater) with an updated version which uses UTF-8 if anyone uploads it (to this bug report would be fine). I am however not qualified (can't write Russian or Chinese, unfortunately) to do it myself.

Comment 4 Alexey Torkhov 2009-10-14 18:40:17 UTC
Oh, it has easy fix then—you need just to recode all .po files with recode or iconv in %prep section, like:

iconv -f iso-8859-2 -t utf-8 pl.po > pl.po.new; mv pl.po.new pl.po
iconv -f koi8-r -t utf-8 ru.po > ru.po.new; mv ru.po.new ru.po
iconv -f koi8-u -t utf-8 uk.po > uk.po.new; mv uk.po.new uk.po
...

I've checked—them seem to be sane after that. You can use sed magic to automate the process :)

grep charset *.po | sed 's|^\(\w*\.po\):.*charset=\([-a-zA-Z0-9]*\)\\n"|iconv -f \2 -t utf-8 \1 > \1.new; sed -i s/charset=\2/charset=utf-8/ \1.new; mv \1.new \1|e'

Comment 5 Gabriel Somlo 2009-10-19 16:23:27 UTC
Tried the iconv trick (nice sed kung-fu, BTW :) ) but unfortunately the converted
files still look wrong after rebuilding aumix. Not at first glance in an editor mind you, but after compilation.

I diff-ed the so-called "utf-8" ru.po against the original koi8-r one, and the patch didn't look right when viewd in the same editor and 
LANG=ru_RU.UTF-8 setting under which the iconv-ed utf8 ru.po seemed fine.

Applying the patch on top of the original koi8-r ru.po also resulted in a file which no longer looked right, in the same editor with the same settings.

Any chance you can send me a ru.po file generated in a clean ru_RU.UTF-8 environment, or even better, a patch against the existing aumix ru.po file in which the "+" (add) lines look right in a ru_RU.UTF-8 environment ? I could then try to apply that patch and see if that results in some progress...

This also makes me wonder whether diff itself handles utf-8 properly, but I guess we'll find out soon enough...

Comment 6 Alexey Torkhov 2009-10-19 17:01:18 UTC
Created attachment 365250 [details]
ru.po converted to utf8

Comment 7 Alexey Torkhov 2009-10-19 17:01:58 UTC
Created attachment 365251 [details]
ru.po patch from old to utf8

Comment 8 Alexey Torkhov 2009-10-19 17:07:13 UTC
Created attachment 365252 [details]
patch for all .po files, but better not to apply it, but convert in %prep or like

Comment 9 Alexey Torkhov 2009-10-19 17:12:42 UTC
I’ve attached files you asked and even patch for all .po files, which, I think, should be result of iconv conversion.

Comment 10 Gabriel Somlo 2009-10-20 02:33:54 UTC
I think this is a bug in ncurses (which also explains the weirdness I observed while trying to look at diff results in vim :) )

I opened bug 529790 against ncurses, let's see where that gets us.

Comment 11 Alexey Torkhov 2009-10-20 07:19:51 UTC
I'm looking at diff with vim too and it shows it fine. May be, that's something with terminal (I'm using gnome-terminal)?

Comment 12 Alexey Torkhov 2009-10-20 07:42:50 UTC
(In reply to comment #11)
> I'm looking at diff with vim too and it shows it fine. May be, that's something
> with terminal (I'm using gnome-terminal)?  

Oh, checked the bug, right, it seem not to work properly.

Comment 13 Gabriel Somlo 2009-10-20 15:32:26 UTC
Shoot, I entered the wrong bug number in bodhi, but here's a link to the new package:

aumix-2.8-21.fc12 has been submitted as an update for Fedora 12.
http://admin.fedoraproject.org/updates/aumix-2.8-21.fc12

I checked it with LANG set to ru_RU.UTF-8 and zh_CN.UTF-8, and they both seem to work now.

Comment 14 Alexey Torkhov 2009-10-20 15:56:13 UTC
(In reply to comment #13)
> Shoot, I entered the wrong bug number in bodhi, but here's a link to the new
> package:

You can edit update, I guess.

> aumix-2.8-21.fc12 has been submitted as an update for Fedora 12.
> http://admin.fedoraproject.org/updates/aumix-2.8-21.fc12
> 
> I checked it with LANG set to ru_RU.UTF-8 and zh_CN.UTF-8, and they both seem
> to work now.  

Yeah, it fixes this for me too.


Note You need to log in before you can comment on or make changes to this bug.