Bug 871916

Summary: [sr_RS.UTF-8@latin] user directories created with Cyrillic, not Latin, characters
Product: [Fedora] Fedora Reporter: Steve Tyler <stephent98>
Component: xdg-user-dirsAssignee: Matthias Clasen <mclasen>
Status: CLOSED EOL QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 22CC: dmach, mfabian, msivak, smaitra, stephent98, tagoh
Target Milestone: ---Keywords: i18n
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-07-19 10:05:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
screenshot showing directory names with Cyrillic characters
none
serbian-latin-user-directories-renaming-does-not-work-in-gnome.png
none
user-directories-created-in-cyrillic-fedora-22.png none

Description Steve Tyler 2012-10-31 17:29:23 UTC
Created attachment 636263 [details]
screenshot showing directory names with Cyrillic characters

Description of problem:
With the 'sr_RS.UTF-8@latin' locale, the directories created in a new user's home directory have names with Cyrillic characters, not Latin characters.

Version-Release number of selected component (if applicable):
firstboot-0:18.4-1.fc18.x86_64

How reproducible:
Presumably always.

Steps to Reproduce:
1. During installation, select 'Serbian (Latin)' for the language and Gnome desktop for the environment.
2. Complete the install, reboot, and login as a normal user.
3. 'ls'
  
Actual results:
Directories have names with Cyrillic characters.
See attached screenshot.

Expected results:
Directories have names with Latin characters.

Additional info:
Reproduced while testing a fix for this bug:
Bug 866730 - invalid locales configured for some languages

Comment 1 Steve Tyler 2012-11-01 15:55:02 UTC
Reproduced in F17 by installing from the F17 DVD with 'Serbian (Latin)' and Gnome desktop.

Changing component to xdg-user-dirs.

The translation has Latin characters:
http://cgit.freedesktop.org/xdg/xdg-user-dirs/tree/po/sr@Latn.po

So the problem could be that the locale 'sr_RS.UTF-8@latin' does not get mapped to 'sr@Latn':

$ rpm -ql xdg-user-dirs | grep '/sr'
/usr/share/locale/sr/LC_MESSAGES/xdg-user-dirs.mo
/usr/share/locale/sr@Latn/LC_MESSAGES/xdg-user-dirs.mo

This is what was installed from the DVD:
xdg-user-dirs-0.14-2.fc17.x86_64
xdg-user-dirs-gtk-0.9-1.fc17.x86_64

Comment 2 Steve Tyler 2012-11-01 16:53:57 UTC
The locale is getting set to 'sr_RS'.

$ cat typescript-1 
Script started on Thu 01 Nov 2012 05:41:19 PM CET
[joeblow@localhost tmp]$ cat ~/.config/user-dirs.locale | xargs
sr_RS
[joeblow@localhost tmp]$ cat ~/.config/user-dirs.dirs 
# This file is written by xdg-user-dirs-update
# If you want to change or add directories, just edit the line you're
# interested in. All local changes will be retained on the next run
# Format is XDG_xxx_DIR="$HOME/yyy", where yyy is a shell-escaped
# homedir-relative path, or XDG_xxx_DIR="/yyy", where /yyy is an
# absolute path. No other format is supported.
# 
XDG_DESKTOP_DIR="$HOME/Радна површ"
XDG_DOWNLOAD_DIR="$HOME/Преузето"
XDG_TEMPLATES_DIR="$HOME/Шаблони"
XDG_PUBLICSHARE_DIR="$HOME/Јавно"
XDG_DOCUMENTS_DIR="$HOME/Документа"
XDG_MUSIC_DIR="$HOME/Музика"
XDG_PICTURES_DIR="$HOME/Слике"
XDG_VIDEOS_DIR="$HOME/Видео"
[joeblow@localhost tmp]$ rpm -qa 'xdg-user-dirs*'
xdg-user-dirs-0.14-2.fc17.x86_64
xdg-user-dirs-gtk-0.9-1.fc17.x86_64
[joeblow@localhost tmp]$ exit

Script done on Thu 01 Nov 2012 05:42:34 PM CET

Comment 3 Steve Tyler 2012-11-01 18:40:26 UTC
(In reply to comment #2)
> The locale is getting set to 'sr_RS'.
...

This code effectively removes tokens after the '.'. If the locale string is "sr_RS.UTF-8@latin", the result is "sr_RS".

$ less -N xdg-user-dirs-0.14/xdg-user-dirs-update.c
...
    702 save_locale (void)
...
    718   locale = strdup (setlocale (LC_MESSAGES, NULL));
    719   /* Skip encoding part */
    720   dot = strchr (locale, '.');
    721   if (dot)
    722     *dot = 0;
    723   fprintf (file, "%s", locale);
...

Comment 4 Fedora Admin XMLRPC Client 2013-05-09 00:44:48 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 5 Fedora End Of Life 2013-12-21 09:15:24 UTC
This message is a reminder that Fedora 18 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 18. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '18'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 18's end of life.

Thank you for reporting this issue and we are sorry that we may not be 
able to fix it before Fedora 18 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior to Fedora 18's end of life.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 6 Akira TAGOH 2014-01-15 06:48:59 UTC
Interesting thing is:
$ rpm -ql xdg-user-dirs-0.15-2.fc20.x86_64 xdg-user-dirs-gtk-0.10-3.fc20.x86_64| grep /sr@
/usr/share/locale/sr@Latn/LC_MESSAGES/xdg-user-dirs.mo
/usr/share/locale/sr@latin/LC_MESSAGES/xdg-user-dirs-gtk.mo

dunno if we should change the locale name from sr@Latn to sr@latin though.

Comment 7 Mike FABIAN 2014-01-15 09:57:49 UTC
Created attachment 850402 [details]
serbian-latin-user-directories-renaming-does-not-work-in-gnome.png

The problem still exists in Fedora 20, see screen shot.

Comment 8 Mike FABIAN 2014-01-15 11:38:16 UTC
Akira TAGOH> dunno if we should change the locale name from sr@Latn to
Akira TAGOH> sr@latin though.

Yes, "Latn" is not the way the Latin script is spelled in glibc locales.
"Latn" is the script code according to ISO 15924

http://www.unicode.org/iso15924/iso15924-codes.html

used by CLDR and libicu.

But glibc locales spell it "latin" and "Latn" it doesn’t work with gettext:

Check contents of the translation files:

    mfabian@ari:~
    $ msgunfmt /usr/share/locale/sr@Latn/LC_MESSAGES/xdg-user-dirs.mo | grep -A1 "Applications"
    msgunfmt /usr/share/locale/sr@Latn/LC_MESSAGES/xdg-user-dirs.mo | grep -A1 "Applications"
    msgid "Applications"
    msgstr "Programi"
    mfabian@ari:~
    $ msgunfmt /usr/share/locale/sr/LC_MESSAGES/xdg-user-dirs.mo | grep -A1 "Applications"
    msgunfmt /usr/share/locale/sr/LC_MESSAGES/xdg-user-dirs.mo | grep -A1 "Applications"
    msgid "Applications"
    msgstr "Програми"
    mfabian@ari:~
    $ msgunfmt /usr/share/locale/sr@latin/LC_MESSAGES/xdg-user-dirs-gtk.mo | grep -A1 "Current folder name"
    msgunfmt /usr/share/locale/sr@latin/LC_MESSAGES/xdg-user-dirs-gtk.mo | grep -A1 "Current folder name"
    msgid "Current folder name"
    msgstr "Trenutno ime"
    mfabian@ari:~
    $ msgunfmt /usr/share/locale/sr/LC_MESSAGES/xdg-user-dirs-gtk.mo | grep -A1 "Current folder name"
    msgunfmt /usr/share/locale/sr/LC_MESSAGES/xdg-user-dirs-gtk.mo | grep -A1 "Current folder name"
    msgid "Current folder name"
    msgstr "Тренутно име"
    mfabian@ari:~
    $ 

Correct.

Check whether the
/usr/share/locale/sr@latin/LC_MESSAGES/xdg-user-dirs-gtk.mo
translations can be retrieved with gettext using the @latin modifier:

    mfabian@ari:~
    $ LC_ALL=sr_RS@latin gettext -d xdg-user-dirs-gtk "Current folder name"
    LC_ALL=sr_RS@latin gettext -d xdg-user-dirs-gtk "Current folder name"
    Trenutno imemfabian@ari:~
    $ LC_ALL=sr_RS.UTF-8@latin gettext -d xdg-user-dirs-gtk "Current folder name"
    LC_ALL=sr_RS.UTF-8@latin gettext -d xdg-user-dirs-gtk "Current folder name"
    Trenutno imemfabian@ari:~
    $ LC_ALL=sr_RS.UTF-8 gettext -d xdg-user-dirs-gtk "Current folder name"
    LC_ALL=sr_RS.UTF-8 gettext -d xdg-user-dirs-gtk "Current folder name"
    Тренутно имеmfabian@ari:~
    $ 
    
Works, with @latin one gets the Serbian translation in latin script,
without @latin the Cyrillic ones.

Now check whether the /usr/share/locale/sr@Latn/LC_MESSAGES/xdg-user-dirs.mo
translations can be retrieved with gettext using the @latin or the @Latn
modifier:

    mfabian@ari:~
    $ LC_ALL=sr_RS.UTF-8@latin gettext -d xdg-user-dirs "Applications"
    LC_ALL=sr_RS.UTF-8@latin gettext -d xdg-user-dirs "Applications"
    Програмиmfabian@ari:~
    $ LC_ALL=sr_RS@latin gettext -d xdg-user-dirs "Applications"
    LC_ALL=sr_RS@latin gettext -d xdg-user-dirs "Applications"
    Програмиmfabian@ari:~
    $ 

So it doesn’t work with "@latin" because the "sr@Latn" directory is
mistakenly used, we get the fallback to Cyrillic instead.
sr_RS@latin and sr_RS.UTF-8@latin are valid locales, so if a suitable
folder containing the @latin cannot be found, gettext falls back to
the "sr" folder which exists and contains Cyrillic.

Now trying with @Latn:

    mfabian@ari:~
    $ LC_ALL=sr_RS.UTF-8@Latn gettext -d xdg-user-dirs "Applications"
    LC_ALL=sr_RS.UTF-8@Latn gettext -d xdg-user-dirs "Applications"
    Applicationsmfabian@ari:~
    $ LC_ALL=sr_RS@Latn gettext -d xdg-user-dirs "Applications"
    LC_ALL=sr_RS@Latn gettext -d xdg-user-dirs "Applications"
    Applicationsmfabian@ari:~
    $ 

We get the fallback to English beause sr_RS.UTF-8@Latn and sr_RS@Latn
are not valid glibc locales. sr_RS.UTF-8@latin and sr_RS@latin
are OK:

    mfabian@ari:~
    $ LC_ALL=sr_RS.UTF-8@latin locale charmap
    LC_ALL=sr_RS.UTF-8@latin locale charmap
    UTF-8
    mfabian@ari:~
    $ LC_ALL=sr_RS@latin locale charmap
    LC_ALL=sr_RS@latin locale charmap
    UTF-8
    mfabian@ari:~

But sr_RS.UTF-8@Latn and sr_RS@Latn are not valid:

    mfabian@ari:~
    $ LC_ALL=sr_RS.UTF-8@Latn locale charmap
    LC_ALL=sr_RS.UTF-8@Latn locale charmap
    locale: Cannot set LC_CTYPE to default locale: No such file or directory
    locale: Cannot set LC_MESSAGES to default locale: No such file or directory
    locale: Cannot set LC_ALL to default locale: No such file or directory
    ANSI_X3.4-1968
    mfabian@ari:~
    $ LC_ALL=sr_RS@Latn locale charmap
    LC_ALL=sr_RS@Latn locale charmap
    locale: Cannot set LC_CTYPE to default locale: No such file or directory
    locale: Cannot set LC_MESSAGES to default locale: No such file or directory
    locale: Cannot set LC_ALL to default locale: No such file or directory
    ANSI_X3.4-1968
    mfabian@ari:~
    $

Comment 9 Mike FABIAN 2014-01-15 12:22:52 UTC
I have now reported the bugs:

bug#1053541 (xdg-user-dirs)
bug#1053543 (dia)
bug#1053545 (glade2)
bug#1053548 (istanbul)
bug#1053550 (kexec-tools)
bug#1053552 (libgda)
bug#1053553 (planner)
bug#1053556 (xmms)

These seem to be all packages in Fedora 20 which have Serbian
translations in Latin script in the wrong subdirectory "sr@Latn".

Comment 10 Fedora End Of Life 2015-05-29 08:48:16 UTC
This message is a reminder that Fedora 20 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 20. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '20'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 20 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 11 Mike FABIAN 2015-05-29 17:32:17 UTC
This problem still exists on the Fedora 22 release.

Comment 12 Mike FABIAN 2015-05-29 17:34:09 UTC
Created attachment 1032216 [details]
user-directories-created-in-cyrillic-fedora-22.png

Comment 13 Satyabrata Maitra 2016-05-30 07:24:03 UTC
It looks Mike is working on this bug already. Once fixed, we will QA this bug.

Thanks Mike!

Comment 14 Fedora End Of Life 2016-07-19 10:05:56 UTC
Fedora 22 changed to end-of-life (EOL) status on 2016-07-19. Fedora 22 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.