Bug 858591 - anaconda setting invalid system locale xx.UTF-8 not xx_YY.UTF-8
anaconda setting invalid system locale xx.UTF-8 not xx_YY.UTF-8
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: anaconda (Show other bugs)
18
Unspecified Unspecified
unspecified Severity urgent
: ---
: ---
Assigned To: Chris Lumens
Fedora Extras Quality Assurance
AcceptedBlocker
: i18n, Regression
: 860276 863510 866108 (view as bug list)
Depends On:
Blocks: F18Beta/F18BetaBlocker 860159 862978
  Show dependency treegraph
 
Reported: 2012-09-19 03:58 EDT by Mike FABIAN
Modified: 2014-09-13 14:56 EDT (History)
19 users (show)

See Also:
Fixed In Version: anaconda-18.13-1
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-10-18 17:17:15 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
first-login-after-default-install-in-japanese.png (508.97 KB, image/png)
2012-09-19 04:00 EDT, Mike FABIAN
no flags Details
anaconda.log (4.25 KB, text/plain)
2012-09-20 05:05 EDT, Mike FABIAN
no flags Details
/var/log/anaconda/syslog (76.69 KB, text/plain)
2012-09-20 05:05 EDT, Mike FABIAN
no flags Details
screenshot showing "cannot change locale" warnings after first logging into a console (14.79 KB, image/png)
2012-10-02 02:12 EDT, Steve Tyler
no flags Details
screenshot showing "Unspecified [ANSI_X3.4-1968]" selected as language in "System Settings:Region & Language" (596.56 KB, image/png)
2012-10-02 10:45 EDT, Steve Tyler
no flags Details
screenshot showing a menu of locales in "System Settings:Region & Language" control center panel (643.19 KB, image/png)
2012-10-02 11:56 EDT, Steve Tyler
no flags Details
anaconda-18-fix-locales.patch (2.63 KB, patch)
2012-10-03 07:50 EDT, Jens Petersen
no flags Details | Diff
anaconda-18-fix-locales.patch (2.74 KB, patch)
2012-10-04 05:31 EDT, Jens Petersen
no flags Details | Diff
anaconda-fix-kk-tg-locales.patch (1.45 KB, patch)
2012-10-05 01:33 EDT, Jens Petersen
no flags Details | Diff
screenshot showing invalid locale eu.UTF-8 for Basque in 18.16 (131.42 KB, image/png)
2012-10-13 14:30 EDT, Steve Tyler
no flags Details
screenshot showing six invalid locales configured by anaconda 18.16 (433.95 KB, image/png)
2012-10-13 15:50 EDT, Steve Tyler
no flags Details
install.py.patch-1 patch to write the locale string to a file before installation (687 bytes, text/plain)
2012-10-13 16:02 EDT, Steve Tyler
no flags Details

  None (edit)
Description Mike FABIAN 2012-09-19 03:58:07 EDT
- default install in Japanese using
  http://dl.fedoraproject.org/pub/alt/stage/18-Alpha-RC3.1/Fedora/x86_64/iso/Fedora-18-Alpha-x86_64-netinst.iso

- Selecting gnome desktop during the installation, everything else
  default

- After the first login into gnome, the desktop is English, not Japanese

- open a gnome-terminal and check the locale, it is POSIX.

- the contents of the file /var/lib/Accounts/Service/users/mfabian are

    [mfabian@localhost ~]$ cat /var/lib/AccountsService/users/mfabian 
    [User]
    Language=
    XSession=
    [mfabian@localhost ~]$ 


- After using “system settings -> region and language” and selecting Japanese there, log out and log in again, the locale is correctly ja_JP.utf8. And the file /var/lib/AccountsService/users/mfabian now contains:

    [mfabian@localhost ~]$ cat /var/lib/AccountsService/users/mfabian 
    [User]
    Language=ja_JP.utf8
    XSession=
    [mfabian@localhost ~]$ 

Expected result:

After an installation done in Japanese, the default language of the
desktop should be Japanese already.
Comment 1 Mike FABIAN 2012-09-19 04:00:17 EDT
Created attachment 614238 [details]
first-login-after-default-install-in-japanese.png
Comment 2 Mike FABIAN 2012-09-19 04:10:25 EDT
The file /etc/sysconfig/i18n does not exist after a default install
of f18 in Japanese.
Comment 3 Mike FABIAN 2012-09-19 04:11:42 EDT
move bug to anaconda because of comment#2.
Comment 4 Mike FABIAN 2012-09-19 04:23:17 EDT
After creating /etc/sysconfig/i18n with the one line content

    LANG=ja_JP.UTF-8

then gnome-session-quit and log in again, the desktop is Japanese.
Comment 5 Chris Lumens 2012-09-19 10:14:11 EDT
Please attach /var/log/anaconda/anaconda.log and /var/log/anaconda/syslog to this bug report.  Thanks.
Comment 6 Mike FABIAN 2012-09-20 05:05:03 EDT
Created attachment 614791 [details]
anaconda.log
Comment 7 Mike FABIAN 2012-09-20 05:05:58 EDT
Created attachment 614792 [details]
/var/log/anaconda/syslog
Comment 8 Chris Lumens 2012-09-20 16:00:29 EDT
I just did a test install of a post-alpha tree and I see that /etc/sysconfig/i18n has:

LANG="ja.UTF-8"

I wonder if that setting works for you.
Comment 9 Mike FABIAN 2012-09-20 20:21:31 EDT
That won’t work, it is an incorrect setting, see:

    mfabian@ari:~
    $ LANG=ja.UTF-8 locale charmap
    locale: Cannot set LC_CTYPE to default locale: No such file or directory
    locale: Cannot set LC_MESSAGES to default locale: No such file or directory
    locale: Cannot set LC_ALL to default locale: No such file or directory
    ANSI_X3.4-1968
    mfabian@ari:~

You see an error message because that locale doesn’t exist on Fedora.

The correct locale is ja_JP.UTF-8, when using this, there is no error message:
    
    mfabian@ari:~
    $ LANG=ja_JP.UTF-8 locale charmap
    UTF-8
    mfabian@ari:~
    $
Comment 10 Akira TAGOH 2012-09-20 22:03:58 EDT
(In reply to comment #8)
> I just did a test install of a post-alpha tree and I see that
> /etc/sysconfig/i18n has:
> 
> LANG="ja.UTF-8"
> 
> I wonder if that setting works for you.

That locale settings perfectly worked until f17. I'm wondering why the kind of this regression happens.
Comment 11 Parag Nemade 2012-09-21 04:22:58 EDT
I tested i18n test day iso and installed in English locale and found that default locale set is en.UTF-8 which should be set as en_US.UTF-8.

Looks like installed locale is missing countrycode?
Comment 12 Mike FABIAN 2012-09-24 01:41:20 EDT
(In reply to comment #11)
> I tested i18n test day iso and installed in English locale and found that
> default locale set is en.UTF-8 which should be set as en_US.UTF-8.
> 
> Looks like installed locale is missing countrycode?

Yes, without the country code it is just an invalid locale:

mfabian@ari:~
$ LC_ALL=en.UTF-8 locale charmap
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
ANSI_X3.4-1968
mfabian@ari:~
$
Comment 13 Jens Petersen 2012-09-25 23:04:18 EDT
(Still happens with anaconda-18.9-1.fc19.)
Comment 14 Adam Williamson 2012-09-26 16:36:38 EDT
Discussed at 2012-09-26 blocker review meeting: http://meetbot.fedoraproject.org/fedora-qa/2012-09-26/f18-beta-blocker-review-1.2012-09-26-16.03.log.txt . No consequences of this that would violate the Beta criteria have been identified, but it clearly violates the Final criterion "All critical path actions on release-blocking desktop environments should correctly display all sufficiently complete translations available for use". It is also accepted as NTH for Beta as it's a highly visible error for non-English users.
Comment 15 Jens Petersen 2012-09-26 22:25:58 EDT
Firstboot crashes because of the wrong locale.

This regression is causing a lot of problems -
probably also the firewall config tool crash?
Comment 16 Jens Petersen 2012-09-26 22:27:09 EDT
Note this bug affects even: en_US -> en.UTF-8.
Comment 17 Jens Petersen 2012-09-27 07:55:09 EDT
Maybe this would help with the en.UTF-8 issue?

diff -u anaconda-18.10/pyanaconda/localization.py\~ anaconda-18.10/pyanaconda/localization.py
--- anaconda-18.10/pyanaconda/localization.py~	2012-09-27 02:26:04.000000000 +0900
+++ anaconda-18.10/pyanaconda/localization.py	2012-09-27 20:50:08.077007658 +0900
@@ -268,8 +268,8 @@
     def __init__(self, preferences={}, territory=None):
         self.translations = {repr(locale):locale for locale in get_available_translations()}
         self.locales = {repr(locale):locale for locale in get_all_locales()}
-        self.preferred_translation = self.translations['en.UTF-8']
-        self.preferred_locales = [self.locales['en.UTF-8']]
+        self.preferred_translation = self.translations['en_US.UTF-8']
+        self.preferred_locales = [self.locales['en_US.UTF-8']]
         self.preferred_locale = self.preferred_locales[0]
 
         self.all_preferences = preferences


So far not sure yet how to fix the general missing territory issue.
Comment 18 Parag Nemade 2012-09-27 09:07:48 EDT
I see localization.py file is supposed to set all the needed things for a language environment. It even sets territory. something must be going wrong in that code that makes it to miss country code.
Comment 19 Jens Petersen 2012-10-01 07:36:12 EDT
I think "data/lang-table" needs to be brought back.
The current language implementation seems to just use the "locales" in po/ which is oversimplified.  GLibc does not support xx.utf8 locales at all.

This regression is making i18n and l10n testing of F18 really difficult.
Comment 20 Adam Williamson 2012-10-01 19:41:06 EDT
jens: if this bug has consequences more severe than we thought when doing the blocker evaluation, do please explain what they are and re-propose as Beta blocker so we can look at it again...
Comment 21 Jens Petersen 2012-10-01 21:51:30 EDT
Or maybe instead of "data/lang-table", anaconda could refer to the list of real
locales listed in "/usr/share/i18n/locales"?


Adam: I am still not sure if it fulfils the Beta blocker criteria.
Having said that if it doesn't, I really feel that the criteria need
to be updated to cover valid locales.  IMHO this should even be
an Alpha blocker (alpha anaconda just hardcoded en_US.UTF-8 so
it did not have this problem fortunately, but both the i18n and l10n
testdays used post-alpha to get gnome-3.5.9x).

FWIW en.UTF-8 doesn't seem to make firstboot crash but ja.UTF-8 does iirc.
Comment 22 Adam Williamson 2012-10-01 22:27:39 EDT
You don't have to phrase it in terms of the criteria, just specify any specific consequences of this bug that are not listed in this report yet.
Comment 23 Akira TAGOH 2012-10-01 22:40:15 EDT
One of example for consequence: Bug#860276

And possibly other applications using scripting language such as python like the above bug. particularly if they deal with localized strings.

the locale support is essential and many many applications are relying on it. the crash and not working properly isn't that surprised things if it's broken.
Comment 24 Steve Tyler 2012-10-02 01:08:03 EDT
firewalld and firewall-config crash:
Bug 860278 - [abrt] firewalld-0.2.7-1.fc18: locale.py:539:setlocale:Error: unsupported locale setting
Comment 25 Steve Tyler 2012-10-02 02:12:05 EDT
Created attachment 620140 [details]
screenshot showing "cannot change locale" warnings after first logging into a console

Another consequence is a flurry of "cannot change locale" warnings after first logging into a console.

Reproduced after a clean install with:
anaconda-18.10-1.fc18.x86_64
Fedora-18-Nightly-20120930.12-x86_64-Live-desktop.iso

Default Language and Keyboard were accepted.
Timezone was set to America/Los_Angeles.
Automatic partitioning was selected.

$ qemu-kvm -m 2048 -hda f18-test-2.img -cdrom ~/xfr/fedora/nightly-composes/Fedora-18-Nightly-20120930.12-x86_64-Live-desktop.iso -vga qxl -boot order=dc,menu=on
Comment 26 Chris Lumens 2012-10-02 09:28:27 EDT
The problem here is we don't know which is the "main" locale for a given language, and python-babel doesn't appear to help us out at all.  For instance, let's look in /usr/share/locale.  Let's say I want to set the language to Spanish.  For that language, which of the seventeen es_* locales should I set as the locale?

This is the kind of problem that lang-table used to solve, but I really don't want to bring it back if we can at all help it.  Not having lang-table means we don't need people to file bugs for new language support.
Comment 27 Steve Tyler 2012-10-02 10:45:37 EDT
Created attachment 620326 [details]
screenshot showing "Unspecified [ANSI_X3.4-1968]" selected as language in "System Settings:Region & Language"

Another consequence is that "Unspecified [ANSI_X3.4-1968]" is displayed as the language in the "System Settings:Region & Language" control center panel.

A partial work-around seems to be to set a specific language from "System Settings:Region & Language" and reboot. Unfortunately, firewalld still crashes and warnings are still displayed when logging into a console.

After selecting "Spanish" from "System Settings:Region & Language" and rebooting, the locale command reports:

$ cat typescript-6
Script iniciado (mar 02 oct 2012 14:26:25 EDT
)[joeblow@localhost xfr]$ locale
LANG=es_ES.utf8
LC_CTYPE="es_ES.utf8"
LC_NUMERIC="es_ES.utf8"
LC_TIME="es_ES.utf8"
LC_COLLATE="es_ES.utf8"
LC_MONETARY="es_ES.utf8"
LC_MESSAGES="es_ES.utf8"
LC_PAPER="es_ES.utf8"
LC_NAME="es_ES.utf8"
LC_ADDRESS="es_ES.utf8"
LC_TELEPHONE="es_ES.utf8"
LC_MEASUREMENT="es_ES.utf8"
LC_IDENTIFICATION="es_ES.utf8"
LC_ALL=
[joeblow@localhost xfr]$ exit

Script terminado (mar 02 oct 2012 14:26:43 EDT
)
Comment 28 Steve Tyler 2012-10-02 11:56:19 EDT
Created attachment 620377 [details]
screenshot showing a menu of locales in "System Settings:Region & Language" control center panel

(In reply to comment #26)
> The problem here is we don't know which is the "main" locale for a given
> language, and python-babel doesn't appear to help us out at all.  For
> instance, let's look in /usr/share/locale.  Let's say I want to set the
> language to Spanish.  For that language, which of the seventeen es_* locales
> should I set as the locale?
> 
> This is the kind of problem that lang-table used to solve, but I really
> don't want to bring it back if we can at all help it.  Not having lang-table
> means we don't need people to file bugs for new language support.

FWIW, in the "System Settings:Region & Language" control center panel, if you click the "+" button, you get a menu of locales (language and region).
Comment 29 Steve Tyler 2012-10-02 14:36:58 EDT
For a work-around, there appear to be three files that must be correct and consistent: [1]

1. /var/lib/AccountsService/users/joeblow
2. /etc/sysconfig/i18n
3. /etc/locale.conf [2]

Work-around procedure:
1. Run "System Settings:Region & Language" to write file #1 above.
2. $ grep Language /var/lib/AccountsService/users/joeblow [3]
3. With a text editor, write the locale from step 2 to file #2 and file #3 above.
4. Reboot.

Test cases:
1. From a gnome terminal:
   $ locale
   There should not be any error messages.
2. Login to a console.
   There should not be any warnings.
3. To verify that firewalld did not crash during booting:
   $ ps -ef | grep firewalld
   Run firewall-config to verify that it does not crash:
   $ firewall-config

[1] Thanks to Mike for identifying the first two files.
[2] /etc/locale.conf does not exist on my F17 system.
[3] For the encoding, the file has "utf8", not "UTF-8".
Comment 30 Adam Williamson 2012-10-02 16:39:10 EDT
steve: I think 3) is a bug that happens only in the case of live installs (or at least only happened in the case of live installs when I first heard about it), and actually is incorrect and can cause problems. I've had an item on my todo list to investigate why it happens *forever*, but never get around to it.
Comment 31 Steve Tyler 2012-10-02 17:14:48 EDT
(In reply to comment #30)
> steve: I think 3) is a bug that happens only in the case of live installs
> (or at least only happened in the case of live installs when I first heard
> about it), and actually is incorrect and can cause problems. I've had an
> item on my todo list to investigate why it happens *forever*, but never get
> around to it.

Thanks for the tip. I did a live install from a nightly in a VM:
anaconda-18.10-1.fc18.x86_64
Fedora-18-Nightly-20120930.12-x86_64-Live-desktop.iso

Which 3? There are four in Comment 29 ... :-)
Comment 32 Adam Williamson 2012-10-02 17:41:47 EDT
I really can't find any decent reference on utf8 vs. UTF-8, it's extremely confusing, but I *think* everything is expected to accept either. 'locale -a' lists the utf8 form, but the UTF-8 form always seems to be recommended when setting LANG or whatever. A bit confusing.
Comment 33 Parag Nemade 2012-10-03 00:05:13 EDT
Because of this bug, iok application got this bug 860159. Consider the case where applications depend on current set locale code. They are all going to get failed.
Comment 34 Parag Nemade 2012-10-03 01:39:22 EDT
Chris,
  I have not looked into source code thoroughly.

  How about changing a anaconda UI a bit by adding available locales once a language is selected and let the user select his own preferred locale?

OR

 Based on timezone and language (hope user will not try to mix this to ask non-available locale), locale can be constructed. Say for Spanish language selected locale is anyway going to start as "es_" then based on timezone, it can be "es_CL" or "es_ES" or any other available locale. If locale is not available then only like current anaconda, locale should have set like LANG=es.utf8

I still not sure how can a locale be selected which have script modifier in it like aa_ER@saaho, de_BE@euro, ks_IN@devanagari etc.
Comment 35 Akira TAGOH 2012-10-03 02:13:35 EDT
(In reply to comment #34)
>  Based on timezone and language (hope user will not try to mix this to ask
> non-available locale), locale can be constructed. Say for Spanish language
> selected locale is anyway going to start as "es_" then based on timezone, it
> can be "es_CL" or "es_ES" or any other available locale. If locale is not
> available then only like current anaconda, locale should have set like
> LANG=es.utf8

In fact there are no locales which don't have a territory. in that case, should fall back to POSIX IMHO.
Comment 36 Steve Tyler 2012-10-03 03:13:20 EDT
The GNU documentation says this:

2.3.1 Locale Names
"A locale name usually has the form ‘ll_CC’. Here ‘ll’ is an ISO 639 two-letter language code, and ‘CC’ is an ISO 3166 two-letter country code."
http://www.gnu.org/software/gettext/manual/html_node/Locale-Names.html#Locale-Names

The 'C' and 'POSIX' locales are exceptions to the 'll_CC' form:

$ locale -a | egrep '^C|POSIX'
C
POSIX

The locale files are text files in /usr/share/i18n/locales/.
Comment 37 Mike FABIAN 2012-10-03 03:32:41 EDT
(In reply to comment #35)
> (In reply to comment #34)
> > If locale is not
> > available then only like current anaconda, locale should have set like
> > LANG=es.utf8
> 
> In fact there are no locales which don't have a territory. in that case,
> should fall back to POSIX IMHO.

I think falling back to en_US.UTF-8 is better in that case because
using POSIX uses ASCII encoding:

mfabian@ari:~
$ LC_ALL=POSIX locale charmap
ANSI_X3.4-1968
mfabian@ari:~
$ LC_ALL=en_US.UTF-8 locale charmap
UTF-8
mfabian@ari:~
$ 

Therefore, using POSIX can cause problems with display of non-ASCII stuff
in some cases:

mfabian@ari:~/tmp/test
$ LC_ALL=en_US.UTF-8 ls
täst  日本語
mfabian@ari:~/tmp/test
$ LC_ALL=POSIX ls
t??st  ?????????
mfabian@ari:~/tmp/test
$ 

and some software then gets ASCII as the default encoding, for example
when reading files.

I think using en_US.UTF-8 is a better fallback.
Comment 38 Mike FABIAN 2012-10-03 03:46:33 EDT
(In reply to comment #32)
> I really can't find any decent reference on utf8 vs. UTF-8, it's extremely
> confusing, but I *think* everything is expected to accept either. 'locale
> -a' lists the utf8 form, but the UTF-8 form always seems to be recommended
> when setting LANG or whatever. A bit confusing.

Yes, "UTF-8" is the preferred spelling but "utf-8" or "utf8" work
as well with glibc.

mfabian@ari:~
$ LC_ALL=en_US.UTF-8 locale charmap
UTF-8
mfabian@ari:~
$ LC_ALL=en_US.UTF8 locale charmap
UTF-8
mfabian@ari:~
$ LC_ALL=en_US.utf-8 locale charmap
UTF-8
mfabian@ari:~
$ LC_ALL=en_US.utf8 locale charmap
UTF-8
mfabian@ari:~
$ LC_ALL=en_US.UtF8 locale charmap
UTF-8
mfabian@ari:~
$ LC_ALL=en_US.UtF---8 locale charmap
UTF-8
mfabian@ari:~

When a spelling is used which does not work with glibc, an error
like this is seen:

mfabian@ari:~
$ LC_ALL=en_US.UTF-9 locale charmap
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
ANSI_X3.4-1968
mfabian@ari:~
$
Comment 39 Mike FABIAN 2012-10-03 03:53:58 EDT
(In reply to comment #34)

> I still not sure how can a locale be selected which have script modifier in
> it like aa_ER@saaho, de_BE@euro, ks_IN@devanagari etc.

aa_ER@saaho and ks_IN@devanagari use UTF-8:

mfabian@ari:~
$ LC_ALL=aa_ER@saaho
mfabian@ari:~
$ LC_ALL=aa_ER@saaho locale charmap
UTF-8
mfabian@ari:~
$ LC_ALL=ks_IN@devanagari locale charmap
UTF-8
mfabian@ari:~

But the xx_YY@euro locales use the legacy ISO-8859-15 encoding:

mfabian@ari:~
$ LC_ALL=de_BE@euro locale charmap
ISO-8859-15
mfabian@ari:~
$ 

So it is not a good idea to use xx_YY@euro locales, these
are just legacy locales and should not be set as defaults,
they are useful only for special purposes.
Comment 40 Jens Petersen 2012-10-03 05:22:46 EDT
I rather like the idea of using timezone selection to also set the locale
but I dunno if that data is available?  Does pytz have locale info?

(Right, utf8 vs UTF-8 is not an issue here - either is fine.)

If there are no other possible ideas then probably
reverting back to using lang-table for now is probably
the simplest viable solution at this point?
Comment 41 Jens Petersen 2012-10-03 05:26:00 EDT
As for F18Beta - no further new info to add.
Presumably firstboot crashing for a Japanese, etc installs
is not a blocker until after Beta?

Anyway I really really hope this can be fixed by Beta
otherwise it is not going to be good.
Comment 42 Jens Petersen 2012-10-03 05:57:25 EDT
(In reply to comment #40)
> but I dunno if that data is available?  Does pytz have locale info?

I don't think tzdata has any locale info so it will not work
and anyway as you know many countries have multiple locales.


One other hack idea: how about renaming the po files to (non-standard) local
based naming:

en.po -> en_US.po
ja.po -> ja_JP.po
:
es.po -> es_ES.po

(all the correct locales should be listed in the old lang-table file.)

That should at least make the current implementation work like
the old lang-table approach giving valid locales.  Of course
l10n people might not like this hack.  Anyway just offering it as
an possible alternative to reverting to lang-table, which some might
still consider the safest.
Comment 43 Akira TAGOH 2012-10-03 06:08:39 EDT
(In reply to comment #42)
> One other hack idea: how about renaming the po files to (non-standard) local
> based naming:
> 
> en.po -> en_US.po
> ja.po -> ja_JP.po
> :
> es.po -> es_ES.po
> 
> (all the correct locales should be listed in the old lang-table file.)
> 
> That should at least make the current implementation work like
> the old lang-table approach giving valid locales.  Of course
> l10n people might not like this hack.

That's true. this may makes some duplicate work and files.
Comment 44 Jens Petersen 2012-10-03 07:46:42 EDT
BTW how are keyboard layouts being defaulted now, or is it always US layout?

Overall lang-table still seems a nice compact way to store all this info,
though if there was some upstream project that could provide all this
data with a nice API that would be ideal surely.

Anyway here is a scratch build
http://koji.fedoraproject.org/koji/taskinfo?taskID=4554802
implementing the hack in comment 42, which seems to work for me.
After installing in Japanese with this build, firstboot now starts,
system locale is correct and locales errors gone. :)
Comment 45 Jens Petersen 2012-10-03 07:50:30 EDT
Created attachment 620776 [details]
anaconda-18-fix-locales.patch

Here is the patch to anaconda.spec used for the above test build.

It is not beautiful but at least it works. ;)

I can convert it into a shell script that can be run from %install, if it helps.

Though thinking of kbd default handling also - maybe still better to return to lang-table?
Comment 46 Steve Tyler 2012-10-03 10:15:06 EDT
According to the POSIX specification[1], the default locale can be "the POSIX locale or any other implementation-defined locale."

7.2 POSIX Locale
http://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap07.html#tag_07_02

[1] The Open Group Base Specifications Issue 6
IEEE Std 1003.1, 2004 Edition
http://pubs.opengroup.org/onlinepubs/009695399/
Comment 47 Chris Lumens 2012-10-03 10:26:52 EDT
I'm going to regret this, but I'll take this bug.
Comment 48 Adam Williamson 2012-10-03 13:05:33 EDT
re-proposing as Beta blocker. For the record, as status of this bug is complex, it's currently accepted as Final blocker and Beta NTH, but rejected as Beta blocker: I am re-proposing it as Beta blocker on the basis that it causes more effects than we were aware of when we first rejected it. Status will be updated after discussion at the blocker meeting today.
Comment 49 Adam Williamson 2012-10-03 13:25:55 EDT
https://bugzilla.redhat.com/show_bug.cgi?id=859632 may possibly be another consequence of this (still being checked).
Comment 50 Adam Williamson 2012-10-03 14:00:39 EDT
*** Bug 860276 has been marked as a duplicate of this bug. ***
Comment 51 Adam Williamson 2012-10-03 14:02:33 EDT
So #860276 gives us a very solid basis to accept this as a blocker: it breaks firstboot. I confirmed this myself. 'LC_ALL=es.UTF-8 firstboot' crashes, firstboot does not run at all. 'LC_ALL=es_ES.UTF-8 firstboot' runs fine. Other languages are affected, at least Japanese. So this bug causes firstboot to fail to run in several languages, breaking the Alpha criterion "In most cases (see Blocker_Bug_FAQ), a system installed according to any of the above criteria (or the appropriate Beta or Final criteria, when applying this criterion to those releases) must boot to the 'firstboot' utility on the first boot after installation, without unintended user intervention, unless the user explicitly chooses to boot in non-graphical mode. This includes correctly accessing any encrypted partitions when the correct passphrase is supplied. The firstboot utility must be able to create a working user account".
Comment 52 Jens Petersen 2012-10-04 05:31:36 EDT
Created attachment 621475 [details]
anaconda-18-fix-locales.patch

Ok I cleaned up my patch a little more and made a shell script for the locale file renaming.  This now takes care of renaming all the translation files (even 2 not yet supported by glibc;).

However English (en) seems to be a special case in anaconda
so en.UTF-8 not fixed yet.  That can probably be fixed on the python side.

But lang-table is probably still a cleaner solution.

Those wanting to install with a valid locale can try installing
http://koji.fedoraproject.org/koji/taskinfo?taskID=4558387
into a Live instance.
Comment 53 Chris Lumens 2012-10-04 10:00:24 EDT
Jens - thanks, but I've got something in anaconda itself for now.  We'll definitely be looking for a better solution in the future, though.  We really don't want to own this kind of data.
Comment 54 Steve Tyler 2012-10-04 10:43:47 EDT
Add in a locale mapping to avoid incorrect system settings (#858591).
http://git.fedorahosted.org/cgit/anaconda.git/commit/?id=45c8c0a28ccca8ac02b89571b4fcb192c33c2103

Thanks.

Does this mean that selecting a language implicitly sets a corresponding country in the resulting locale?

"en": "en_US"
"es": "es_ES"
"ja": "ja_JP"
Comment 55 Parag Nemade 2012-10-04 10:57:53 EDT
Some languages are still missing and these are important languages.

Bengali(India) bn_IN
Chinese(Simplified)     zh_CN
Chinese(Traditional)    zh_TW  
Portuguese(Brazilian)   pt_BR 
Serbian(Latin)  sr@latin
Comment 56 Chris Lumens 2012-10-04 11:06:54 EDT
Those don't need to be mangled, they're already good as-is.
Comment 57 Steve Tyler 2012-10-04 11:14:36 EDT
I ran a validation test of the locales against /usr/share/i18n/locales/ in F17:

ls: cannot access /usr/share/i18n/locales/ilo_PH: No such file or directory
ls: cannot access /usr/share/i18n/locales/kk_KK: No such file or directory
ls: cannot access /usr/share/i18n/locales/tg_TG: No such file or directory

$ cat locale-list-1.txt | sed -e 's@^@/usr/share/i18n/locales/@' | xargs ls

Locales were extracted with:

#!/bin/bash

# Extract locales from language-list-1.txt
# The list is from:
# http://git.fedorahosted.org/cgit/anaconda.git/plain/pyanaconda/localization.py

sed -e 's/[:,]/\n/g' -e 's/[" }]//g' language-list-1.txt | grep '_'
Comment 58 Parag Nemade 2012-10-04 11:23:08 EDT
Right.
kk_KK should be kk_KZ
tg_TG should be tg_TJ

I wonder where ilo locale gone?
Comment 59 Chris Lumens 2012-10-04 11:27:15 EDT
Guys - if you want to be involved in development, do it on the anaconda-devel and anaconda-patches mailing list, not in bugzilla.
Comment 60 Steve Tyler 2012-10-04 11:31:17 EDT
(In reply to comment #59)
> Guys - if you want to be involved in development, do it on the
> anaconda-devel and anaconda-patches mailing list, not in bugzilla.

On F17:
$ ls /usr/share/i18n/locales/*PH
/usr/share/i18n/locales/en_PH  /usr/share/i18n/locales/fil_PH  /usr/share/i18n/locales/tl_PH

Consider this a bug report, then ... :-)

This bug has caused a lot of problems and a lot of work, so any fixes deserve a lot of scrutiny.
Comment 62 Jens Petersen 2012-10-05 01:29:22 EDT
(In reply to comment #53)
> Jens - thanks, but I've got something in anaconda itself for now.  We'll
> definitely be looking for a better solution in the future, though.  We
> really don't want to own this kind of data.

Thanks, glad to hear it, and good to know.

(In reply to comment #54)
> Add in a locale mapping to avoid incorrect system settings (#858591).
> http://git.fedorahosted.org/cgit/anaconda.git/commit/
> ?id=45c8c0a28ccca8ac02b89571b4fcb192c33c2103

Thanks for the pointer.  I should have looked yesterday...

http://koji.fedoraproject.org/koji/taskinfo?taskID=4561583
is a test build of current anaconda HEAD.

I tested it from Live and the patch looks good so far.
So, great news that with 18.13 we should have working locales again.

(In reply to comment #58)
> I wonder where ilo locale gone?

It is not in glibc.
Comment 63 Jens Petersen 2012-10-05 01:33:55 EDT
Created attachment 621937 [details]
anaconda-fix-kk-tg-locales.patch

> kk_KK should be kk_KZ
> tg_TG should be tg_TJ

Right - I had already noticed this while working on my hack.

Chris: can you please apply this patch to correct those two locales.
Comment 64 Jens Petersen 2012-10-05 04:55:33 EDT
(In reply to comment #62)
> http://koji.fedoraproject.org/koji/taskinfo?taskID=4561583
> is a test build of current anaconda HEAD.

Here is an corresponding updates.img file too (untested)
if anyone wants to test with a net install or dvd.

http://petersen.fedorapeople.org/anaconda/updates-858591.img
Comment 65 Steve Tyler 2012-10-05 09:50:47 EDT
(In reply to comment #58)
...
> I wonder where ilo locale gone?

Good question. ilo_PH was invalid in F17 too:
Bug 863450 - invalid locale ilo_PH after installing in Iloko language from F17 DVD
Comment 66 Chris Lumens 2012-10-05 10:36:29 EDT
>Chris: can you please apply this patch to correct those two locales.

Done.
Comment 67 Chris Lumens 2012-10-05 13:29:13 EDT
*** Bug 863510 has been marked as a duplicate of this bug. ***
Comment 68 Fedora Update System 2012-10-08 20:22:21 EDT
anaconda-18.14-1.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/anaconda-18.14-1.fc18
Comment 69 Fedora Update System 2012-10-09 13:22:14 EDT
Package anaconda-18.14-1.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing anaconda-18.14-1.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-15707/anaconda-18.14-1.fc18
then log in and leave karma (feedback).
Comment 70 Adam Williamson 2012-10-09 23:37:10 EDT
F18 Beta TC3 - https://dl.fedoraproject.org/pub/alt/stage/18-Beta-TC3/ - should include this fix. Please re-test and check. Thanks!
Comment 71 Adam Williamson 2012-10-10 00:07:45 EDT
I checked this myself, and it looks fine. Did an install in French (with UK English keyboard layout, just for fun) - the installed system shows up in French and firstboot ran fine. Setting VERIFIED.
Comment 72 Steve Tyler 2012-10-10 00:22:43 EDT
I verified that installing in Spanish results in a valid locale of es_ES.UTF-8 and that the desktop displays Spanish. Some messages in terminals are in Spanish and some are not. Compare "ls /tmp/foo" and "sudo foo". 

It could take a while to test all of the available languages by installing ...

Tested with:
$ qemu-kvm -m 2048 -hda f18-test-1.img -cdrom ~/xfr/fedora/F18/F18-Beta/TC3/Fedora-18-Beta-TC3-x86_64-Live-Desktop.iso -usb -vga qxl -boot menu=on
Comment 73 Parag Nemade 2012-10-10 00:45:07 EDT
I can confirm here that installation in Marathi language with US keyboard layout, set the correct locale to mr_IN.UTF-8
Comment 74 Dan Mashal 2012-10-10 01:41:12 EDT
 work around for me was 'export LC_ALL="en_US"'
Comment 75 Jan Sedlák 2012-10-10 09:54:05 EDT
It seems OK, localization is setted correctly.
Comment 76 Adam Williamson 2012-10-10 13:36:47 EDT
Discussed at 2012-10-10 blocker review meeting: http://meetbot.fedoraproject.org/fedora-qa/2012-10-10/f18beta-blocker-review-3.2012-10-10-16.05.log.txt . Accepted as a blocker per Alpha criterion "In most cases (see Blocker_Bug_FAQ), a system installed according to any of the above criteria (or the appropriate Beta or Final criteria, when applying this criterion to those releases) must boot to the 'firstboot' utility on the first boot after installation, without unintended user intervention, unless the user explicitly chooses to boot in non-graphical mode. This includes correctly accessing any encrypted partitions when the correct passphrase is supplied. The firstboot utility must be able to create a working user account", in the case of just about any non-English install firstboot fails to run.
Comment 77 Steve Tyler 2012-10-13 14:30:10 EDT
Created attachment 626612 [details]
screenshot showing invalid locale eu.UTF-8 for Basque in 18.16

anaconda 18.16 is writing an invalid locale eu.UTF-8 for Basque.

This is the first result for a semi-automated locale validation test. It requires patching anaconda, but hopefully in a non-destructive way. When anaconda writes /mnt/sysimage/etc/sysconfig/i18n and /mnt/sysimage/etc/locale.conf, the patch appends the same strings to files in /tmp: langlist_i18n_1.txt and langlist_locale_conf_1.txt. (The strings appear to be identical, so having two is redundant.)

The main problem with this procedure is that the strings are written at the very end of the install, so a test for each language requires configuring the disk and waiting for the install to complete.
Comment 78 Steve Tyler 2012-10-13 15:50:06 EDT
Created attachment 626660 [details]
screenshot showing six invalid locales configured by anaconda 18.16

The language menu items that do not show a country all produce invalid locales:
be
bs
eu
hy
ka
sr
Comment 79 Steve Tyler 2012-10-13 16:02:53 EDT
Created attachment 626661 [details]
install.py.patch-1 patch to write the locale string to a file before installation

Writing out the locale string before installation speeded up checking the language menu items greatly. For each language tested, after confirming that /tmp/langlist_ksdata_1.txt had been updated, it was safe to kill anaconda and start a new test of the next language. I was lazy and only checked the languages that did not have a corresponding country in the language menu. :-)

$ diff -u install.py.ORIG-18.16 install.py.NEW-1 > install.py.patch-1
Comment 80 Steve Tyler 2012-10-13 22:38:25 EDT
*** Bug 866108 has been marked as a duplicate of this bug. ***
Comment 81 Steve Tyler 2012-10-14 01:24:02 EDT
Georgian gives an invalid locale.

Package: anaconda-18.16-1.fc18.x86_64
OS Release: Fedora release 18
Comment 82 Steve Tyler 2012-10-14 11:37:19 EDT
Test results for Fedora-18-Beta-TC4-x86_64-DVD.iso (which has anaconda 18.16):

Selecting these languages results in an invalid locale:

Belarusian      be
Bosnian         bs
Basque          eu
Armenian        hy
Georgian        ka
Serbian         sr

These alternative languages from the language menu result in a valid locale:
Basque (Spain)      eu_ES
Serbian (Serbia)    sr_RS

There are no alternatives for Belarusian, Bosnian, Armenian, Georgian.

There are 74 languages in the language menu. I did not test every one ...

Tested by repeatedly running clean, minimal installs from the DVD.

Command-line:
$ qemu-kvm -m 2048 -hda f18-test-1.img -cdrom ~/xfr/fedora/F18/F18-Beta/TC4/Fedora-18-Beta-TC4-x86_64-DVD.iso -usb -vga qxl -boot menu=on -usbdevice mouse
Comment 83 Adam Williamson 2012-10-15 19:03:28 EDT
Steve: can you please open a new bug for the cases which still give invalid locales? The main bug here - that anaconda was just completely broken regarding locale setting - is clearly addressed, and overloading the report will only lead to confusion. Since what you're seeing now is effectively a new issue - anaconda now does actually try to do locale setting properly, but in a few cases gets it wrong - filing it as a new bug is the appropriate approach.
Comment 84 Steve Tyler 2012-10-15 22:14:18 EDT
Bug 866730 - invalid locales configured for some languages
Comment 85 Adam Williamson 2012-10-18 17:17:15 EDT
This was fixed in 18.13 and 18.14 went stable, so closing.

Note You need to log in before you can comment on or make changes to this bug.