Bug 858591

Created attachment 614238 [details]
first-login-after-default-install-in-japanese.png

The file /etc/sysconfig/i18n does not exist after a default install
of f18 in Japanese.

move bug to anaconda because of comment#2.

After creating /etc/sysconfig/i18n with the one line content

    LANG=ja_JP.UTF-8

then gnome-session-quit and log in again, the desktop is Japanese.

Please attach /var/log/anaconda/anaconda.log and /var/log/anaconda/syslog to this bug report.  Thanks.

Created attachment 614791 [details]
anaconda.log

Created attachment 614792 [details]
/var/log/anaconda/syslog

I just did a test install of a post-alpha tree and I see that /etc/sysconfig/i18n has:

LANG="ja.UTF-8"

I wonder if that setting works for you.

That won’t work, it is an incorrect setting, see:

    mfabian@ari:~
    $ LANG=ja.UTF-8 locale charmap
    locale: Cannot set LC_CTYPE to default locale: No such file or directory
    locale: Cannot set LC_MESSAGES to default locale: No such file or directory
    locale: Cannot set LC_ALL to default locale: No such file or directory
    ANSI_X3.4-1968
    mfabian@ari:~

You see an error message because that locale doesn’t exist on Fedora.

The correct locale is ja_JP.UTF-8, when using this, there is no error message:
    
    mfabian@ari:~
    $ LANG=ja_JP.UTF-8 locale charmap
    UTF-8
    mfabian@ari:~
    $

(In reply to comment #8)
> I just did a test install of a post-alpha tree and I see that
> /etc/sysconfig/i18n has:
> 
> LANG="ja.UTF-8"
> 
> I wonder if that setting works for you.

That locale settings perfectly worked until f17. I'm wondering why the kind of this regression happens.

I tested i18n test day iso and installed in English locale and found that default locale set is en.UTF-8 which should be set as en_US.UTF-8.

Looks like installed locale is missing countrycode?

(In reply to comment #11)
> I tested i18n test day iso and installed in English locale and found that
> default locale set is en.UTF-8 which should be set as en_US.UTF-8.
> 
> Looks like installed locale is missing countrycode?

Yes, without the country code it is just an invalid locale:

mfabian@ari:~
$ LC_ALL=en.UTF-8 locale charmap
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
ANSI_X3.4-1968
mfabian@ari:~
$

(Still happens with anaconda-18.9-1.fc19.)

Discussed at 2012-09-26 blocker review meeting: http://meetbot.fedoraproject.org/fedora-qa/2012-09-26/f18-beta-blocker-review-1.2012-09-26-16.03.log.txt . No consequences of this that would violate the Beta criteria have been identified, but it clearly violates the Final criterion "All critical path actions on release-blocking desktop environments should correctly display all sufficiently complete translations available for use". It is also accepted as NTH for Beta as it's a highly visible error for non-English users.

Firstboot crashes because of the wrong locale.

This regression is causing a lot of problems -
probably also the firewall config tool crash?

Note this bug affects even: en_US -> en.UTF-8.

Maybe this would help with the en.UTF-8 issue?

diff -u anaconda-18.10/pyanaconda/localization.py\~ anaconda-18.10/pyanaconda/localization.py
--- anaconda-18.10/pyanaconda/localization.py~	2012-09-27 02:26:04.000000000 +0900
+++ anaconda-18.10/pyanaconda/localization.py	2012-09-27 20:50:08.077007658 +0900
@@ -268,8 +268,8 @@
     def __init__(self, preferences={}, territory=None):
         self.translations = {repr(locale):locale for locale in get_available_translations()}
         self.locales = {repr(locale):locale for locale in get_all_locales()}
-        self.preferred_translation = self.translations['en.UTF-8']
-        self.preferred_locales = [self.locales['en.UTF-8']]
+        self.preferred_translation = self.translations['en_US.UTF-8']
+        self.preferred_locales = [self.locales['en_US.UTF-8']]
         self.preferred_locale = self.preferred_locales[0]
 
         self.all_preferences = preferences


So far not sure yet how to fix the general missing territory issue.

I see localization.py file is supposed to set all the needed things for a language environment. It even sets territory. something must be going wrong in that code that makes it to miss country code.

I think "data/lang-table" needs to be brought back.
The current language implementation seems to just use the "locales" in po/ which is oversimplified.  GLibc does not support xx.utf8 locales at all.

This regression is making i18n and l10n testing of F18 really difficult.

jens: if this bug has consequences more severe than we thought when doing the blocker evaluation, do please explain what they are and re-propose as Beta blocker so we can look at it again...

Or maybe instead of "data/lang-table", anaconda could refer to the list of real
locales listed in "/usr/share/i18n/locales"?


Adam: I am still not sure if it fulfils the Beta blocker criteria.
Having said that if it doesn't, I really feel that the criteria need
to be updated to cover valid locales.  IMHO this should even be
an Alpha blocker (alpha anaconda just hardcoded en_US.UTF-8 so
it did not have this problem fortunately, but both the i18n and l10n
testdays used post-alpha to get gnome-3.5.9x).

FWIW en.UTF-8 doesn't seem to make firstboot crash but ja.UTF-8 does iirc.

You don't have to phrase it in terms of the criteria, just specify any specific consequences of this bug that are not listed in this report yet.

One of example for consequence: Bug#860276

And possibly other applications using scripting language such as python like the above bug. particularly if they deal with localized strings.

the locale support is essential and many many applications are relying on it. the crash and not working properly isn't that surprised things if it's broken.

firewalld and firewall-config crash:
Bug 860278 - [abrt] firewalld-0.2.7-1.fc18: locale.py:539:setlocale:Error: unsupported locale setting

Created attachment 620140 [details]
screenshot showing "cannot change locale" warnings after first logging into a console

Another consequence is a flurry of "cannot change locale" warnings after first logging into a console.

Reproduced after a clean install with:
anaconda-18.10-1.fc18.x86_64
Fedora-18-Nightly-20120930.12-x86_64-Live-desktop.iso

Default Language and Keyboard were accepted.
Timezone was set to America/Los_Angeles.
Automatic partitioning was selected.

$ qemu-kvm -m 2048 -hda f18-test-2.img -cdrom ~/xfr/fedora/nightly-composes/Fedora-18-Nightly-20120930.12-x86_64-Live-desktop.iso -vga qxl -boot order=dc,menu=on

The problem here is we don't know which is the "main" locale for a given language, and python-babel doesn't appear to help us out at all.  For instance, let's look in /usr/share/locale.  Let's say I want to set the language to Spanish.  For that language, which of the seventeen es_* locales should I set as the locale?

This is the kind of problem that lang-table used to solve, but I really don't want to bring it back if we can at all help it.  Not having lang-table means we don't need people to file bugs for new language support.

Created attachment 620326 [details]
screenshot showing "Unspecified [ANSI_X3.4-1968]" selected as language in "System Settings:Region & Language"

Another consequence is that "Unspecified [ANSI_X3.4-1968]" is displayed as the language in the "System Settings:Region & Language" control center panel.

A partial work-around seems to be to set a specific language from "System Settings:Region & Language" and reboot. Unfortunately, firewalld still crashes and warnings are still displayed when logging into a console.

After selecting "Spanish" from "System Settings:Region & Language" and rebooting, the locale command reports:

$ cat typescript-6
Script iniciado (mar 02 oct 2012 14:26:25 EDT
)[joeblow@localhost xfr]$ locale
LANG=es_ES.utf8
LC_CTYPE="es_ES.utf8"
LC_NUMERIC="es_ES.utf8"
LC_TIME="es_ES.utf8"
LC_COLLATE="es_ES.utf8"
LC_MONETARY="es_ES.utf8"
LC_MESSAGES="es_ES.utf8"
LC_PAPER="es_ES.utf8"
LC_NAME="es_ES.utf8"
LC_ADDRESS="es_ES.utf8"
LC_TELEPHONE="es_ES.utf8"
LC_MEASUREMENT="es_ES.utf8"
LC_IDENTIFICATION="es_ES.utf8"
LC_ALL=
[joeblow@localhost xfr]$ exit

Script terminado (mar 02 oct 2012 14:26:43 EDT
)

Created attachment 620377 [details]
screenshot showing a menu of locales in "System Settings:Region & Language" control center panel

(In reply to comment #26)
> The problem here is we don't know which is the "main" locale for a given
> language, and python-babel doesn't appear to help us out at all.  For
> instance, let's look in /usr/share/locale.  Let's say I want to set the
> language to Spanish.  For that language, which of the seventeen es_* locales
> should I set as the locale?
> 
> This is the kind of problem that lang-table used to solve, but I really
> don't want to bring it back if we can at all help it.  Not having lang-table
> means we don't need people to file bugs for new language support.

FWIW, in the "System Settings:Region & Language" control center panel, if you click the "+" button, you get a menu of locales (language and region).

For a work-around, there appear to be three files that must be correct and consistent: [1]

1. /var/lib/AccountsService/users/joeblow
2. /etc/sysconfig/i18n
3. /etc/locale.conf [2]

Work-around procedure:
1. Run "System Settings:Region & Language" to write file #1 above.
2. $ grep Language /var/lib/AccountsService/users/joeblow [3]
3. With a text editor, write the locale from step 2 to file #2 and file #3 above.
4. Reboot.

Test cases:
1. From a gnome terminal:
   $ locale
   There should not be any error messages.
2. Login to a console.
   There should not be any warnings.
3. To verify that firewalld did not crash during booting:
   $ ps -ef | grep firewalld
   Run firewall-config to verify that it does not crash:
   $ firewall-config

[1] Thanks to Mike for identifying the first two files.
[2] /etc/locale.conf does not exist on my F17 system.
[3] For the encoding, the file has "utf8", not "UTF-8".

steve: I think 3) is a bug that happens only in the case of live installs (or at least only happened in the case of live installs when I first heard about it), and actually is incorrect and can cause problems. I've had an item on my todo list to investigate why it happens *forever*, but never get around to it.

(In reply to comment #30)
> steve: I think 3) is a bug that happens only in the case of live installs
> (or at least only happened in the case of live installs when I first heard
> about it), and actually is incorrect and can cause problems. I've had an
> item on my todo list to investigate why it happens *forever*, but never get
> around to it.

Thanks for the tip. I did a live install from a nightly in a VM:
anaconda-18.10-1.fc18.x86_64
Fedora-18-Nightly-20120930.12-x86_64-Live-desktop.iso

Which 3? There are four in Comment 29 ... :-)

I really can't find any decent reference on utf8 vs. UTF-8, it's extremely confusing, but I *think* everything is expected to accept either. 'locale -a' lists the utf8 form, but the UTF-8 form always seems to be recommended when setting LANG or whatever. A bit confusing.

Because of this bug, iok application got this bug 860159. Consider the case where applications depend on current set locale code. They are all going to get failed.

Chris,
  I have not looked into source code thoroughly.

  How about changing a anaconda UI a bit by adding available locales once a language is selected and let the user select his own preferred locale?

OR

 Based on timezone and language (hope user will not try to mix this to ask non-available locale), locale can be constructed. Say for Spanish language selected locale is anyway going to start as "es_" then based on timezone, it can be "es_CL" or "es_ES" or any other available locale. If locale is not available then only like current anaconda, locale should have set like LANG=es.utf8

I still not sure how can a locale be selected which have script modifier in it like aa_ER@saaho, de_BE@euro, ks_IN@devanagari etc.

(In reply to comment #34)
>  Based on timezone and language (hope user will not try to mix this to ask
> non-available locale), locale can be constructed. Say for Spanish language
> selected locale is anyway going to start as "es_" then based on timezone, it
> can be "es_CL" or "es_ES" or any other available locale. If locale is not
> available then only like current anaconda, locale should have set like
> LANG=es.utf8

In fact there are no locales which don't have a territory. in that case, should fall back to POSIX IMHO.

The GNU documentation says this:

2.3.1 Locale Names
"A locale name usually has the form ‘ll_CC’. Here ‘ll’ is an ISO 639 two-letter language code, and ‘CC’ is an ISO 3166 two-letter country code."
http://www.gnu.org/software/gettext/manual/html_node/Locale-Names.html#Locale-Names

The 'C' and 'POSIX' locales are exceptions to the 'll_CC' form:

$ locale -a | egrep '^C|POSIX'
C
POSIX

The locale files are text files in /usr/share/i18n/locales/.

(In reply to comment #35)
> (In reply to comment #34)
> > If locale is not
> > available then only like current anaconda, locale should have set like
> > LANG=es.utf8
> 
> In fact there are no locales which don't have a territory. in that case,
> should fall back to POSIX IMHO.

I think falling back to en_US.UTF-8 is better in that case because
using POSIX uses ASCII encoding:

mfabian@ari:~
$ LC_ALL=POSIX locale charmap
ANSI_X3.4-1968
mfabian@ari:~
$ LC_ALL=en_US.UTF-8 locale charmap
UTF-8
mfabian@ari:~
$ 

Therefore, using POSIX can cause problems with display of non-ASCII stuff
in some cases:

mfabian@ari:~/tmp/test
$ LC_ALL=en_US.UTF-8 ls
täst  日本語
mfabian@ari:~/tmp/test
$ LC_ALL=POSIX ls
t??st  ?????????
mfabian@ari:~/tmp/test
$ 

and some software then gets ASCII as the default encoding, for example
when reading files.

I think using en_US.UTF-8 is a better fallback.

(In reply to comment #32)
> I really can't find any decent reference on utf8 vs. UTF-8, it's extremely
> confusing, but I *think* everything is expected to accept either. 'locale
> -a' lists the utf8 form, but the UTF-8 form always seems to be recommended
> when setting LANG or whatever. A bit confusing.

Yes, "UTF-8" is the preferred spelling but "utf-8" or "utf8" work
as well with glibc.

mfabian@ari:~
$ LC_ALL=en_US.UTF-8 locale charmap
UTF-8
mfabian@ari:~
$ LC_ALL=en_US.UTF8 locale charmap
UTF-8
mfabian@ari:~
$ LC_ALL=en_US.utf-8 locale charmap
UTF-8
mfabian@ari:~
$ LC_ALL=en_US.utf8 locale charmap
UTF-8
mfabian@ari:~
$ LC_ALL=en_US.UtF8 locale charmap
UTF-8
mfabian@ari:~
$ LC_ALL=en_US.UtF---8 locale charmap
UTF-8
mfabian@ari:~

When a spelling is used which does not work with glibc, an error
like this is seen:

mfabian@ari:~
$ LC_ALL=en_US.UTF-9 locale charmap
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
ANSI_X3.4-1968
mfabian@ari:~
$

(In reply to comment #34)

> I still not sure how can a locale be selected which have script modifier in
> it like aa_ER@saaho, de_BE@euro, ks_IN@devanagari etc.

aa_ER@saaho and ks_IN@devanagari use UTF-8:

mfabian@ari:~
$ LC_ALL=aa_ER@saaho
mfabian@ari:~
$ LC_ALL=aa_ER@saaho locale charmap
UTF-8
mfabian@ari:~
$ LC_ALL=ks_IN@devanagari locale charmap
UTF-8
mfabian@ari:~

But the xx_YY@euro locales use the legacy ISO-8859-15 encoding:

mfabian@ari:~
$ LC_ALL=de_BE@euro locale charmap
ISO-8859-15
mfabian@ari:~
$ 

So it is not a good idea to use xx_YY@euro locales, these
are just legacy locales and should not be set as defaults,
they are useful only for special purposes.

I rather like the idea of using timezone selection to also set the locale
but I dunno if that data is available?  Does pytz have locale info?

(Right, utf8 vs UTF-8 is not an issue here - either is fine.)

If there are no other possible ideas then probably
reverting back to using lang-table for now is probably
the simplest viable solution at this point?

As for F18Beta - no further new info to add.
Presumably firstboot crashing for a Japanese, etc installs
is not a blocker until after Beta?

Anyway I really really hope this can be fixed by Beta
otherwise it is not going to be good.

(In reply to comment #40)
> but I dunno if that data is available?  Does pytz have locale info?

I don't think tzdata has any locale info so it will not work
and anyway as you know many countries have multiple locales.


One other hack idea: how about renaming the po files to (non-standard) local
based naming:

en.po -> en_US.po
ja.po -> ja_JP.po
:
es.po -> es_ES.po

(all the correct locales should be listed in the old lang-table file.)

That should at least make the current implementation work like
the old lang-table approach giving valid locales.  Of course
l10n people might not like this hack.  Anyway just offering it as
an possible alternative to reverting to lang-table, which some might
still consider the safest.

(In reply to comment #42)
> One other hack idea: how about renaming the po files to (non-standard) local
> based naming:
> 
> en.po -> en_US.po
> ja.po -> ja_JP.po
> :
> es.po -> es_ES.po
> 
> (all the correct locales should be listed in the old lang-table file.)
> 
> That should at least make the current implementation work like
> the old lang-table approach giving valid locales.  Of course
> l10n people might not like this hack.

That's true. this may makes some duplicate work and files.

BTW how are keyboard layouts being defaulted now, or is it always US layout?

Overall lang-table still seems a nice compact way to store all this info,
though if there was some upstream project that could provide all this
data with a nice API that would be ideal surely.

Anyway here is a scratch build
http://koji.fedoraproject.org/koji/taskinfo?taskID=4554802
implementing the hack in comment 42, which seems to work for me.
After installing in Japanese with this build, firstboot now starts,
system locale is correct and locales errors gone. :)

Created attachment 620776 [details]
anaconda-18-fix-locales.patch

Here is the patch to anaconda.spec used for the above test build.

It is not beautiful but at least it works. ;)

I can convert it into a shell script that can be run from %install, if it helps.

Though thinking of kbd default handling also - maybe still better to return to lang-table?

According to the POSIX specification[1], the default locale can be "the POSIX locale or any other implementation-defined locale."

7.2 POSIX Locale
http://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap07.html#tag_07_02

[1] The Open Group Base Specifications Issue 6
IEEE Std 1003.1, 2004 Edition
http://pubs.opengroup.org/onlinepubs/009695399/

I'm going to regret this, but I'll take this bug.

re-proposing as Beta blocker. For the record, as status of this bug is complex, it's currently accepted as Final blocker and Beta NTH, but rejected as Beta blocker: I am re-proposing it as Beta blocker on the basis that it causes more effects than we were aware of when we first rejected it. Status will be updated after discussion at the blocker meeting today.

https://bugzilla.redhat.com/show_bug.cgi?id=859632 may possibly be another consequence of this (still being checked).

*** Bug 860276 has been marked as a duplicate of this bug. ***

So #860276 gives us a very solid basis to accept this as a blocker: it breaks firstboot. I confirmed this myself. 'LC_ALL=es.UTF-8 firstboot' crashes, firstboot does not run at all. 'LC_ALL=es_ES.UTF-8 firstboot' runs fine. Other languages are affected, at least Japanese. So this bug causes firstboot to fail to run in several languages, breaking the Alpha criterion "In most cases (see Blocker_Bug_FAQ), a system installed according to any of the above criteria (or the appropriate Beta or Final criteria, when applying this criterion to those releases) must boot to the 'firstboot' utility on the first boot after installation, without unintended user intervention, unless the user explicitly chooses to boot in non-graphical mode. This includes correctly accessing any encrypted partitions when the correct passphrase is supplied. The firstboot utility must be able to create a working user account".

Created attachment 621475 [details]
anaconda-18-fix-locales.patch

Ok I cleaned up my patch a little more and made a shell script for the locale file renaming.  This now takes care of renaming all the translation files (even 2 not yet supported by glibc;).

However English (en) seems to be a special case in anaconda
so en.UTF-8 not fixed yet.  That can probably be fixed on the python side.

But lang-table is probably still a cleaner solution.

Those wanting to install with a valid locale can try installing
http://koji.fedoraproject.org/koji/taskinfo?taskID=4558387
into a Live instance.

Jens - thanks, but I've got something in anaconda itself for now.  We'll definitely be looking for a better solution in the future, though.  We really don't want to own this kind of data.

Add in a locale mapping to avoid incorrect system settings (#858591).
http://git.fedorahosted.org/cgit/anaconda.git/commit/?id=45c8c0a28ccca8ac02b89571b4fcb192c33c2103

Thanks.

Does this mean that selecting a language implicitly sets a corresponding country in the resulting locale?

"en": "en_US"
"es": "es_ES"
"ja": "ja_JP"

Some languages are still missing and these are important languages.

Bengali(India) bn_IN
Chinese(Simplified)     zh_CN
Chinese(Traditional)    zh_TW  
Portuguese(Brazilian)   pt_BR 
Serbian(Latin)  sr@latin

Those don't need to be mangled, they're already good as-is.

I ran a validation test of the locales against /usr/share/i18n/locales/ in F17:

ls: cannot access /usr/share/i18n/locales/ilo_PH: No such file or directory
ls: cannot access /usr/share/i18n/locales/kk_KK: No such file or directory
ls: cannot access /usr/share/i18n/locales/tg_TG: No such file or directory

$ cat locale-list-1.txt | sed -e 's@^@/usr/share/i18n/locales/@' | xargs ls

Locales were extracted with:

#!/bin/bash

# Extract locales from language-list-1.txt
# The list is from:
# http://git.fedorahosted.org/cgit/anaconda.git/plain/pyanaconda/localization.py

sed -e 's/[:,]/\n/g' -e 's/[" }]//g' language-list-1.txt | grep '_'

Right.
kk_KK should be kk_KZ
tg_TG should be tg_TJ

I wonder where ilo locale gone?

Guys - if you want to be involved in development, do it on the anaconda-devel and anaconda-patches mailing list, not in bugzilla.

(In reply to comment #59)
> Guys - if you want to be involved in development, do it on the
> anaconda-devel and anaconda-patches mailing list, not in bugzilla.

On F17:
$ ls /usr/share/i18n/locales/*PH
/usr/share/i18n/locales/en_PH  /usr/share/i18n/locales/fil_PH  /usr/share/i18n/locales/tl_PH

Consider this a bug report, then ... :-)

This bug has caused a lot of problems and a lot of work, so any fixes deserve a lot of scrutiny.

(In reply to comment #53)
> Jens - thanks, but I've got something in anaconda itself for now.  We'll
> definitely be looking for a better solution in the future, though.  We
> really don't want to own this kind of data.

Thanks, glad to hear it, and good to know.

(In reply to comment #54)
> Add in a locale mapping to avoid incorrect system settings (#858591).
> http://git.fedorahosted.org/cgit/anaconda.git/commit/
> ?id=45c8c0a28ccca8ac02b89571b4fcb192c33c2103

Thanks for the pointer.  I should have looked yesterday...

http://koji.fedoraproject.org/koji/taskinfo?taskID=4561583
is a test build of current anaconda HEAD.

I tested it from Live and the patch looks good so far.
So, great news that with 18.13 we should have working locales again.

(In reply to comment #58)
> I wonder where ilo locale gone?

It is not in glibc.

Created attachment 621937 [details]
anaconda-fix-kk-tg-locales.patch

> kk_KK should be kk_KZ
> tg_TG should be tg_TJ

Right - I had already noticed this while working on my hack.

Chris: can you please apply this patch to correct those two locales.

(In reply to comment #62)
> http://koji.fedoraproject.org/koji/taskinfo?taskID=4561583
> is a test build of current anaconda HEAD.

Here is an corresponding updates.img file too (untested)
if anyone wants to test with a net install or dvd.

http://petersen.fedorapeople.org/anaconda/updates-858591.img

(In reply to comment #58)
...
> I wonder where ilo locale gone?

Good question. ilo_PH was invalid in F17 too:
Bug 863450 - invalid locale ilo_PH after installing in Iloko language from F17 DVD

>Chris: can you please apply this patch to correct those two locales.

Done.

*** Bug 863510 has been marked as a duplicate of this bug. ***

anaconda-18.14-1.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/anaconda-18.14-1.fc18

Package anaconda-18.14-1.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing anaconda-18.14-1.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-15707/anaconda-18.14-1.fc18
then log in and leave karma (feedback).

F18 Beta TC3 - https://dl.fedoraproject.org/pub/alt/stage/18-Beta-TC3/ - should include this fix. Please re-test and check. Thanks!

I checked this myself, and it looks fine. Did an install in French (with UK English keyboard layout, just for fun) - the installed system shows up in French and firstboot ran fine. Setting VERIFIED.

I verified that installing in Spanish results in a valid locale of es_ES.UTF-8 and that the desktop displays Spanish. Some messages in terminals are in Spanish and some are not. Compare "ls /tmp/foo" and "sudo foo". 

It could take a while to test all of the available languages by installing ...

Tested with:
$ qemu-kvm -m 2048 -hda f18-test-1.img -cdrom ~/xfr/fedora/F18/F18-Beta/TC3/Fedora-18-Beta-TC3-x86_64-Live-Desktop.iso -usb -vga qxl -boot menu=on

I can confirm here that installation in Marathi language with US keyboard layout, set the correct locale to mr_IN.UTF-8

 work around for me was 'export LC_ALL="en_US"'

It seems OK, localization is setted correctly.

Discussed at 2012-10-10 blocker review meeting: http://meetbot.fedoraproject.org/fedora-qa/2012-10-10/f18beta-blocker-review-3.2012-10-10-16.05.log.txt . Accepted as a blocker per Alpha criterion "In most cases (see Blocker_Bug_FAQ), a system installed according to any of the above criteria (or the appropriate Beta or Final criteria, when applying this criterion to those releases) must boot to the 'firstboot' utility on the first boot after installation, without unintended user intervention, unless the user explicitly chooses to boot in non-graphical mode. This includes correctly accessing any encrypted partitions when the correct passphrase is supplied. The firstboot utility must be able to create a working user account", in the case of just about any non-English install firstboot fails to run.

Created attachment 626612 [details]
screenshot showing invalid locale eu.UTF-8 for Basque in 18.16

anaconda 18.16 is writing an invalid locale eu.UTF-8 for Basque.

This is the first result for a semi-automated locale validation test. It requires patching anaconda, but hopefully in a non-destructive way. When anaconda writes /mnt/sysimage/etc/sysconfig/i18n and /mnt/sysimage/etc/locale.conf, the patch appends the same strings to files in /tmp: langlist_i18n_1.txt and langlist_locale_conf_1.txt. (The strings appear to be identical, so having two is redundant.)

The main problem with this procedure is that the strings are written at the very end of the install, so a test for each language requires configuring the disk and waiting for the install to complete.

Created attachment 626660 [details]
screenshot showing six invalid locales configured by anaconda 18.16

The language menu items that do not show a country all produce invalid locales:
be
bs
eu
hy
ka
sr

Created attachment 626661 [details]
install.py.patch-1 patch to write the locale string to a file before installation

Writing out the locale string before installation speeded up checking the language menu items greatly. For each language tested, after confirming that /tmp/langlist_ksdata_1.txt had been updated, it was safe to kill anaconda and start a new test of the next language. I was lazy and only checked the languages that did not have a corresponding country in the language menu. :-)

$ diff -u install.py.ORIG-18.16 install.py.NEW-1 > install.py.patch-1

*** Bug 866108 has been marked as a duplicate of this bug. ***

Georgian gives an invalid locale.

Package: anaconda-18.16-1.fc18.x86_64
OS Release: Fedora release 18

Test results for Fedora-18-Beta-TC4-x86_64-DVD.iso (which has anaconda 18.16):

Selecting these languages results in an invalid locale:

Belarusian      be
Bosnian         bs
Basque          eu
Armenian        hy
Georgian        ka
Serbian         sr

These alternative languages from the language menu result in a valid locale:
Basque (Spain)      eu_ES
Serbian (Serbia)    sr_RS

There are no alternatives for Belarusian, Bosnian, Armenian, Georgian.

There are 74 languages in the language menu. I did not test every one ...

Tested by repeatedly running clean, minimal installs from the DVD.

Command-line:
$ qemu-kvm -m 2048 -hda f18-test-1.img -cdrom ~/xfr/fedora/F18/F18-Beta/TC4/Fedora-18-Beta-TC4-x86_64-DVD.iso -usb -vga qxl -boot menu=on -usbdevice mouse

Steve: can you please open a new bug for the cases which still give invalid locales? The main bug here - that anaconda was just completely broken regarding locale setting - is clearly addressed, and overloading the report will only lead to confusion. Since what you're seeing now is effectively a new issue - anaconda now does actually try to do locale setting properly, but in a few cases gets it wrong - filing it as a new bug is the appropriate approach.

Bug 866730 - invalid locales configured for some languages

This was fixed in 18.13 and 18.14 went stable, so closing.