Bug 872423

Created attachment 637384 [details]
Trivial patch to add [ia] code to the list of languages in po/LINGUAS

What is the preferred $LANG setting for this locale?

This is a tricky question.
The ISO code for the language is "ia".
The $LANG setting should be defined in the glibc locale and usually it has a form "language_REGION.encoding". Interlingua does not have a well established region and the upstream glibc maintainer refuses to accept any locales for languages like Interlingua as a matter of principle. Debian, for example, has to carry a patch for the upstream locales to add "ia_XX".

However, for the purposes of the Anaconda localization it should be enough just to have file /usr/share/locale/ia/LC_MESSAGES/anaconda.mo included into the anaconda rpm. This will make the language available in the list of the languages used during the installation. To achieve this we need to add code "ia" into the list of languages in the po/LINGUAS file in the Anaconda sources and pull data from Transifex.

To run applications with Interlingua messages after the installation it should be enough to add line LANGUAGE="ia" to files /etc/locale.conf and /etc/sysconfig/i18n. It would be ideal if Anaconda could do this but perhaps this change should be handled through another request.

To answer your question, at this point the preferred $LANG setting for Interlingua is en_US.utf8

Correction to my previous post (comment #3): Interlingua locale was accepted upstream to glibc 2.17. The locale name is ia_FR, so the LANG should be ia_FR.utf8
See: http://sourceware.org/bugzilla/show_bug.cgi?id=14879

The ia.po file was pulled from Transifex and committed to git.
However, the LINGUAS file has not been modified. It does not include the code "ia". As a result, the Intrelingua translation is not included into rpm and not available in the current Anaconda (F18 Beta).

Please, commit the patch attached to this BZ.

Transifex still lists the ia translation at 35% complete, which is pretty incomplete.  I'm holding off on this until it is more complete.  Additionally, this is not a blocker and we shouldn't be committing non-blockers at this time.

You have looked at the wrong statistics. Check anaconda->Master. It is 100% done.
35% is the statistics for the total Anaconda resources including all old branches like f15, f16, f17, rhel, etc.

Please, include Anaconda->Master to the f18 release.

Alright, but this is still not a blocker, so I can't commit it to f18-branch.

Why not? This is just a translation. The BZ was opened well before the beta. 
The translation does not require any changes in the code of the program.

I would like to add that the trivial patch - addition of the language code "ia" to the po/LINGUAS file, has not been applied even to the master branch.

As I understand, without this change the translations are not included into rpm even if the po/ia.po file is in the git repo.

In addition to a locale file[1], there needs to be support in Babel[2]. Babel supports 'ia', but not 'ia_FR':

[liveuser@localhost ~]$ python
...
>>> import babel
>>> print babel.Locale.parse('ia').english_name
Interlingua
>>> print babel.Locale.parse('ia').display_name
interlingua
>>> print babel.Locale.parse('ia_FR').display_name
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/babel/core.py", line 212, in parse
    return cls(*parse_locale(identifier, sep=sep))
  File "/usr/lib/python2.7/site-packages/babel/core.py", line 137, in __init__
    raise UnknownLocaleError(identifier)
babel.core.UnknownLocaleError: unknown locale 'ia_FR'
>>> 
[liveuser@localhost ~]$ rpm -q python-babel
python-babel-0.9.6-5.fc18.noarch
[liveuser@localhost ~]$ 

[1] $ ls /usr/share/i18n/locales/ia*
[2] Babel provides a Python interface to the CLDR:
    http://babel.edgewall.org/
    http://cldr.unicode.org/

(In reply to comment #4)
> Correction to my previous post (comment #3): Interlingua locale was accepted
> upstream to glibc 2.17. The locale name is ia_FR, so the LANG should be
> ia_FR.utf8
> See: http://sourceware.org/bugzilla/show_bug.cgi?id=14879

ATM, glibc-2.16-28.fc18 is in F18:
https://admin.fedoraproject.org/updates/search/glibc

And there are no builds of glibc-2.17:
http://koji.fedoraproject.org/koji/packageinfo?packageID=57

BTW, you can propose bugs that you open as blockers by adding the appropriate bug alias[1] to the Blocks field for the bug. QA will then review the proposal and decide whether to accept or reject the bug as a blocker.[2] At this late stage in the release process[3] and given the current lack of glibc support, there probably isn't much point in doing that for F18[4].

[1] http://fedoraproject.org/wiki/BugZappers/HouseKeeping/Trackers
[2] https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process
    https://fedoraproject.org/wiki/Fedora_Release_Criteria
[3] https://fedoraproject.org/wiki/Releases/18/Schedule
[4] F18 blocker bugs are listed here:
    http://qa.fedoraproject.org/blockerbugs/milestone/18/final/buglist

I appreciate all latest comments. I am sure all these notes will help to make better support for the language. However, I would like to reiterate my request from comment #10. Please, commit the attached patch to the git repo. This request is not a blocker but the patch does not change the code, it just adds the translated strings to the Anaconda rpm. The inclusion of these strings will facilitate the QA for the translation because it will show them in the real context of a working program. It would be a shame to waste this opportunity and wait till April-May for the next chance. Taking into account the constant flow of code changes still coming into Anaconda even at this stage of the release cycle, adding the 100% ready translation is hardly any risk for the quality of the release.

Comment #11 shows that the current version of Babel does not support ia_FR.
This is a separate issue and we will fix it separately. As I understand, Anaconda has embedded fail back code that will use 'ia' language if 'ia_FR' locale is not available in Babel. See function expand_langs in pyanaconda/localization.py.

I would like to add that the same test (as in comment #11) fails on the following locales: ast, bal, en@quote, en@boldquote, ilo, mai, nds.  The translations for all these locales are included into the current version of Anaconda and shipped with binary rpm:
anaconda-18.37.8-1.fc18
python-babel-0.9.6-5.fc18 - does not include these locales
Moreover, locales 'bal' and 'ilo' are not in the glibc at all.

Re comment #12: ia_FR is not in glibc-2.16 but it is in glibc-2.16.90 which is glibc-2.17-pre1.

(In reply to comment #14)
...
> Re comment #12: ia_FR is not in glibc-2.16 but it is in glibc-2.16.90 which
> is glibc-2.17-pre1.

Thanks for pointing that out. The glibc-2.16.90 builds are for F19:
http://koji.fedoraproject.org/koji/packageinfo?packageID=57

Unfortunately, a missing locale file causes severe problems. For the background, see:
Bug 858591 - anaconda setting invalid system locale xx.UTF-8 not xx_YY.UTF-8 
Bug 866730 - invalid locales configured for some languages

This test case shows the least of the problems caused by a missing locale file:
$ LANG=ia_FR.UTF-8 locale
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=ia_FR.UTF-8
LC_CTYPE="ia_FR.UTF-8"
LC_NUMERIC="ia_FR.UTF-8"
LC_TIME="ia_FR.UTF-8"
LC_COLLATE="ia_FR.UTF-8"
LC_MONETARY="ia_FR.UTF-8"
LC_MESSAGES="ia_FR.UTF-8"
LC_PAPER="ia_FR.UTF-8"
LC_NAME="ia_FR.UTF-8"
LC_ADDRESS="ia_FR.UTF-8"
LC_TELEPHONE="ia_FR.UTF-8"
LC_MEASUREMENT="ia_FR.UTF-8"
LC_IDENTIFICATION="ia_FR.UTF-8"
LC_ALL=

This test patch adds "ia" to the anaconda languages menu. You can do some testing by updating anaconda while running the Live CD and then applying it to localization.py:

$ cat interlingua-test-1.patch 
--- localization.py.ORIG	2012-12-21 20:20:23.000000000 -0500
+++ localization.py.EXP1	2012-12-22 11:07:35.567768375 -0500
@@ -178,6 +178,7 @@
     # usually there are no message files for en
     if 'en' not in languages:
         languages.append('en')
+    languages.append('ia')
 
     for langcode in languages:
         try:

(In reply to comment #14)
...
> I would like to add that the same test (as in comment #11) fails on the
> following locales: ast, bal, en@quote, en@boldquote, ilo, mai, nds.  The
...
> Moreover, locales 'bal' and 'ilo' are not in the glibc at all.
...

Those are either lacking a locale file or Babel support. To add a language, there needs to be:
1. a translation file
2. a locale file
3. Babel support

(See Bug 866730.)

Now, after the F18 release, I would like to push this change to the rawhide with the target to F19.

I tested the translation with F18 Live Desktop using the following procedure:
1. Boot F18 Live Desktop media.
2. Download ia_FR locale definition file and add it to the running system:
   localedef -c -f UTF-8 -i ia_FR ia_FR.UTF-8
3. Make directory for Anaconda localization file:
   mkdir -p /usr/share/locale/ia/LC_MESSAGES
4. Download anaconda.mo file with Interlingua strings to the created directory.
5. Start Anaconda by running liveinst.
6. Select "Interlingua" on the first screen.

Note that the patch described in comment 16 was not necessary.
Everything was OK and I was able to install the system with Interlingua locale configured in /etc/locale.conf, however this locale was "ia.UTF-8" because the mangleLocale in localization.py did not have the record mapping "ia" to "ia_FR".

The attached patch fixes this minor problem.

I would like to note that all requirements specified in comment 17 are satisfied for rawhide/F19:

1. Translation file is ready and complete (but not included into binary rpm)
2. Locale file ia_FR is available in glibc-2.17
3. The existing python-babel package contains locale data for "ia" locale and this is enough for normal operations. The sublocale "ia_FR" is not necessary.

Please, commit the attached patch.

Created attachment 680268 [details]
Add ia to ia_FR mapping in mangleLocale

I was wrong in my comment 18: if we apply a patch mapping "ia" to "ia_FR" in the mangleLocale then that locale must be in the python-babel package. Unfortunately, python-babel-0.9.6 does not contain "ia_FR" locale, it has only "ia". The chances for "ia_FR" to be added into python-babel are slim. The package is still using CLDR data version 1.7 released in 2009.

We need a workaround that will allow Anaconda to use data from the "ia" Babel/CLDR locale to get proper language name for use in GUI and then store the value "ia_FR.UTF-8" in the /etc/locale.conf file. The problem is that the "ia" locale in CLDR does not contain data about the territory. This is available in the glibc locale. 

It would be better if the localization code in Anaconda allow users to run Anaconda GUI in the language of their choice based on the available translations and than select the default system locale based on the list of locales available in glibc. This way an operator performing system provisioning can use her localized GUI to install a system for users in other locale. Moreover, there are some disparity between number of locales in glibc and babel/CLDR and the default system locale must be based on the data from glibc.

CLDR database was updated and now it includes ia_FR locale.
However, this change has not been reflected in the python-babel packaged. It seems that the Babel project is stuck on version 0.9 which is not compatible with the latest CLDR format. No updates to the locale data were included into Babel since 2009. I doubt that Anaconda's dependency from python-babel is such a good thing.

Could the developers responsible for Anaconda localization and spokes using locale data for system configuration comment about reliance on Babel or possible plans for its substitution?

Thanks for the update.

I can't find ia_FR at the CLDR web site:
http://cldr.unicode.org/

Do you have a link or a link to a Trac ticket?

There is a Babel Trac ticket to update the CLDR:
Upgrade to CLDR 21
http://babel.edgewall.org/ticket/312

CLDR ticket about ia_FR is here:
http://unicode.org/cldr/trac/ticket/5573

The ia_FR.xml record itself is in svn repo. Target release is CLDR 23dres (due 3/1/13). As you can see, Babel is quite behind. Their ticket #312 was opened 6 months ago and targets CLDR 21. The last activity in that branch of code was on 8/21.

The current CLDR release is 22.1 and CLDR 23 is scheduled on 3/15/13.

This bug appears to have been reported against 'rawhide' during the Fedora 19 development cycle.
Changing version to '19'.

(As we did not run this process for some time, it could affect also pre-Fedora 19 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 19 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora19

Discussed at 2013-04-24 blocker review meeting: http://meetbot.fedoraproject.org/fedora-blocker-review/2013-04-24/f19beta-blocker-review-1.2013-04-24-16.00.log.txt . Rejected as a blocker as this does not appear to meet any of the Beta criteria. I'm re-proposing for Final under the criterion "All critical path actions on release-blocking desktop environments should correctly display all sufficiently complete translations available for use".

As a matter of fact important for the consideration for Final, full translation of Anaconda is available for F19 (at Transifex, but it is not committed to git yet) and sufficiently complete localization of KDE desktop is included into F19 already.

I can hack around the mangleMap problem for now, but my doing so is contingent upon a bug being filed against python-babel to include the correct information (doesn't have to be a new upstream release - could be a patch on their end).  I don't want to carry hacks in anaconda forever.  We have worked extremely hard to get rid of hacks.

Second, there is no appetite here for replacing python-babel usage in anaconda.  We're all pretty tired of having to chase internationalization problems and have more pressing issues to deal with.  Like it or not, that's the situation here and it's not going to be changing soon.

Nik, what keyboard layout would you like to use by default for Interlingua?

langtable-0.0.4-1.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/langtable-0.0.4-1.fc19

https://github.com/mike-fabian/langtable/commit/fcb7e923fe2b00987aee91f39c082882b5c54f23

Package langtable-0.0.4-1.fc19:
* should fix your issue,
* was pushed to the Fedora 19 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing langtable-0.0.4-1.fc19'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-9365/langtable-0.0.4-1.fc19
then log in and leave karma (feedback).

Discussed at 2013-05-29 blocker review meeting: http://meetbot.fedoraproject.org/fedora-blocker-review/2013-05-29/f19final-blocker-review-1.2013-05-29-16.02.log.txt . Accepted as a blocker per criterion cited in c#25.

Also fixed in 19.30.1-1, which is now in updates-testing.

Looks good in 19.30.1-1: Interlingua is on the list of languages, picking it works, the translations are used.

...but see https://bugzilla.redhat.com/show_bug.cgi?id=968808 .

Answer to Mike Fabian, comment #28: us(euro)

(In reply to Nik Kalach from comment #36)
> Answer to Mike Fabian, comment #28: us(euro)

https://github.com/mike-fabian/langtable/commit/77ad6167566d417248b94e96a965b8af1454e898

langtable-0.0.4-1.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.