Bug 872423 - Add Interlingua to the list of available languages
Summary: Add Interlingua to the list of available languages
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: anaconda
Version: 19
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Chris Lumens
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedBlocker
Depends On:
Blocks: F19Blocker, F19FinalBlocker
TreeView+ depends on / blocked
 
Reported: 2012-11-02 03:30 UTC by Nik Kalach
Modified: 2013-06-05 03:22 UTC (History)
8 users (show)

Fixed In Version: langtable-0.0.4-1.fc19
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-06-05 03:22:49 UTC


Attachments (Terms of Use)
Trivial patch to add [ia] code to the list of languages in po/LINGUAS (739 bytes, patch)
2012-11-03 04:49 UTC, Nik Kalach
no flags Details | Diff
Add ia to ia_FR mapping in mangleLocale (3.27 KB, patch)
2013-01-17 15:03 UTC, Nik Kalach
no flags Details | Diff

Description Nik Kalach 2012-11-02 03:30:36 UTC
Description of problem:

Anaconda does not include support for Interlingua.
The /usr/share/locales/ia/LC_MESSAGES/anaconda.mo is not in the binary rpm.
Anaconda-master resource on Transifex is 100% translated to Interlingua but Anaconda package does not include the translation. The source rpm does not have the po/ia.po file and the language code "ia" is not in the list in the po/LINGUAS file.


Version-Release number of selected component (if applicable):
anaconda-18.22-1

How reproducible:
Always

Steps to Reproduce:
1. rpm -qlp anaconda-18.22-1.fc18.x86_64.rpm | grep locale/ia
  
Actual results:
Nothing

Expected results:
/usr/share/locale/ia/LC_MESSAGES/anaconda.mo

Comment 1 Nik Kalach 2012-11-03 04:49:09 UTC
Created attachment 637384 [details]
Trivial patch to add [ia] code to the list of languages in po/LINGUAS

Comment 2 Chris Lumens 2012-11-05 15:08:37 UTC
What is the preferred $LANG setting for this locale?

Comment 3 Nik Kalach 2012-11-05 17:14:30 UTC
This is a tricky question.
The ISO code for the language is "ia".
The $LANG setting should be defined in the glibc locale and usually it has a form "language_REGION.encoding". Interlingua does not have a well established region and the upstream glibc maintainer refuses to accept any locales for languages like Interlingua as a matter of principle. Debian, for example, has to carry a patch for the upstream locales to add "ia_XX".

However, for the purposes of the Anaconda localization it should be enough just to have file /usr/share/locale/ia/LC_MESSAGES/anaconda.mo included into the anaconda rpm. This will make the language available in the list of the languages used during the installation. To achieve this we need to add code "ia" into the list of languages in the po/LINGUAS file in the Anaconda sources and pull data from Transifex.

To run applications with Interlingua messages after the installation it should be enough to add line LANGUAGE="ia" to files /etc/locale.conf and /etc/sysconfig/i18n. It would be ideal if Anaconda could do this but perhaps this change should be handled through another request.

To answer your question, at this point the preferred $LANG setting for Interlingua is en_US.utf8

Comment 4 Nik Kalach 2012-11-29 16:12:20 UTC
Correction to my previous post (comment #3): Interlingua locale was accepted upstream to glibc 2.17. The locale name is ia_FR, so the LANG should be ia_FR.utf8
See: http://sourceware.org/bugzilla/show_bug.cgi?id=14879

Comment 5 Nik Kalach 2012-12-07 22:22:44 UTC
The ia.po file was pulled from Transifex and committed to git.
However, the LINGUAS file has not been modified. It does not include the code "ia". As a result, the Intrelingua translation is not included into rpm and not available in the current Anaconda (F18 Beta).

Please, commit the patch attached to this BZ.

Comment 6 Chris Lumens 2012-12-10 15:15:57 UTC
Transifex still lists the ia translation at 35% complete, which is pretty incomplete.  I'm holding off on this until it is more complete.  Additionally, this is not a blocker and we shouldn't be committing non-blockers at this time.

Comment 7 Nik Kalach 2012-12-10 15:47:29 UTC
You have looked at the wrong statistics. Check anaconda->Master. It is 100% done.
35% is the statistics for the total Anaconda resources including all old branches like f15, f16, f17, rhel, etc.

Please, include Anaconda->Master to the f18 release.

Comment 8 Chris Lumens 2012-12-10 16:01:27 UTC
Alright, but this is still not a blocker, so I can't commit it to f18-branch.

Comment 9 Nik Kalach 2012-12-10 16:30:56 UTC
Why not? This is just a translation. The BZ was opened well before the beta. 
The translation does not require any changes in the code of the program.

Comment 10 Nik Kalach 2012-12-10 16:47:20 UTC
I would like to add that the trivial patch - addition of the language code "ia" to the po/LINGUAS file, has not been applied even to the master branch.

As I understand, without this change the translations are not included into rpm even if the po/ia.po file is in the git repo.

Comment 11 Steve Tyler 2012-12-22 16:51:22 UTC
In addition to a locale file[1], there needs to be support in Babel[2]. Babel supports 'ia', but not 'ia_FR':

[liveuser@localhost ~]$ python
...
>>> import babel
>>> print babel.Locale.parse('ia').english_name
Interlingua
>>> print babel.Locale.parse('ia').display_name
interlingua
>>> print babel.Locale.parse('ia_FR').display_name
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/babel/core.py", line 212, in parse
    return cls(*parse_locale(identifier, sep=sep))
  File "/usr/lib/python2.7/site-packages/babel/core.py", line 137, in __init__
    raise UnknownLocaleError(identifier)
babel.core.UnknownLocaleError: unknown locale 'ia_FR'
>>> 
[liveuser@localhost ~]$ rpm -q python-babel
python-babel-0.9.6-5.fc18.noarch
[liveuser@localhost ~]$ 

[1] $ ls /usr/share/i18n/locales/ia*
[2] Babel provides a Python interface to the CLDR:
    http://babel.edgewall.org/
    http://cldr.unicode.org/

Comment 12 Steve Tyler 2012-12-22 16:58:37 UTC
(In reply to comment #4)
> Correction to my previous post (comment #3): Interlingua locale was accepted
> upstream to glibc 2.17. The locale name is ia_FR, so the LANG should be
> ia_FR.utf8
> See: http://sourceware.org/bugzilla/show_bug.cgi?id=14879

ATM, glibc-2.16-28.fc18 is in F18:
https://admin.fedoraproject.org/updates/search/glibc

And there are no builds of glibc-2.17:
http://koji.fedoraproject.org/koji/packageinfo?packageID=57

Comment 13 Steve Tyler 2012-12-22 22:10:52 UTC
BTW, you can propose bugs that you open as blockers by adding the appropriate bug alias[1] to the Blocks field for the bug. QA will then review the proposal and decide whether to accept or reject the bug as a blocker.[2] At this late stage in the release process[3] and given the current lack of glibc support, there probably isn't much point in doing that for F18[4].

[1] http://fedoraproject.org/wiki/BugZappers/HouseKeeping/Trackers
[2] https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process
    https://fedoraproject.org/wiki/Fedora_Release_Criteria
[3] https://fedoraproject.org/wiki/Releases/18/Schedule
[4] F18 blocker bugs are listed here:
    http://qa.fedoraproject.org/blockerbugs/milestone/18/final/buglist

Comment 14 Nik Kalach 2012-12-24 20:32:52 UTC
I appreciate all latest comments. I am sure all these notes will help to make better support for the language. However, I would like to reiterate my request from comment #10. Please, commit the attached patch to the git repo. This request is not a blocker but the patch does not change the code, it just adds the translated strings to the Anaconda rpm. The inclusion of these strings will facilitate the QA for the translation because it will show them in the real context of a working program. It would be a shame to waste this opportunity and wait till April-May for the next chance. Taking into account the constant flow of code changes still coming into Anaconda even at this stage of the release cycle, adding the 100% ready translation is hardly any risk for the quality of the release.

Comment #11 shows that the current version of Babel does not support ia_FR.
This is a separate issue and we will fix it separately. As I understand, Anaconda has embedded fail back code that will use 'ia' language if 'ia_FR' locale is not available in Babel. See function expand_langs in pyanaconda/localization.py.

I would like to add that the same test (as in comment #11) fails on the following locales: ast, bal, en@quote, en@boldquote, ilo, mai, nds.  The translations for all these locales are included into the current version of Anaconda and shipped with binary rpm:
anaconda-18.37.8-1.fc18
python-babel-0.9.6-5.fc18 - does not include these locales
Moreover, locales 'bal' and 'ilo' are not in the glibc at all.

Re comment #12: ia_FR is not in glibc-2.16 but it is in glibc-2.16.90 which is glibc-2.17-pre1.

Comment 15 Steve Tyler 2012-12-24 21:11:54 UTC
(In reply to comment #14)
...
> Re comment #12: ia_FR is not in glibc-2.16 but it is in glibc-2.16.90 which
> is glibc-2.17-pre1.

Thanks for pointing that out. The glibc-2.16.90 builds are for F19:
http://koji.fedoraproject.org/koji/packageinfo?packageID=57

Unfortunately, a missing locale file causes severe problems. For the background, see:
Bug 858591 - anaconda setting invalid system locale xx.UTF-8 not xx_YY.UTF-8 
Bug 866730 - invalid locales configured for some languages

This test case shows the least of the problems caused by a missing locale file:
$ LANG=ia_FR.UTF-8 locale
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=ia_FR.UTF-8
LC_CTYPE="ia_FR.UTF-8"
LC_NUMERIC="ia_FR.UTF-8"
LC_TIME="ia_FR.UTF-8"
LC_COLLATE="ia_FR.UTF-8"
LC_MONETARY="ia_FR.UTF-8"
LC_MESSAGES="ia_FR.UTF-8"
LC_PAPER="ia_FR.UTF-8"
LC_NAME="ia_FR.UTF-8"
LC_ADDRESS="ia_FR.UTF-8"
LC_TELEPHONE="ia_FR.UTF-8"
LC_MEASUREMENT="ia_FR.UTF-8"
LC_IDENTIFICATION="ia_FR.UTF-8"
LC_ALL=

Comment 16 Steve Tyler 2012-12-24 21:21:52 UTC
This test patch adds "ia" to the anaconda languages menu. You can do some testing by updating anaconda while running the Live CD and then applying it to localization.py:

$ cat interlingua-test-1.patch 
--- localization.py.ORIG	2012-12-21 20:20:23.000000000 -0500
+++ localization.py.EXP1	2012-12-22 11:07:35.567768375 -0500
@@ -178,6 +178,7 @@
     # usually there are no message files for en
     if 'en' not in languages:
         languages.append('en')
+    languages.append('ia')
 
     for langcode in languages:
         try:

Comment 17 Steve Tyler 2012-12-24 21:29:40 UTC
(In reply to comment #14)
...
> I would like to add that the same test (as in comment #11) fails on the
> following locales: ast, bal, en@quote, en@boldquote, ilo, mai, nds.  The
...
> Moreover, locales 'bal' and 'ilo' are not in the glibc at all.
...

Those are either lacking a locale file or Babel support. To add a language, there needs to be:
1. a translation file
2. a locale file
3. Babel support

(See Bug 866730.)

Comment 18 Nik Kalach 2013-01-17 15:00:28 UTC
Now, after the F18 release, I would like to push this change to the rawhide with the target to F19.

I tested the translation with F18 Live Desktop using the following procedure:
1. Boot F18 Live Desktop media.
2. Download ia_FR locale definition file and add it to the running system:
   localedef -c -f UTF-8 -i ia_FR ia_FR.UTF-8
3. Make directory for Anaconda localization file:
   mkdir -p /usr/share/locale/ia/LC_MESSAGES
4. Download anaconda.mo file with Interlingua strings to the created directory.
5. Start Anaconda by running liveinst.
6. Select "Interlingua" on the first screen.

Note that the patch described in comment 16 was not necessary.
Everything was OK and I was able to install the system with Interlingua locale configured in /etc/locale.conf, however this locale was "ia.UTF-8" because the mangleLocale in localization.py did not have the record mapping "ia" to "ia_FR".

The attached patch fixes this minor problem.

I would like to note that all requirements specified in comment 17 are satisfied for rawhide/F19:

1. Translation file is ready and complete (but not included into binary rpm)
2. Locale file ia_FR is available in glibc-2.17
3. The existing python-babel package contains locale data for "ia" locale and this is enough for normal operations. The sublocale "ia_FR" is not necessary.

Please, commit the attached patch.

Comment 19 Nik Kalach 2013-01-17 15:03:31 UTC
Created attachment 680268 [details]
Add ia to ia_FR mapping in mangleLocale

Comment 20 Nik Kalach 2013-01-18 15:15:24 UTC
I was wrong in my comment 18: if we apply a patch mapping "ia" to "ia_FR" in the mangleLocale then that locale must be in the python-babel package. Unfortunately, python-babel-0.9.6 does not contain "ia_FR" locale, it has only "ia". The chances for "ia_FR" to be added into python-babel are slim. The package is still using CLDR data version 1.7 released in 2009.

We need a workaround that will allow Anaconda to use data from the "ia" Babel/CLDR locale to get proper language name for use in GUI and then store the value "ia_FR.UTF-8" in the /etc/locale.conf file. The problem is that the "ia" locale in CLDR does not contain data about the territory. This is available in the glibc locale. 

It would be better if the localization code in Anaconda allow users to run Anaconda GUI in the language of their choice based on the available translations and than select the default system locale based on the list of locales available in glibc. This way an operator performing system provisioning can use her localized GUI to install a system for users in other locale. Moreover, there are some disparity between number of locales in glibc and babel/CLDR and the default system locale must be based on the data from glibc.

Comment 21 Nik Kalach 2013-02-15 14:42:46 UTC
CLDR database was updated and now it includes ia_FR locale.
However, this change has not been reflected in the python-babel packaged. It seems that the Babel project is stuck on version 0.9 which is not compatible with the latest CLDR format. No updates to the locale data were included into Babel since 2009. I doubt that Anaconda's dependency from python-babel is such a good thing.

Could the developers responsible for Anaconda localization and spokes using locale data for system configuration comment about reliance on Babel or possible plans for its substitution?

Comment 22 Steve Tyler 2013-02-15 20:45:41 UTC
Thanks for the update.

I can't find ia_FR at the CLDR web site:
http://cldr.unicode.org/

Do you have a link or a link to a Trac ticket?

There is a Babel Trac ticket to update the CLDR:
Upgrade to CLDR 21
http://babel.edgewall.org/ticket/312

Comment 23 Nik Kalach 2013-02-15 22:39:55 UTC
CLDR ticket about ia_FR is here:
http://unicode.org/cldr/trac/ticket/5573

The ia_FR.xml record itself is in svn repo. Target release is CLDR 23dres (due 3/1/13). As you can see, Babel is quite behind. Their ticket #312 was opened 6 months ago and targets CLDR 21. The last activity in that branch of code was on 8/21.

The current CLDR release is 22.1 and CLDR 23 is scheduled on 3/15/13.

Comment 24 Fedora End Of Life 2013-04-03 13:39:35 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 19 development cycle.
Changing version to '19'.

(As we did not run this process for some time, it could affect also pre-Fedora 19 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 19 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora19

Comment 25 Adam Williamson 2013-04-24 16:28:35 UTC
Discussed at 2013-04-24 blocker review meeting: http://meetbot.fedoraproject.org/fedora-blocker-review/2013-04-24/f19beta-blocker-review-1.2013-04-24-16.00.log.txt . Rejected as a blocker as this does not appear to meet any of the Beta criteria. I'm re-proposing for Final under the criterion "All critical path actions on release-blocking desktop environments should correctly display all sufficiently complete translations available for use".

Comment 26 Nik Kalach 2013-04-24 18:18:16 UTC
As a matter of fact important for the consideration for Final, full translation of Anaconda is available for F19 (at Transifex, but it is not committed to git yet) and sufficiently complete localization of KDE desktop is included into F19 already.

Comment 27 Chris Lumens 2013-05-24 19:48:56 UTC
I can hack around the mangleMap problem for now, but my doing so is contingent upon a bug being filed against python-babel to include the correct information (doesn't have to be a new upstream release - could be a patch on their end).  I don't want to carry hacks in anaconda forever.  We have worked extremely hard to get rid of hacks.

Second, there is no appetite here for replacing python-babel usage in anaconda.  We're all pretty tired of having to chase internationalization problems and have more pressing issues to deal with.  Like it or not, that's the situation here and it's not going to be changing soon.

Comment 28 Mike FABIAN 2013-05-27 10:13:21 UTC
Nik, what keyboard layout would you like to use by default for Interlingua?

Comment 29 Fedora Update System 2013-05-27 10:41:27 UTC
langtable-0.0.4-1.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/langtable-0.0.4-1.fc19

Comment 31 Fedora Update System 2013-05-27 17:01:39 UTC
Package langtable-0.0.4-1.fc19:
* should fix your issue,
* was pushed to the Fedora 19 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing langtable-0.0.4-1.fc19'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-9365/langtable-0.0.4-1.fc19
then log in and leave karma (feedback).

Comment 32 Adam Williamson 2013-05-29 16:32:54 UTC
Discussed at 2013-05-29 blocker review meeting: http://meetbot.fedoraproject.org/fedora-blocker-review/2013-05-29/f19final-blocker-review-1.2013-05-29-16.02.log.txt . Accepted as a blocker per criterion cited in c#25.

Comment 33 Adam Williamson 2013-05-30 02:37:05 UTC
Also fixed in 19.30.1-1, which is now in updates-testing.

Comment 34 Adam Williamson 2013-05-30 02:42:02 UTC
Looks good in 19.30.1-1: Interlingua is on the list of languages, picking it works, the translations are used.

Comment 35 Adam Williamson 2013-05-30 02:57:31 UTC
...but see https://bugzilla.redhat.com/show_bug.cgi?id=968808 .

Comment 36 Nik Kalach 2013-06-03 04:14:06 UTC
Answer to Mike Fabian, comment #28: us(euro)

Comment 37 Mike FABIAN 2013-06-03 06:24:11 UTC
(In reply to Nik Kalach from comment #36)
> Answer to Mike Fabian, comment #28: us(euro)

https://github.com/mike-fabian/langtable/commit/77ad6167566d417248b94e96a965b8af1454e898

Comment 38 Fedora Update System 2013-06-05 03:22:49 UTC
langtable-0.0.4-1.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.