Bug 1238406 - Glibc locale subpackaging
Glibc locale subpackaging
Status: CLOSED NEXTRELEASE
Product: Fedora
Classification: Fedora
Component: Changes Tracking (Show other bugs)
24
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Mike FABIAN
ChangeAcceptedF23, SystemWideChange, ...
:
Depends On: 1300569
Blocks:
  Show dependency treegraph
 
Reported: 2015-07-01 15:45 EDT by Jan Kurik
Modified: 2016-04-20 10:46 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-03-26 15:02:14 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Jan Kurik 2015-07-01 15:45:21 EDT
This is a tracking bug for Change: Glibc locale subpackaging
For more details, see: https://fedoraproject.org//wiki/Changes/Glibc_locale_subpackaging

This change should make it possible to install or uninstall locales individually.
Comment 1 Jan Kurik 2015-07-14 10:02:52 EDT
This message is a reminder that Fedora 23 Change Checkpoint: Completion deadline (testable) is on 2015-07-28 [1].

At this point, all accepted Changes should be substantially complete, and testable. Additionally, if a change is to be enabled by default, it must be so enabled at Change Completion deadline.

This bug should be set at least to the MODIFIED state to indicate that it achieved completeness. Status will be provided to FESCo right after the deadline. If, for any reasons, your Change is not in required state, let me know and we will try to find solution. For Changes you decide to cancel/move to the next release, please use the NEW status and set needinfo on me and it will be acted upon. 

In case of any questions, don't hesitate to ask Wrangler (jkurik). Thank you.

[1] https://fedoraproject.org/wiki/Releases/23/Schedule
Comment 2 Jan Kurik 2015-07-15 09:52:04 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 23 development cycle.
Changing version to '23'.

(As we did not run this process for some time, it could affect also pre-Fedora 23 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 23 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora23
Comment 3 Mike FABIAN 2015-08-03 09:58:27 EDT
There are still too many open questions and I have not yet started the implementation. Therefore, I think this must be moved to rawhide.
Comment 4 Parag Nemade 2015-08-05 02:40:15 EDT
Thanks Mike for providing update, I am removing(by commenting) this Change from https://fedoraproject.org/wiki/Releases/23/ChangeSet and moving back the category of wiki page as ChangePageIncomplete state.
Comment 5 Mike FABIAN 2016-01-20 06:00:20 EST
The proposed change is pushed to the
git branch "private-mfabian-locale-subpackaging-folders" in
our Fedora git repository for the glibc package.

Packages for testing the locale sub-packaging on F23 are here: https://mfabian.fedorapeople.org/glibc/
Comment 6 Mike FABIAN 2016-01-21 03:15:00 EST
Copr repo for testing:

https://copr.fedoraproject.org/coprs/mfabian/glibc/
Comment 7 Mike FABIAN 2016-02-15 11:50:02 EST
New branch for this in the git-dist repo for glibc:

private-mfabian-locale-subpackaging-folders-new
Comment 8 Jan Kurik 2016-02-24 09:26:17 EST
On 2016-Feb-23, we have reached Fedora 24 Change Checkpoint: Completion deadline (testable).

At this point, all accepted changes should be substantially complete, and testable. Additionally, if a change is to be enabled by default, it must be so enabled at Change Completion deadline.

Change tracking bug should be set to the MODIFIED state to indicate it achieved completeness.

Incomplete and non testable Changes will be reported to FESCo on 2016-Feb-26 meeting.  Contingency plan for System Wide Changes, if planned for Alpha (or in case of serious doubts regarding Change completion), will be activated.
Comment 9 Mike FABIAN 2016-02-25 09:31:22 EST
The latest work is now in the 

private-mfabian-locale-subpackaging-folders-2016-02-25

branch in dist-git.

Seems to work fine. After a final review by Carlos O'Donell, I think
I can push it to rawhide/f24 today.
Comment 10 Stas Sergeev 2016-02-29 16:50:58 EST
Hello.

After the today's "yum update" (on f24) I've lost my locale
and half of the programs stopped working. Others are unreadable.

$ locale
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=ru_RU.UTF-8
LC_CTYPE="ru_RU.UTF-8"
LC_NUMERIC="ru_RU.UTF-8"
LC_TIME="ru_RU.UTF-8"
LC_COLLATE="ru_RU.UTF-8"
LC_MONETARY="ru_RU.UTF-8"
LC_MESSAGES="ru_RU.UTF-8"
LC_PAPER="ru_RU.UTF-8"
LC_NAME="ru_RU.UTF-8"
LC_ADDRESS="ru_RU.UTF-8"
LC_TELEPHONE="ru_RU.UTF-8"
LC_MEASUREMENT="ru_RU.UTF-8"
LC_IDENTIFICATION="ru_RU.UTF-8"
LC_ALL=

$ locale -a
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_COLLATE to default locale: No such file or directory
C
C.utf8
POSIX


$ gnome-terminal 
(process:3770): Gtk-WARNING **: Locale not supported by C library.
        Using the fallback 'C' locale.
Error constructing proxy for org.gnome.Terminal:/org/gnome/Terminal/Factory0: Error calling StartServiceByName for org.gnome.Terminal: GDBus.Error:org.freedesktop.DBus.Error.Spawn.ChildExited: Process org.gnome.Terminal exited with status 9


How to bring the system back to live? Help!
Comment 11 Carlos O'Donell 2016-02-29 22:55:27 EST
(In reply to Stas Sergeev from comment #10)
> Hello.
> 
> After the today's "yum update" (on f24) I've lost my locale
> and half of the programs stopped working. Others are unreadable.
> 
> $ locale
> locale: Cannot set LC_CTYPE to default locale: No such file or directory
> locale: Cannot set LC_MESSAGES to default locale: No such file or directory
> locale: Cannot set LC_ALL to default locale: No such file or directory
> LANG=ru_RU.UTF-8
> LC_CTYPE="ru_RU.UTF-8"
> LC_NUMERIC="ru_RU.UTF-8"
> LC_TIME="ru_RU.UTF-8"
> LC_COLLATE="ru_RU.UTF-8"
> LC_MONETARY="ru_RU.UTF-8"
> LC_MESSAGES="ru_RU.UTF-8"
> LC_PAPER="ru_RU.UTF-8"
> LC_NAME="ru_RU.UTF-8"
> LC_ADDRESS="ru_RU.UTF-8"
> LC_TELEPHONE="ru_RU.UTF-8"
> LC_MEASUREMENT="ru_RU.UTF-8"
> LC_IDENTIFICATION="ru_RU.UTF-8"
> LC_ALL=
> 
> $ locale -a
> locale: Cannot set LC_CTYPE to default locale: No such file or directory
> locale: Cannot set LC_MESSAGES to default locale: No such file or directory
> locale: Cannot set LC_COLLATE to default locale: No such file or directory
> C
> C.utf8
> POSIX
> 
> 
> $ gnome-terminal 
> (process:3770): Gtk-WARNING **: Locale not supported by C library.
>         Using the fallback 'C' locale.
> Error constructing proxy for
> org.gnome.Terminal:/org/gnome/Terminal/Factory0: Error calling
> StartServiceByName for org.gnome.Terminal:
> GDBus.Error:org.freedesktop.DBus.Error.Spawn.ChildExited: Process
> org.gnome.Terminal exited with status 9
> 
> 
> How to bring the system back to live? Help!

`dnf install glibc-all-langpacks` will fix this. We are working to resolver this issue as quickly as possible.
Comment 12 Carlos O'Donell 2016-03-01 03:36:12 EST
With glibc-2.23.90-3.fc25 and glibc-2.23.1-5.fc24 we have now fixed
the issue in several important ways.

See:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/WXOOUGA3YCB2O4O77JGLSJZ25BS4RFK5/

You should now always get glibc-all-langpacks by default.
Comment 13 Stas Sergeev 2016-03-01 07:56:46 EST
I still don't understand why things broke in a
way they did. If I would be switched the the C
locale then fine. Why instead programs started
failing to start, or write unreadable chars?
Surely the fall-back should be handled better.
Comment 14 Carlos O'Donell 2016-03-01 09:47:17 EST
(In reply to Stas Sergeev from comment #13)
> I still don't understand why things broke in a
> way they did. If I would be switched the the C
> locale then fine. Why instead programs started
> failing to start, or write unreadable chars?
> Surely the fall-back should be handled better.

The failure is entirely on the part of the glibc team. We didn't test enough end-user applications to determine that several key applications would actually fail if a call to setlocale failed. In theory these applications are poorly written and not robust. These applications need to all be updated to handle running in C or C.UTF-8 locales. In practice we had to fix these cases by changing the dependency requirements to install all the glibc language packs when transitioning from a non-language-pack system to one that supports language-packs. We will continue to work to ensure this transition is smooth. We will also look into running systems in pure C.UTF-8 mode i.e. only glibc-minimal-langpack installed and file bugs for every program that fails to operate.
Comment 15 Stas Sergeev 2016-03-01 10:07:13 EST
> The failure is entirely on the part of the glibc team. We didn't test enough 
> end-user applications to determine that several key applications would
> actually fail if a call to setlocale failed. In theory these applications are
> poorly written and not robust.
It seems to me that even setlocale(LC_ALL,"") fails.
According to the man page,
       For  glibc, first (regardless of cate‐
       gory), the environment variable LC_ALL is inspected, next the  environ‐
       ment variable with the same name as the category (see the table above),
       and finally the environment variable LANG.  The first existing environ‐
       ment  variable  is used.  If its value is not a valid locale specifica‐
       tion, the locale is unchanged, and setlocale() returns NULL.

... so in its current implementation it should indeed fail.
Wouldn't it be better if it try some fall-back like "C" first?
IMHO that glibc-specific implementation is not robust, not the apps.
You want the apps to simply ignore the setlocale(LC_ALL,"") return value?
Comment 16 Carlos O'Donell 2016-03-01 10:49:24 EST
(In reply to Stas Sergeev from comment #15)
> > The failure is entirely on the part of the glibc team. We didn't test enough 
> > end-user applications to determine that several key applications would
> > actually fail if a call to setlocale failed. In theory these applications are
> > poorly written and not robust.
> It seems to me that even setlocale(LC_ALL,"") fails.

Correct.

> ... so in its current implementation it should indeed fail.

Correct.

> Wouldn't it be better if it try some fall-back like "C" first?

No. It's the responsibility of the application to do this, because only the application knows if a translation is critical, particuarly when UTF-8 support is needed.

> IMHO that glibc-specific implementation is not robust, not the apps.
> You want the apps to simply ignore the setlocale(LC_ALL,"") return value?

No. The application must decide if the locale is critical to operation or not. In many of the cases I've seen, particularly gnome-terminal, the terminal could have continued to operate with the C or C.UTF-8 locale. I see no reason for gnome-terminal to exit with an error. Therefor it should be robust and (a) try C.UTF-8 to get UTF-8 support, or otherwise (b) fall back to trying C, and using that.

For something like ssh the question is more difficult to answer because converting from UTF-8 input to ASCII is lossy, and may result in output that is unrepresentable and makes the terminal useless. Even then ssh could also fall back to C.UTF-8 and if that doesn't work (it would allow you to represent all language characters but not their cultural conventions) then you might fail instead of trying C.

Does that clarify our position?
Comment 17 Stas Sergeev 2016-03-01 11:15:51 EST
> No. It's the responsibility of the application to do this, because only the
> application knows if a translation is critical, particuarly when UTF-8 support
> is needed.
Could you please confirm that you mean exactly setlocale(LC_ALL,"")?
I thought if an apps does this, it implies that the translation
is not critical. Please note that the problem is not only that
the apps do not start. I suppose the ones that do not start, are
indeed locale-sensitive, and they do not do setlocale(LC_ALL,"")
at first place (maybe they do setlocale(LC_ALL,"C.UTF-8") or something -
then they should be fixed the way you suggest).
The problem is that the apps that DO start, still write the unreadable
chars. This presumably comes from the fact that setlocale(LC_ALL,"")
fails, but they ignore the return value.
I wonder if someone ever checks the return value of setlocale(LC_ALL,"")
at all. So maybe at least this part of the problem can be fixed on a
glibc level?
Comment 18 Carlos O'Donell 2016-03-01 11:21:56 EST
(In reply to Stas Sergeev from comment #17)
> > No. It's the responsibility of the application to do this, because only the
> > application knows if a translation is critical, particuarly when UTF-8 support
> > is needed.
> Could you please confirm that you mean exactly setlocale(LC_ALL,"")?

Calling setlocale(LC_ALL,"") simply means that the application has been tested with and expects to work with *all* locales the user might possible set via environment variables. Such a call *may* fail if the user sets invalid values in their environment variables, and the application needs to decide if it should or should not support this case.

> I thought if an apps does this, it implies that the translation
> is not critical. 

It does not imply that at all. Calling setlocale(LC_ALL,"") means that the environment is examined to set the locale. The application must still decide if failure is fatal or not.

> The problem is that the apps that DO start, still write the unreadable
> chars. This presumably comes from the fact that setlocale(LC_ALL,"")
> fails, but they ignore the return value.

Yes. If you used C.UTF-8 you'd always be able to write *any* characters.

> I wonder if someone ever checks the return value of setlocale(LC_ALL,"")
> at all. So maybe at least this part of the problem can be fixed on a
> glibc level?

The only fix at the glibc level is to provide C.UTF-8 for application usage and to make it a standard for all distributions such that upstream can rely on it.

If you find a distribution that *doesn't* have C.UTF-8, please contact me and I'll work with the liason for that distribution in upstream glibc to discuss options and solutions.
Comment 19 Mike FABIAN 2016-03-01 11:27:48 EST
(In reply to Stas Sergeev from comment #17)
> > No. It's the responsibility of the application to do this, because only the
> > application knows if a translation is critical, particuarly when UTF-8 support
> > is needed.
> Could you please confirm that you mean exactly setlocale(LC_ALL,"")?
> I thought if an apps does this, it implies that the translation
> is not critical. Please note that the problem is not only that
> the apps do not start. I suppose the ones that do not start, are
> indeed locale-sensitive, and they do not do setlocale(LC_ALL,"")
> at first place (maybe they do setlocale(LC_ALL,"C.UTF-8") or something -
> then they should be fixed the way you suggest).

I think you misunderstand what setlocale(LC_ALL,"") means.

man setlocale> If locale  is an empty string,  "", each part of  the locale
man setlocale> that should be modified is  set according to the environment
man setlocale> variables.

So if an app uses setlocale(LC_ALL, ""), it does not mean at all that
it does not want to use translations. It sets the locales according ot
the environment variables, which contain the locales the user wants to
use, including the locale the use wants to use for the translation.

setlocale(LC_ALL,"C.UTF-8") does *not* use the environment variables
but just sets the C.UTF-8 locale. The user will not get translations
in that case.
Comment 20 Stas Sergeev 2016-03-01 11:29:05 EST
> Calling setlocale(LC_ALL,"") simply means that the application has been tested
> with and expects to work with *all* locales the user might possible set via
> environment variables.
Yes, so since it can work with all and any locale
(even "C" or "C.UTF-8"), then I don't see a big problem
if such locale is forced to it as a fall-back. As you say,
it can work with anything, so why would it care? Why would
it break on such a fall-back? So that's the logic I'd like
to put for considerations.
Thanks for your explanations!
Comment 21 Stas Sergeev 2016-03-01 11:30:56 EST
> So if an app uses setlocale(LC_ALL, ""), it does not mean at all that
> it does not want to use translations.
No, I didn't mean that either.
Its just that it can work with anything user wants.
And since it can work with anything, it looks exactly
like the right place for the fall-back. Is it not? :)
Anyway, that's just an idea.
Comment 22 Carlos O'Donell 2016-03-01 13:25:32 EST
(In reply to Stas Sergeev from comment #21)
> > So if an app uses setlocale(LC_ALL, ""), it does not mean at all that
> > it does not want to use translations.
> No, I didn't mean that either.
> Its just that it can work with anything user wants.
> And since it can work with anything, it looks exactly
> like the right place for the fall-back. Is it not? :)
> Anyway, that's just an idea.

Except that glibc is the wrong place to decide this and it would violate the specification of the API. Only the application knows if a fallback is OK if the user's selected locale is missing. There is too much application specific knowledge required to make that decision. So we leave it up to the application:
...
/* Try to load the locale the user wanted.  */
char old_locale;
if ((old_locale = setlocale (LC_ALL, "")) == NULL)
  {
    /* Given that we are mostly a UI app, we'll ignore
       the failure, but we'll add an icon in the UI to
       warn the user that their locale settings are wrong.
       However, because we might need to dispaly all sorts
       of characters, we want UTF-8 support, so try C.UTF-8.  */
    failed_locale_load = true;
    if ((old_locale = setlocale (LC_ALL, "C.UTF-8")) == NULL)
      {
        /* Error!  */
        ...
      } 
  }
/* Either we set the user's locale or we set C.UTF-8.  */
...
Comment 23 Stas Sergeev 2016-03-01 15:08:37 EST
OK, how about the simpler fix: the default
locale can be "C.UTF-8" instead of "C".
Then the fall-back is not needed and the app still
has the control over the wrongly set locale. But if
setlocale() fails, the "C.UTF-8" locale will remain.
Can this work?

> if ((old_locale = setlocale (LC_ALL, "")) == NULL)
>   {
>     if ((old_locale = setlocale (LC_ALL, "C.UTF-8")) == NULL)
This work-around may be reasonable for an apps that
want to handle the wrongly set locale, but 99% of them
do not want to deal with that. I maintain at least 2
apps that do setlocale(LC_ALL, "") without checking
an error, and I would object to calling them "poorly
written and not robust" as this method was very common
and widely used - all until yesterday.
Lets just find a good compromise, I am sure it exists somewhere. :)
Comment 24 Stas Sergeev 2016-03-01 16:33:57 EST
Or if the default can't be changed
that easily, how about adding an env var that
specifies the default locale?
In this case all apps that do not call setlocale(),
will work with whatever this var contains, or "C" if
it is not set. The same will be true if setlocale()
fails, so the problem will be solved.
Comment 25 Carlos O'Donell 2016-03-02 07:04:11 EST
(In reply to Stas Sergeev from comment #23)
> OK, how about the simpler fix: the default
> locale can be "C.UTF-8" instead of "C".
> Then the fall-back is not needed and the app still
> has the control over the wrongly set locale. But if
> setlocale() fails, the "C.UTF-8" locale will remain.
> Can this work?

That can work. We would need to discuss this upstream. Essentially expanding C to support UTF-8 transparently (since C is a subset of C.UTF-8). It would be hard to argue a case where you don't want UTF-8 support.

> This work-around may be reasonable for an apps that
> want to handle the wrongly set locale, but 99% of them
> do not want to deal with that. I maintain at least 2
> apps that do setlocale(LC_ALL, "") without checking
> an error, and I would object to calling them "poorly
> written and not robust" as this method was very common
> and widely used - all until yesterday.

That's the exact definition of non-robust e.g. unable to adapt to allowed system changes (removal of locales).

> Lets just find a good compromise, I am sure it exists somewhere. :)

Agreed.
Comment 26 Carlos O'Donell 2016-03-02 07:07:59 EST
(In reply to Carlos O'Donell from comment #25)
> (In reply to Stas Sergeev from comment #23)
> > OK, how about the simpler fix: the default
> > locale can be "C.UTF-8" instead of "C".
> > Then the fall-back is not needed and the app still
> > has the control over the wrongly set locale. But if
> > setlocale() fails, the "C.UTF-8" locale will remain.
> > Can this work?
> 
> That can work. We would need to discuss this upstream. Essentially expanding
> C to support UTF-8 transparently (since C is a subset of C.UTF-8). It would
> be hard to argue a case where you don't want UTF-8 support.
>

We can fix this immediately in Fedora Rawhide and experiment a bit with it.
https://bugzilla.redhat.com/show_bug.cgi?id=1313818
Comment 27 Stas Sergeev 2016-03-02 07:14:47 EST
(In reply to Carlos O'Donell from comment #26)
> We can fix this immediately in Fedora Rawhide and experiment a bit with it.
> https://bugzilla.redhat.com/show_bug.cgi?id=1313818
Thanks!
I didn't even went for other work-arounds and have
the system still broken, in a hope for something like
this to happen. :) I'd be glad to test that fix on f24.

I am not sure though why you write this:
---
* Change setlocale code to try C.UTF-8 loading before falling back to the builtin C/POSIX locales.
---
What fix do you have in mind when saying that?
The fix I mentioned, didn't involve setlocale code,
because I proposed to have "C.UTF-8" by default, i.e.
when setlocale() is not even called. So you probably
went for something else after all?

Note You need to log in before you can comment on or make changes to this bug.