Bug 156477

Summary: RFE: option to reduce disk usage of /usr/lib/locale/locale-archive by only installing required language
Product: [Fedora] Fedora Reporter: Ricky Ng-Adam <rngadam>
Component: glibcAssignee: Carlos O'Donell <codonell>
Status: CLOSED NEXTRELEASE QA Contact: Mike McLean <mikem>
Severity: medium Docs Contact:
Priority: medium    
Version: rawhideCC: ascrane, codonell, dshea, dwalsh, edgar.hoch, ernest.beinrohr, fvzwieten, fweimer, goemon, i18n-bugs, ichavero, jakub, jeder, jfilak, jpazdziora, jperrin, lars, law, mattdm, mfabian, msuchy, petersen, pfrankli, pmatilai, pnemade, psatpute, rjones, rmarko, rvokal, sandro, sardella, self, sjensen, spoyarek, wnefal+redhatbugzilla
Target Milestone: ---Keywords: i18n, Reopened
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: anaconda-21.25-1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-03-26 19:07:47 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
0001-Change-build-locale-archive.c-to-support-an-option-i.patch none

Description Ricky Ng-Adam 2005-05-01 00:22:25 UTC
Description of problem:

Steps here seem to lead to a significant reduction in memory usage (30 some MB...)

http://lists.ssc.com/pipermail/linux-list/2004-December/022665.html

   First make a backup of /usr/lib/locale directory:

   # cd /usr/lib
   # tar czf locale.tgz ./locale

   Now go into the locale directory, and delete
   locale-archive file as well as all locales
   that you don't need (that's all directories
   other than en_US.* for you US-centric folks :)

   # cd locale
   # rm locale-archive
   ... remove unused locale directories here ...

   Now rebuild the locale-archive file:

   # build-locale-archive

Version-Release number of selected component (if applicable):

glibc-common-2.3.5-1

How reproducible:

always

Steps to Reproduce:
1. install system with support for two or three languages
2.
3.
  
Actual results:

heavy locale-archive with ALL languages in it loaded ...

Expected results:

only minimal languages required installed...

Additional info:

Comment 1 Jakub Jelinek 2005-05-02 08:38:52 UTC
And the bug is?

The file is mmaped by programs and only the header and locales you really
use are accessed from it, therefore having all the locales in the archive
is not a problem.

Furthermore, if you want, you can tweak rpm config files and override the default
%_install_langs all
with a list of langs you really use.  Then when you install glibc-common next
time, only the /usr/lib/locale/* files that are requested will be installed
and locale archive will be built only from those.

Comment 2 Ricky Ng-Adam 2005-05-03 13:22:12 UTC
s/memory/disk usage/g ...

Why are all these installed when only one, two or three languages are needed and
specified at install? Yes, I can "tweak" stuff, but I'd expect this to be done
automatically. Shouldn't %_install_langs initially be set in relation with what
was selected during install or by kickstart?

If I install GNOME or KDE or any of their applications, do I get all the
languages with it?  If so, it would seems to be wasteful...   

Comment 3 Jakub Jelinek 2005-05-03 13:27:45 UTC
%_install_langs used to be set by the installer in the past, but the gains
of using it (minor disk space savings) were smaller than the disadvantages
(e.g. when you install with some %_install_langs, then after install find
out you need X, Y and Z locales as well, you suddenly need to reinstall all the
packages that include those language files).
Anyway, this has nothing to do with glibc, it is installer's decision.
So if you want to argue about this, file a bug against anaconda and argue there.

Comment 4 Ricky Ng-Adam 2005-05-03 15:01:25 UTC
Would it be possible to reopen this one and reassign to component anaconda?

Comment 5 Jakub Jelinek 2005-05-03 15:02:24 UTC
Sure.

Comment 6 Jeremy Katz 2005-05-03 15:28:14 UTC
Not going to change the behavior back for the arguments Jakub detailed above. 

Comment 7 Ricky Ng-Adam 2005-09-25 19:22:41 UTC
I'd just like to note that as a comparison, under Ubuntu Linux (and probably
most Debian distros), locale-archive is 1.8M, even with a few non-english
languages added at install and post-install time (Chinese, French). So it would
seem that it is possible to keep this file at a reasonable size.

Comment 8 Dan Hollis 2006-01-17 04:18:19 UTC
50mb makes it that much harder to install FC into small systems (eg SD,
compactflash, etc).

Comment 9 Matthew Miller 2012-11-13 04:49:35 UTC
And, seven years later, cloud.

Comment 10 Fred van Zwieten 2012-11-23 07:28:10 UTC
Not sure how I can re-open this. I think it is important to be able to have a real small footprint. I need Fedora minimals in the cloud and this is costing me. Also, spins on ARM based devices must be able to become very minimal.

Comment 11 Matthew Miller 2012-11-26 13:50:41 UTC
I think it's probably best left closed as we're wrapping up F18 but maybe reopened after that. I can think of two possibilities:

1. kickstart option respected by anaconda (and pykickstart) for easier cloud images
2. a more grand scheme to make lang subpackages automatically a la debuginfo, and related distro infrastructure for handling those.

Or, some other magical solution might be devised.

Comment 12 Brendan Long 2013-08-15 22:36:56 UTC
I just want to point out, this folder is > 500 MB now. That's pretty significant..

Comment 13 Matthew Miller 2013-09-23 20:51:11 UTC
This is highly desirable for the case of creating lightweight containers. And in turn, creating lightweight containers with anaconda is desirable because _we already have enough tools_ and I don't want yet another one.

Comment 14 Bill Nottingham 2013-12-04 14:26:21 UTC
Is it better to just solve this via packaging? Package glibc-common requiring some minimal locale set (C, en_US.UTF-8) that builds a locale archive, and then a glibc-alllocales that packages the rest that is included in most installs?

Comment 15 Bill Nottingham 2013-12-04 14:28:24 UTC
CC'ing Jakub for the above - this is obviously a bit of a hack that would avoid touching anaconda/rpm.

Comment 16 Jakub Jelinek 2013-12-04 14:35:01 UTC
I must say I already forgot how exactly it is built, so I'll defer it to our current glibc maintainers.  That said, you need to test also the cases where users install (into locale-archive) some custom locales, and that needs to be dealt properly during upgrades etc.

Comment 17 Bill Nottingham 2013-12-04 14:41:34 UTC
CC'ing Carlos, sorry about that Jakub.

Comment 18 David Shea 2014-02-24 22:43:43 UTC
*** Bug 1051816 has been marked as a duplicate of this bug. ***

Comment 19 David Shea 2014-02-24 22:45:09 UTC
%packages --instLangs in kickstart is supposed to provide this. I guess we should implement it!

Comment 20 Carlos O'Donell 2014-02-25 16:29:46 UTC
If glibc can help in any way, please tell me. What kind of interface do you need? We really need someone to sit down and think through a coherent distribution story for handling localization in a dynamic way that doesn't consume so much disk space for cloud/vm/containers etc.

Comment 21 Matthew Miller 2014-02-25 21:25:39 UTC
(In reply to Carlos O'Donell from comment #20)
> If glibc can help in any way, please tell me. What kind of interface do you
> need? We really need someone to sit down and think through a coherent
> distribution story for handling localization in a dynamic way that doesn't
> consume so much disk space for cloud/vm/containers etc.

That sounds awesome -- but maybe it should be tracked elsewhere? This is pretty simple as it is, even if not very elegant.

Comment 22 Bill Nottingham 2014-02-25 21:50:04 UTC
%packages --instLangs doesn't solve the "I would like the locales back after installing", does it? (At least, not without 'yum upgrade/yum reinstall').

Comment 23 David Shea 2014-02-25 21:53:54 UTC
(In reply to Bill Nottingham from comment #22)
> %packages --instLangs doesn't solve the "I would like the locales back after
> installing", does it? (At least, not without 'yum upgrade/yum reinstall').

If you want the locales put back without reinstalling then how are you going to remove them in the first place? I thought that was what this bug was about!

Comment 24 Matthew Miller 2014-02-25 22:15:04 UTC
(In reply to David Shea from comment #23)
> If you want the locales put back without reinstalling then how are you going
> to remove them in the first place? I thought that was what this bug was
> about!

We want to make a base image without the languages installed, but it would be nice if people could easily add them back after starting from that base.

Bill is right that the current state of the art in doing this is to change the RPM lang options and then do yum reinstall. But one step at a time.

Comment 25 David Shea 2014-02-27 22:36:48 UTC
Implemented the part that anaconda can do, which doesn't actually change the size of /usr/lib/locale/locale-archive, but that's out of our hands for now. And because I care, %packages --excludedocs now works again, too. (PS: %packages --excludedocs wasn't working)

Comment 26 Daniel Walsh 2014-03-24 20:35:15 UTC
Then David, I take it that glibc is not following the rules set by anaconda?  

I will reassign to them.

We have gotten a container image with apache down to 200 Meg, but 50% of the image is now local-archive.

Comment 27 Carlos O'Donell 2014-03-24 20:41:32 UTC
(In reply to Daniel Walsh from comment #26)
> Then David, I take it that glibc is not following the rules set by anaconda?
> I will reassign to them.

I'm happy to do that, can you provide me a link (mojo, upstream wiki) to any documentation that outlines what I have to do?

> We have gotten a container image with apache down to 200 Meg, but 50% of the
> image is now local-archive.

We can still do better. The glibc team is happy to help.

Comment 28 Daniel Walsh 2014-03-24 20:43:55 UTC
This little hack from the internet seems to  shrink it.

localedef --delete-from-archive $(localedef --list-archive | grep -v -i ^$LANG | xargs)
mv /usr/lib/locale/locale-archive /tmp/
build-locale-archive

Comment 29 David Shea 2014-03-24 20:46:55 UTC
(In reply to Carlos O'Donell from comment #27)
> (In reply to Daniel Walsh from comment #26)
> > Then David, I take it that glibc is not following the rules set by anaconda?
> > I will reassign to them.
> 
> I'm happy to do that, can you provide me a link (mojo, upstream wiki) to any
> documentation that outlines what I have to do?

From the anaconda side, what we're doing is copying the --instLangs setting in kickstart to the macro _install_langs in the rpm transaction. I don't know if the rpm macro is something that you'd be able to use, but if there is a way to make what goes into locale-archive configurable at install time we can hook that up to instLangs in anaconda.

Comment 30 Daniel Walsh 2014-03-24 20:54:35 UTC
Fixed.

localedef --delete-from-archive $(localedef --list-archive | grep -v -i ^$LANG | xargs)
mv /usr/lib/locale/locale-archive  /usr/lib/locale/locale-archive.tmpl
build-locale-archive

Of course this only works for a single language.

Comment 31 Jens Petersen 2014-05-19 00:56:13 UTC
+1 to subpackaging the locales so that users can install only (or remove)
the locales they (don't) need.  With the increasing focus on Virt
and Cloud I think this is now a priority and also one of the advantages
that Ubuntu still has over Fedora currently.

Comment 32 Jim Perrin 2014-07-16 16:56:23 UTC
This is impacting us for docker container size as well.

Comment 33 Matthew Miller 2014-11-08 17:40:42 UTC
So, with the F21 cloud base image, the locale archive file is _literally_ one quarter of the space on disk.

Comment 34 Carlos O'Donell 2014-11-10 02:59:08 UTC
(In reply to Matthew Miller from comment #33)
> So, with the F21 cloud base image, the locale archive file is _literally_
> one quarter of the space on disk.

Heard loud and clear. I have a plan to reduce the image size, but I'm not going to get to it for a couple of months. In the mean time the cloud image creation can always run a post-processing step to remove locales, it just won't be the default.

Comment 35 Matthew Miller 2014-11-10 14:11:21 UTC
Thanks Carlos. We'll do the hack in the short term. (Any suggestions for best possible way?)

"A couple of months" actually sounds very positive for targeting a real solution for Fedora 22. We're targeting a spring release, so if we could have something testable by February/March, that would work out well.

Comment 36 Mike FABIAN 2015-02-11 07:49:22 UTC
Created attachment 990365 [details]
0001-Change-build-locale-archive.c-to-support-an-option-i.patch

Patch against current master of our glibc package.

Comment 37 Mike FABIAN 2015-02-11 07:54:08 UTC
The patch in comment#36 adds an option --install-langs to
build-locale-archive.c and uses this option in the %post an %triggerin
scripts in glibc.rpm. The rpm macro %_install_langs is give as an
argument to --install-langs.

I tested this on Fedora 21:

[mfabian@localhost ~]$ sudo rpm -Uhv -D '%_install_langs ko:ja:de_DE:en_US.utf8' glibc-*x86_64.rpm nscd-2.20.90-20.fc21.x86_64.rpm --force
準備しています...              ################################# [100%]
更新中 / インストール中...
   1:glibc-common-2.20.90-20.fc21     ################################# [ 11%]
   2:glibc-2.20.90-20.fc21            ################################# [ 22%]
   3:glibc-headers-2.20.90-20.fc21    ################################# [ 33%]
   4:glibc-devel-2.20.90-20.fc21      ################################# [ 44%]
   5:glibc-debuginfo-common-2.20.90-20################################# [ 56%]
   6:glibc-debuginfo-2.20.90-20.fc21  ################################# [ 67%]
   7:glibc-static-2.20.90-20.fc21     ################################# [ 78%]
   8:glibc-utils-2.20.90-20.fc21      ################################# [ 89%]
   9:nscd-2.20.90-20.fc21             ################################# [100%]
[mfabian@localhost ~]$ locale -a 
C
POSIX
de_DE
de_DE.iso88591
de_DE.iso885915@euro
de_DE.utf8
de_DE@euro
deutsch
en_US.utf8
german
ja_JP
ja_JP.eucjp
ja_JP.ujis
ja_JP.utf8
japanese
japanese.euc
ko_KR
ko_KR.euckr
ko_KR.utf8
korean
korean.euc
[mfabian@localhost ~]$ cat /etc/redhat-release 
Fedora release 21 (Twenty One)
[mfabian@localhost ~]$

Comment 38 Mike FABIAN 2015-02-11 07:59:14 UTC
Including only en_US.utf8 makes locale-archive 1.6M big, including all en_US
locales 2.1M, including all English locales 3.4M:

[mfabian@localhost ~]$ sudo rpm -Uhv -D '%_install_langs en_US' glibc-common*x86_64.rpm --force
Preparing...                          ################################# [100%]
Updating / installing...
   1:glibc-common-2.20.90-20.fc21     ################################# [100%]
[mfabian@localhost ~]$ locale -a 
C
POSIX
en_US
en_US.iso88591
en_US.iso885915
en_US.utf8
[mfabian@localhost ~]$ ls -lh /usr/lib/locale/locale-archive
-rw-r--r--. 1 root root 2.1M Dec 25 19:18 /usr/lib/locale/locale-archive
[mfabian@localhost ~]$ sudo rpm -Uhv -D '%_install_langs en_US.utf8' glibc-common*x86_64.rpm --force
Preparing...                          ################################# [100%]
Updating / installing...
   1:glibc-common-2.20.90-20.fc21     ################################# [100%]
[mfabian@localhost ~]$ locale -a 
C
POSIX
en_US.utf8
[mfabian@localhost ~]$ ls -lh /usr/lib/locale/locale-archive
-rw-r--r--. 1 root root 1.6M Dec 25 19:18 /usr/lib/locale/locale-archive
[mfabian@localhost ~]$ sudo rpm -Uhv -D '%_install_langs en' glibc-common*x86_64.rpm --force
Preparing...                          ################################# [100%]
Updating / installing...
   1:glibc-common-2.20.90-20.fc21     ################################# [100%]
[mfabian@localhost ~]$ locale -a 
C
POSIX
en_AG
en_AG.utf8
en_AU
en_AU.iso88591
en_AU.utf8
en_BW
en_BW.iso88591
en_BW.utf8
en_CA
en_CA.iso88591
en_CA.utf8
en_DK
en_DK.iso88591
en_DK.utf8
en_GB
en_GB.iso88591
en_GB.iso885915
en_GB.utf8
en_HK
en_HK.iso88591
en_HK.utf8
en_IE
en_IE.iso88591
en_IE.iso885915@euro
en_IE.utf8
en_IE@euro
en_IN
en_IN.utf8
en_NG
en_NG.utf8
en_NZ
en_NZ.iso88591
en_NZ.utf8
en_PH
en_PH.iso88591
en_PH.utf8
en_SG
en_SG.iso88591
en_SG.utf8
en_US
en_US.iso88591
en_US.iso885915
en_US.utf8
en_ZA
en_ZA.iso88591
en_ZA.utf8
en_ZM
en_ZM.utf8
en_ZW
en_ZW.iso88591
en_ZW.utf8
[mfabian@localhost ~]$ ls -lh /usr/lib/locale/locale-archive
-rw-r--r--. 1 root root 3.4M Dec 25 19:19 /usr/lib/locale/locale-archive
[mfabian@localhost ~]$

Comment 39 Daniel Walsh 2015-02-15 13:51:17 UTC
Awesome

Comment 40 Carlos O'Donell 2015-02-17 15:18:48 UTC
(In reply to Mike FABIAN from comment #38)
> Including only en_US.utf8 makes locale-archive 1.6M big, including all en_US
> locales 2.1M, including all English locales 3.4M:

Please post to glibc.org for inclusion into Rawhide. I'll review and we'll get it committed. Thanks again for all the hard work!

Comment 41 Carlos O'Donell 2015-03-05 21:12:01 UTC
Committed!

I just finished testing on Rawhide with a /usr/lib/rpm/macros that defines `%_install_langs en` and a full upgrade works as expected and reduces the set of installed languages to en_* and only a ~3.5MB /usr/lib/locale/locale-archive.

This enhancement means that with an edited /usr/lib/rpm/macros, or an installer setting %_install_langs you'll have a minimized set of installed locales.

Maintaining the minimum set of installed locales requires modifying /usr/lib/rpm/macros `%_install_langs`.

Reinstalling the missing locales you want requires reinstalling glibc-common with an updated value for `%_install_langs`.

The next step is going to be to split up the locales into sub-packages and deal with that part-by-part, but the immediate need for minimal cloud, server, or docker images is now met.

Comment 42 Edgar Hoch 2015-03-05 21:55:20 UTC
(In reply to Carlos O'Donell from comment #41)
> Maintaining the minimum set of installed locales requires modifying
> /usr/lib/rpm/macros `%_install_langs`.
> 
> Reinstalling the missing locales you want requires reinstalling glibc-common
> with an updated value for `%_install_langs`.

Wouldn't it be better to put a file in /etc/rpm/ which sets (overwrites) the value of `%_install_langs` instead of modifying the global config file /usr/lib/rpm/macros, _and_ requiring reinstalling glibc-common?

I think that files in /usr/ should never contain a host-specific configuration.

Comment 43 Carlos O'Donell 2015-03-05 22:03:26 UTC
(In reply to Edgar Hoch from comment #42)
> (In reply to Carlos O'Donell from comment #41)
> > Maintaining the minimum set of installed locales requires modifying
> > /usr/lib/rpm/macros `%_install_langs`.
> > 
> > Reinstalling the missing locales you want requires reinstalling glibc-common
> > with an updated value for `%_install_langs`.
> 
> Wouldn't it be better to put a file in /etc/rpm/ which sets (overwrites) the
> value of `%_install_langs` instead of modifying the global config file
> /usr/lib/rpm/macros, _and_ requiring reinstalling glibc-common?
> 
> I think that files in /usr/ should never contain a host-specific
> configuration.

I agree, but that's a decision RPM has to make. glibc is simply following whatever value is provided by rpm in %_install_langs. File another bug?

Comment 44 Panu Matilainen 2015-03-09 11:02:46 UTC
Um, one should *never* modify /usr/lib/rpm/macros, that's a "factory config" setting and any changes will be overwritten on rpm upgrades. Local settings go into /etc/rpm files.

As for how I ended up in this bug... this change broke copr. I dont know what ancient OS it runs on, perhaps RHEL-6, but the rpm there doesn't support scriptlet expansion so glibc, and thus everything, is uninstallable now due to: 
DEBUG util.py:388:  ERROR You need to update rpm to handle:
DEBUG util.py:388:  rpmlib(ScriptletExpansion) <= 4.9.0-1 is needed by glibc-common-2.21.90-5.fc23.x86_64

Mostly this is really copr's headache, not yours, but in this case there's no actual need to use scriptlet expansion, because from lua you can always expand macros. Something like this (and the same for the triggerin) would avoid the rpmlib dependency without affecting the functionality:

-%post common -e -p <lua>
+%post common -p <lua>
 if posix.access("/etc/ld.so.cache") then
   if posix.stat("%{_prefix}/lib/locale/locale-archive.tmpl", "size") > 0 then
     pid = posix.fork()
     if pid == 0 then
-      posix.exec("%{_prefix}/sbin/build-locale-archive", "--install-langs", "%%{_install_langs}")
+      posix.exec("%{_prefix}/sbin/build-locale-archive", "--install-langs", rpm.expand("%%{_install_langs}"))
     elseif pid > 0 then
       posix.wait(pid)
     end

Comment 45 Carlos O'Donell 2015-03-09 13:18:21 UTC
(In reply to Panu Matilainen from comment #44)
> Um, one should *never* modify /usr/lib/rpm/macros, that's a "factory config"
> setting and any changes will be overwritten on rpm upgrades. Local settings
> go into /etc/rpm files.

Agreed. Which is why I asked Edgar Hoch to file a new bug for rpm if it wasn't already solvable. I expect it is simply by setting the macros in /etc/rpm.

> As for how I ended up in this bug... this change broke copr. I dont know
> what ancient OS it runs on, perhaps RHEL-6, but the rpm there doesn't
> support scriptlet expansion so glibc, and thus everything, is uninstallable
> now due to: 
> DEBUG util.py:388:  ERROR You need to update rpm to handle:
> DEBUG util.py:388:  rpmlib(ScriptletExpansion) <= 4.9.0-1 is needed by
> glibc-common-2.21.90-5.fc23.x86_64

I'm really sorry about that, that was an unforseen consquence, but the truth is that Rawhide will continue to move forward picking up features, either it happens now or it happens later.

> Mostly this is really copr's headache, not yours, but in this case there's
> no actual need to use scriptlet expansion, because from lua you can always
> expand macros. Something like this (and the same for the triggerin) would
> avoid the rpmlib dependency without affecting the functionality:
> 
> -%post common -e -p <lua>
> +%post common -p <lua>
>  if posix.access("/etc/ld.so.cache") then
>    if posix.stat("%{_prefix}/lib/locale/locale-archive.tmpl", "size") > 0
> then
>      pid = posix.fork()
>      if pid == 0 then
> -      posix.exec("%{_prefix}/sbin/build-locale-archive", "--install-langs",
> "%%{_install_langs}")
> +      posix.exec("%{_prefix}/sbin/build-locale-archive", "--install-langs",
> rpm.expand("%%{_install_langs}"))
>      elseif pid > 0 then
>        posix.wait(pid)
>      end

I believe we tried this before but it had some negative consequences. I'm going to ask Fabian to comment and test it out.

Mike,

Do you remember if we talked about using rpm.expand and then didn't?

Comment 46 Miroslav Suchý 2015-03-09 14:00:02 UTC
>As for how I ended up in this bug... this change broke copr. I dont know what
>ancient OS it runs on, perhaps RHEL-6

Yes. ancient EL6. We plan to move to Fedora21 in one month.

Comment 47 Edgar Hoch 2015-03-09 14:06:28 UTC
(In reply to Carlos O'Donell from comment #45)
> (In reply to Panu Matilainen from comment #44)
> > Um, one should *never* modify /usr/lib/rpm/macros, that's a "factory config"
> > setting and any changes will be overwritten on rpm upgrades. Local settings
> > go into /etc/rpm files.
> 
> Agreed. Which is why I asked Edgar Hoch to file a new bug for rpm if it
> wasn't already solvable. I expect it is simply by setting the macros in
> /etc/rpm.

I was thinking about the problem. I came to the result that if a package is installed the files in /usr/... should be always the same for the same package, on every system. The locale data file is not a configuration file, it is a data file. Data files should not change while they are installed.

I would never expect that I have to reinstall a package if I would need more locales (languages). It should be predefined what a package provides and what not.

I think a reasonable solution would be to provide only the minimum of locales with the basic glibc package and to create additional (dependent) packages for each locale (as the exist for other packages, for example libreoffice, aspell, etc.). This language packages can be handled in the usual way by yum, dnf, etc.

So I think the rpm *install script should not call build-locale-archive, but locale-archive should only contain the minimum locales and glibc should search for other locales in other language dependent locale files.

Comment 48 Carlos O'Donell 2015-03-09 14:16:53 UTC
(In reply to Edgar Hoch from comment #47)
> So I think the rpm *install script should not call build-locale-archive, but
> locale-archive should only contain the minimum locales and glibc should
> search for other locales in other language dependent locale files.

I general I agree.

However, if you read the context above you'll notice that this solution as presented today is a "workaround" for small cloud images and containers. It is not intended as a final solution.

The final solution includes splitting all locale data out of glibc-common into subpackages that include fixed locale data. That way you can install "English" or "German" locales and get all the en_* and de_* locales with their various character maps.

In such a final solution we would transition the system using a utility that looks at the value of %_install_langs and installs the required subpackages to match what you already have on your system.

Most users will never change any of this and have it remain "all" which means you continue to get all of the loales, and in the future, all of the locale subpakcages.

After the transition we would probably allow the user to use the utility to change their set of installed langauges, or simply get rid of the utility and have them manage the locale packages themselves. We don't know exactly what we're going to do here, and it's an open question.

Comment 49 Mike FABIAN 2015-03-09 21:49:07 UTC
(In reply to Carlos O'Donell from comment #45)
> (In reply to Panu Matilainen from comment #44)
> > -%post common -e -p <lua>
> > +%post common -p <lua>
> >  if posix.access("/etc/ld.so.cache") then
> >    if posix.stat("%{_prefix}/lib/locale/locale-archive.tmpl", "size") > 0
> > then
> >      pid = posix.fork()
> >      if pid == 0 then
> > -      posix.exec("%{_prefix}/sbin/build-locale-archive", "--install-langs",
> > "%%{_install_langs}")
> > +      posix.exec("%{_prefix}/sbin/build-locale-archive", "--install-langs",
> > rpm.expand("%%{_install_langs}"))
> >      elseif pid > 0 then
> >        posix.wait(pid)
> >      end
> 
> I believe we tried this before but it had some negative consequences. I'm
> going to ask Fabian to comment and test it out.
> 
> Mike,
> 
> Do you remember if we talked about using rpm.expand and then didn't?

We didn’t talk about rpm.expand.

Florian Weimer mentioned rpm.getconf:

    <fweimer> mfabian: I think you need to add something like:
                                                          [15年02月09日 14:11:35]
    <fweimer> posix.exec("/usr/sbin/build-locale-archive",
              rpm.getconf("_install_langs")) [15年02月09日 14:12:05]

But later said:

    <fweimer> mfabian: Use -e if it works.  I was mistaken, there is no way to
              expand RPM macros from the current transaction using Lua.
                                                          [15年02月09日 15:48:02]
    * fweimer sighs. [15年02月09日 15:48:12]

I can try whether rpm.expand does the job.

Comment 50 Carlos O'Donell 2015-03-09 22:35:19 UTC
(In reply to Mike FABIAN from comment #49)
> I can try whether rpm.expand does the job.

Mike, Would you give it a try and see? That way we can unblock the copr rawhide builds by removing the dependency on this apparently new feature. Otherwise we'll have broken rawhide builds for a month or more before the copr upgrade completes (to using F21 build hosts).

Comment 51 Panu Matilainen 2015-03-10 06:31:43 UTC
There's no such thing as rpm.getconf() and never was, but rpm.expand() goes back to at least RHEL-5 and does exactly what the name says - macro expansion. For the full list of rpm lua extensions see http://rpm.org/wiki/PackagerDocs/RpmLua

In the future if you need advise on rpm, I'd suggest asking the rpm maintainers directly to avoid having to guess.

Comment 52 Mike FABIAN 2015-03-10 10:50:34 UTC
(In reply to Carlos O'Donell from comment #50)
> (In reply to Mike FABIAN from comment #49)
> > I can try whether rpm.expand does the job.
> 
> Mike, Would you give it a try and see? That way we can unblock the copr
> rawhide builds by removing the dependency on this apparently new feature.
> Otherwise we'll have broken rawhide builds for a month or more before the
> copr upgrade completes (to using F21 build hosts).

Yes, this change works:

diff --git a/glibc.spec b/glibc.spec
index a845182..2c166ad 100644
--- a/glibc.spec
+++ b/glibc.spec
@@ -1579,22 +1579,22 @@ end
 
 %postun -p /sbin/ldconfig
 
-%triggerin common -e -p <lua> -- glibc
+%triggerin common -p <lua> -- glibc
 if posix.stat("%{_prefix}/lib/locale/locale-archive.tmpl", "size") > 0 then
   pid = posix.fork()
   if pid == 0 then
-    posix.exec("%{_prefix}/sbin/build-locale-archive", "--install-langs", "%%{_install_langs}")
+    posix.exec("%{_prefix}/sbin/build-locale-archive", "--install-langs", rpm.expand("%%{_install_langs}"))
   elseif pid > 0 then
     posix.wait(pid)
   end
 end
 
-%post common -e -p <lua>
+%post common -p <lua>
 if posix.access("/etc/ld.so.cache") then
   if posix.stat("%{_prefix}/lib/locale/locale-archive.tmpl", "size") > 0 then
     pid = posix.fork()
     if pid == 0 then
-      posix.exec("%{_prefix}/sbin/build-locale-archive", "--install-langs", "%%{_install_langs}")
+      posix.exec("%{_prefix}/sbin/build-locale-archive", "--install-langs", rpm.expand("%%{_install_langs}"))
     elseif pid > 0 then
       posix.wait(pid)
     end
lines 1-37/37 (END)

Comment 53 Miroslav Suchý 2015-03-10 11:11:19 UTC
>As for how I ended up in this bug... this change broke copr.

I just tried to use F21 for builder on Copr .and this bug appear even then.

Comment 54 Panu Matilainen 2015-03-10 11:20:24 UTC
(In reply to Miroslav Suchý from comment #53)
> >As for how I ended up in this bug... this change broke copr.
> 
> I just tried to use F21 for builder on Copr .and this bug appear even then.

I dont know what Copr architecture looks like, but the thing that populates the chroots by issuing yum install commands needs to have rpm >= 4.9 (so rhel-7 or any current fedora) beneath that yum for this to work.

Comment 55 Carlos O'Donell 2015-03-17 04:05:29 UTC
Testing a build without `-e` to eventually push to rawhide.

Comment 56 Carlos O'Donell 2015-03-17 05:56:37 UTC
Fixed. I've pushed a non-`-e` using version to Rawhide. This should fix the Copr builds.

Comment 57 Miroslav Suchý 2015-03-19 14:03:58 UTC
I can confirm that it indeed fixed Copr rawhide builds.

Comment 58 Ivan Chavero 2016-03-26 18:11:36 UTC
How about the /usr/share/i18n directory, is this something that could be reduced?
it's almost the size of 3 Alpine Linux containers

# du -h /usr/share/ | grep "[0-9]M" | grep i18n
3.3M    /usr/share/i18n/charmaps
6.3M    /usr/share/i18n/locales
9.5M    /usr/share/i18n

maybe ship this directory in a separate package?

Comment 59 Mike FABIAN 2016-03-26 19:06:35 UTC
(In reply to Ivan Chavero from comment #58)
> How about the /usr/share/i18n directory, is this something that could be
> reduced?
> it's almost the size of 3 Alpine Linux containers
> 
> # du -h /usr/share/ | grep "[0-9]M" | grep i18n
> 3.3M    /usr/share/i18n/charmaps
> 6.3M    /usr/share/i18n/locales
> 9.5M    /usr/share/i18n
> 
> maybe ship this directory in a separate package?

We have this in Fedora 24 and rawhide, see: 
https://bugzilla.redhat.com/show_bug.cgi?id=1238406

There are glibc-langpack-.. packages now for each language containing
folders in /usr/lib/locale/ for that language,
one glibc-all-langpacks package which contains all locales in a single
package using the /usr/lib/locale/locale-archive database
and a package glibc-locale-source which contains the /usr/share/i18n directory.