When upgrading a system with glibc-all-langpacks installed from Fedora 29 or Fedora 30 to Fedora Rawhide, the file /usr/lib/locale/locale-archive somehow disappears entirely. To reproduce, simply install Fedora 29 or Fedora 30, ensure glibc-all-langpacks is installed, and upgrade to Fedora Rawhide (using dnf system-upgrade). Then run 'rpm -V glibc-all-langpacks' and 'ls -l /usr/lib/locale/locale-archive.tmpl'. These show that the file is missing. This seems to be the cause of https://bugzilla.redhat.com/show_bug.cgi?id=1716691 .
(In reply to Adam Williamson from comment #0) > When upgrading a system with glibc-all-langpacks installed from Fedora 29 or > Fedora 30 to Fedora Rawhide, the file /usr/lib/locale/locale-archive somehow > disappears entirely. > > To reproduce, simply install Fedora 29 or Fedora 30, ensure > glibc-all-langpacks is installed, and upgrade to Fedora Rawhide (using dnf > system-upgrade). Then run 'rpm -V glibc-all-langpacks' and 'ls -l > /usr/lib/locale/locale-archive.tmpl'. These show that the file is missing. > > This seems to be the cause of > https://bugzilla.redhat.com/show_bug.cgi?id=1716691 . Thanks for filing this Adam. As I noted here: https://bugzilla.redhat.com/show_bug.cgi?id=1716691#c22 This is probably a transitional issue. I think we're going to have to find a way to transition between the old and new package. Perhaps by releasing changes in the current active Fedora branches that allow the upgrade to happen smoothly.
Oh. Hey. I bet I know what this is. Before https://src.fedoraproject.org/rpms/glibc/c/50bcae98df7d383733d4d6d896b9e1e829914eb0?branch=master , this was glibc-all-langpacks %postun: %postun all-langpacks -p <lua> -- In the postun we always remove the locale cache. -- We are being uninstalled and if this is an upgrade -- then the new packages template will be used to -- recreate a new copy of the cache. os.remove("%{_prefix}/lib/locale/locale-archive") now let's remind ourselves of RPM scriptlet ordering: https://docs.fedoraproject.org/en-US/packaging-guidelines/Scriptlets/ so when we update from an old-style glibc-all-langpacks (e.g. the ones in F29 or F30) to the new-style one in Rawhide, we do %pretrans, the new package %pre, then we install the new package (with its /usr/lib/locale/locale-archive file), then we do the new package's %post, then the old package's %preun, then we remove the old package...then we do the old package's %postun, which deletes /usr/lib/locale/locale-archive . Whoops. Off the top of my head I dunno how we should fix this, but that's what I reckon the problem is.
Proposing as a Beta blocker, as I'd be pretty surprised if all locales suddenly disappearing during an upgrade didn't cause lots of problems...
I've also see a variant of this which results in no new languages being brought in. Upgrading : glibc-all-langpacks-2.29.9000-23.fc31.x86_64 1/38 warning: /usr/lib/locale/locale-archive saved as /usr/lib/locale/locale-archive.rpmsave
(In reply to Adam Williamson from comment #4) > Proposing as a Beta blocker, as I'd be pretty surprised if all locales > suddenly disappearing during an upgrade didn't cause lots of problems... Agreed, we'll have this sorted before then.
I saw that line in the logs for my case also (and that .rpmsave file really is created and is there on the system when the upgrade completes). I think it's basically rpm getting confused by you changing the file from %ghost %config to...you know...just a regular file, thinking it's going away, and 'saving' it. You normally get that message when you have a package with a %config file, and you upgrade to a newer version of the package with that file removed - rpm saves the file just in case you might've needed anything from it. I don't think *that* particular mechanism kicking in would cause any problems (besides this unnecessary .rpmsave file wasting a couple hundred megs of disk space), but hey, IMBW.
Actually, thinking it through a bit more, it might be entirely intentional/correct that this mechanism kicks in when you change a file from %config to a regular file. After all, that means the file will now get overwritten; if it was a config file, it's reasonable to think it may have some local changes that the user might still need, and so it should not be thrown away entirely but the rpmsave backup created.
I think we should just patch glibc to use a different path for the locale archive. It's not a public interface because the format is undocumented and internal to glibc. Carlos, what do you think?
(In reply to Florian Weimer from comment #9) > I think we should just patch glibc to use a different path for the locale > archive. It's not a public interface because the format is undocumented and > internal to glibc. > > Carlos, what do you think? This is one of the simplest options. No user should be depending on the location of the locale archive. I suggest we keep it in the same path but rename the name of the file e.g. locale-archive.2, something obviously the same as the locale-archive but marked as being "the second" and the tooling will use this new named file. Then configure the glibc build to use the new alternative name (we just need to plumb that through). I've looked at altering the existing installs to honour this and we introduce more complexity than a simple file rename. For example, who deletes the old locale-archive? When you remove glibc-all-langpacks the %ghost'd locale-archive is removed vi os.remove in %post. If we don't do this then the file stays around and that's wrong. So we we would need a marker to indicate "new style" locale-archive and avoid the removal in that case, this marker might as well be the name of the file. This is already much more complicated than changing the name to locale_archive. I support a name change and I propose "locale-archive.2" and we can bump to .3 or .4 in the future if we need to change format or semantics.
(In reply to Carlos O'Donell from comment #10) > I support a name change and I propose "locale-archive.2" and we can bump to > .3 or .4 in the future if we need to change format or semantics. I'm testing a build which just uses locale-archive.%{version}-%{release}, so that we don't have to maintain that number. This will also help with format changes during upgrades. It's not nice for statically linked, localized binaries, but we hopefully do not have many of those. They would also have been affected by the lack of format compatibility for the locale data anyway.
(In reply to Florian Weimer from comment #11) > (In reply to Carlos O'Donell from comment #10) > > I support a name change and I propose "locale-archive.2" and we can bump to > > .3 or .4 in the future if we need to change format or semantics. > > I'm testing a build which just uses locale-archive.%{version}-%{release}, so > that we don't have to maintain that number. This will also help with format > changes during upgrades. Looks good to me. > It's not nice for statically linked, localized binaries, but we hopefully do > not have many of those. They would also have been affected by the lack of > format compatibility for the locale data anyway. Correct. We have always said that static binaries may not be able to load locales because of format changes. We may need to do a better job versioning that data to avoid problems.
Patch posted upstream: https://sourceware.org/ml/libc-alpha/2019-06/msg00038.html
Adam, would you please test this again if you can? The build has just finished. Thanks.
It'll get tested automatically with the next Rawhide compose, if I get around to doing it manually before that I'll let you know.
Upstream pointed out that the patch reduces compatibility of statically linked binaries. We will need a different solution for this.
*** Bug 1718785 has been marked as a duplicate of this bug. ***
glibc-2.29.9000-26.fc31 reverts the original patch and now simply restores the deleted file on %posttrans, by shipping the contents as a hard-linked file. The actual bug fix is in the old glibc-all-langpacks, see bug 1717367. We will keep the workaround now in rawhide for a couple of Fedora releases.
I tried -26 release. Locales seems to be back but I still issues sh$ rpm -qa glibc \*langpack\* glibc-all-langpacks-2.29.9000-26.fc31.x86_64 glibc-2.29.9000-26.fc31.x86_64 libreoffice-langpack-en-6.2.3.2-5.fc31.x86_64 sh$ locale -a | wc -l 865 sh-5.0$ locale LANG=C.utf8 LC_CTYPE="C.utf8" LC_NUMERIC="C.utf8" LC_TIME="C.utf8" LC_COLLATE="C.utf8" LC_MONETARY="C.utf8" LC_MESSAGES="C.utf8" LC_PAPER="C.utf8" LC_NAME="C.utf8" LC_ADDRESS="C.utf8" LC_TELEPHONE="C.utf8" LC_MEASUREMENT="C.utf8" LC_IDENTIFICATION="C.utf8" LC_ALL= sh-5.0$ rpm -q --changelog openssl | head * Mon Jun 03 2019 Tomáš Mráz <tmraz> 1.1.1c-3 - add upstream patch to defer sending KeyUpdate after pending writes are complete * Thu May 30 2019 Tomáš Mráz <tmraz> 1.1.1c-2 - fix use of uninitialized memory * Wed May 29 2019 Tomáš Mráz <tmraz> 1.1.1c-1 - update to the 1.1.1c release
(In reply to Lukas Slebodnik from comment #19) > I tried -26 release. Locales seems to be back but I still issues Did you reboot or otherwise restart your terminal?
(In reply to Florian Weimer from comment #20) > (In reply to Lukas Slebodnik from comment #19) > > I tried -26 release. Locales seems to be back but I still issues > > Did you reboot or otherwise restart your terminal? I rebooted machine to be sure. But it seems to be some problem with xterm because xfce4-terminal works well.
(In reply to Lukas Slebodnik from comment #21) > (In reply to Florian Weimer from comment #20) > > (In reply to Lukas Slebodnik from comment #19) > > > I tried -26 release. Locales seems to be back but I still issues > > > > Did you reboot or otherwise restart your terminal? > > I rebooted machine to be sure. > > But it seems to be some problem with xterm because xfce4-terminal works well. Right, it's probably an unrelated issue.
(In reply to Florian Weimer from comment #22) > (In reply to Lukas Slebodnik from comment #21) > > (In reply to Florian Weimer from comment #20) > > > (In reply to Lukas Slebodnik from comment #19) > > > > I tried -26 release. Locales seems to be back but I still issues > > > > > > Did you reboot or otherwise restart your terminal? > > > > I rebooted machine to be sure. > > > > But it seems to be some problem with xterm because xfce4-terminal works well. > > Right, it's probably an unrelated issue. And it might be partially related also to this BZ and not just issue with xterm (https://bugzilla.redhat.com/show_bug.cgi?id=1718934 in case you are interested) I've just realized that I did not configure LANG=C.utf8. And my desktop manager properly set LANG=en_US.UTF-8 OR is there issue with en_US.UTF-8 vs en_US.utf8 ? sh$ locale -a | grep en_US en_US en_US.iso88591 en_US.iso885915 en_US.utf8 sh$ pgrep lightdm | while read pid; do echo ---$pid---; cat /proc/$pid/environ | tr '\0' '\n' | sort; done ---1389--- INVOCATION_ID=5bf5bfc9dc124bb0ace55a3a7372e7f0 JOURNAL_STREAM=9:32378 LANG=en_US.UTF-8 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin ---3510--- INVOCATION_ID=5bf5bfc9dc124bb0ace55a3a7372e7f0 JOURNAL_STREAM=9:32378 LANG=en_US.UTF-8 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
(In reply to Lukas Slebodnik from comment #23) > OR is there issue with en_US.UTF-8 vs en_US.utf8 ? They are the same thing. It's en_US.utf8 on disk, but en_US.UTF-8 is a valid alias.
OK, I still try to debug what has change a and why my xsession use C.utf8 The locale C.utf8 is present on my system sh$ rpm -qf /usr/lib/locale/C.utf8/* glibc-common-2.29.9000-26.fc31.x86_64 glibc-common-2.29.9000-26.fc31.x86_64 glibc-common-2.29.9000-26.fc31.x86_64 glibc-common-2.29.9000-26.fc31.x86_64 glibc-common-2.29.9000-26.fc31.x86_64 glibc-common-2.29.9000-26.fc31.x86_64 glibc-common-2.29.9000-26.fc31.x86_64 glibc-common-2.29.9000-26.fc31.x86_64 glibc-common-2.29.9000-26.fc31.x86_64 glibc-common-2.29.9000-26.fc31.x86_64 glibc-common-2.29.9000-26.fc31.x86_64 glibc-common-2.29.9000-26.fc31.x86_64 sh$ ls -l /usr/lib/locale/C.utf8/ total 1852 -rw-r--r--. 1 root root 131 Jun 5 12:27 LC_ADDRESS -rw-r--r--. 1 root root 1519058 Jun 5 12:27 LC_COLLATE -rw-r--r--. 1 root root 339912 Jun 5 12:27 LC_CTYPE -rw-r--r--. 1 root root 252 Jun 5 12:27 LC_IDENTIFICATION -rw-r--r--. 1 root root 23 Jun 5 12:27 LC_MEASUREMENT drwxr-xr-x. 1 root root 30 Jun 10 14:11 LC_MESSAGES -rw-r--r--. 1 root root 282 Jun 5 12:27 LC_MONETARY -rw-r--r--. 1 root root 62 Jun 5 12:27 LC_NAME -rw-r--r--. 1 root root 50 Jun 5 12:27 LC_NUMERIC -rw-r--r--. 1 root root 34 Jun 5 12:27 LC_PAPER -rw-r--r--. 1 root root 47 Jun 5 12:27 LC_TELEPHONE -rw-r--r--. 1 root root 3360 Jun 5 12:27 LC_TIME But I could see in ~/.xsession-errors that xrdb does not know it sh-5.0$ locale LANG=C.utf8 LC_CTYPE="C.utf8" LC_NUMERIC="C.utf8" LC_TIME="C.utf8" LC_COLLATE="C.utf8" LC_MONETARY="C.utf8" LC_MESSAGES="C.utf8" LC_PAPER="C.utf8" LC_NAME="C.utf8" LC_ADDRESS="C.utf8" LC_TELEPHONE="C.utf8" LC_MEASUREMENT="C.utf8" LC_IDENTIFICATION="C.utf8" LC_ALL= sh-5.0$ xrdb -merge /home/alcik/.Xresources /home/alcik/.Xresources:0: warning: Unknown encoding: C.utf8 sh-5.0$ sh-5.0$ LC_ALL=C.utf8 xrdb -merge /home/alcik/.Xresources /home/alcik/.Xresources:0: warning: Unknown encoding: C.utf8 sh-5.0$ sh-5.0$ LC_ALL=en_US.utf8 xrdb -merge /home/alcik/.Xresources sh-5.0$ echo $? 0 sh-5.0$ rpm -V glibc-common sh-5.0$ echo $? 0
(In reply to Lukas Slebodnik from comment #25) > sh-5.0$ xrdb -merge /home/alcik/.Xresources > /home/alcik/.Xresources:0: warning: Unknown encoding: C.utf8 This is a bug in mcpp. Please file it there. 383 strcpy( norm, name); 384 if (norm[ 5] == '.') 385 memmove( norm, norm + 5, strlen( norm + 5) + 1); 386 /* Remove initial 'xxxxx.' as 'ja_JP.', 'en_US.' or any other */ 387 conv_case( norm, norm + strlen( norm), LOWER); 388 strip_bar( norm); It assumes a 5 char locale, and C or eo (Esperanto without country) don't conform to that. Then it tries to lookup 'cutf8' and fails. A quick hack would be to add 'cutf8' as an alias for the UTF8 line of encoding_name.