What's the output of this command?

rpm -qa | grep glibc

Thanks.

(In reply to Florian Weimer from comment #1)
> What's the output of this command?
> 
> rpm -qa | grep glibc
> 
> Thanks.

[root@localhost ~]# rpm -qa | grep glibc
glibc-gconv-extra-2.34-2.el9.x86_64
glibc-minimal-langpack-2.34-2.el9.x86_64
glibc-common-2.34-2.el9.x86_64
glibc-2.34-2.el9.x86_64

The issue is that the minimal installation (with glibc-minimal-langpack) does not support the en_US.UTF-8 locale. Hower, due to OpenSSH defaults, the locale settings are transferred from the client, which presumably uses en_US.UTF-8.

I think we should consider disabling locale forwarding in OpenSSH.

I brought this up on the OpenSSH list:

Phasing out forwarding of locale settings
https://lists.mindrot.org/pipermail/openssh-unix-dev/2021-September/039582.html

This is the link of the rhel9 guest image:
https://download-node-02.eng.bos.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.0.0-20210901.d.2/compose/BaseOS/x86_64/images/rhel-guest-image-9.0-9.0.0.20210901.2.x86_64.qcow2

And I can also see the error information when I use 'yum':
[root@localhost ~]# yum 
Failed to set locale, defaulting to C.UTF-8

I am raising up this issue's priority to high since manual is not working in rhel9 guest image now.
eg.
#  man grubby
man: can't set the locale; make sure $LC_* and $LANG are correct
No manual entry for grubby
# rpm -ql grubby|grep man
/usr/share/man/man8/grubby.8.gz

A real fix will require some coordination among multiple packages. As a workaround, you could install the glibc-all-langpacks package or remove the AcceptEnv lines from /etc/ssh/sshd_config.d/50-redhat.conf.

(In reply to Florian Weimer from comment #7)
> A real fix will require some coordination among multiple packages. As a
> workaround, you could install the glibc-all-langpacks package or remove the
> AcceptEnv lines from /etc/ssh/sshd_config.d/50-redhat.conf.

Thanks, Florian
It is working after installed glibc-all-langpacks now, and I will install glibc-all-langpacks by default when build test images.

For systems already have this problem, I need to reinstall the pkgs to refresh manual page.
eg. #yum reinstall -y grubby

(In reply to Frank Liang from comment #8)
> For systems already have this problem, I need to reinstall the pkgs to
> refresh manual page.
> eg. #yum reinstall -y grubby

Interesting. Does this mean that after installation with an unsupported locale, the manual pages index isn't correctly updated? That's probably worthy of a separate man-db bug.

(In reply to Florian Weimer from comment #9)
> (In reply to Frank Liang from comment #8)
> > For systems already have this problem, I need to reinstall the pkgs to
> > refresh manual page.
> > eg. #yum reinstall -y grubby
> 
> Interesting. Does this mean that after installation with an unsupported
> locale, the manual pages index isn't correctly updated? That's probably
> worthy of a separate man-db bug.
Yes, I guess so. After installed glibc-all-langpacks, no 'man: can't set the locale; make sure $LC_* and $LANG are correct' is reported.
But still got info like before 'No manual entry for grubby'.

And it did not work after tried to reinstall man-db and ran mandb to update manual page index.

So I tried to reinstall grubby to refresh manual page about grubby and systemd to refresh manual page about systemd.special.
Obviously, it is not a good idea if you want to fix all manual pages, but I did not find other ways till now.

man page bug is a separated issue just for rhel-guest-image and it's already tracked by https://bugzilla.redhat.com/show_bug.cgi?id=2002034

Hi Ondřej,
 Per c7, could we consider to install glibc-all-langpacks or at least just glibc-langpack-en explicitly in rhel-guest-image-9.0?

Hello Wei,

We can surely consider that in 9.0 GA. The downside is that it will make the image bigger.

Ondřej

(In reply to Ondřej Budai from comment #13)
> We can surely consider that in 9.0 GA. The downside is that it will make the
> image bigger.

glibc-all-langpacks is *huge*:

# ls -lh /usr/lib/locale/locale-archive
-rw-r--r--. 2 root root 214M Aug 10 02:42 /usr/lib/locale/locale-archive

So that's probably a non-starter.

glibc-langpack-en is smaller, but it will only help with some client settings:

# du -hcs /usr/lib/locale/en_*
[…]
6.9M	total

We (the glibc team) have been working on the assumption that C.UTF-8 will be the locale used by small images, and focused our efforts on reducing its size. We have not investigated size reduction of en_US.UTF-8.

I think disabling locale forwarding in OpenSSH is the way to fix this.

(In reply to Wei Shi from comment #11)
> man page bug is a separated issue just for rhel-guest-image and it's already
> tracked by https://bugzilla.redhat.com/show_bug.cgi?id=2002034

I read  this as completely unrelated to locale settings. I initially expected a locale dependency there as well. But it seems that there isn't any glibc work to be done for bug 2002034.

(In reply to Florian Weimer from comment #14)
> We (the glibc team) have been working on the assumption that C.UTF-8 will be
> the locale used by small images, and focused our efforts on reducing its
> size. We have not investigated size reduction of en_US.UTF-8.
> 
> I think disabling locale forwarding in OpenSSH is the way to fix this.

I couldn't agree more, thanks for the info.

I brought the matter of locale forwarding to the Fedora devel list:

Disable locale forwarding in OpenSSH
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/U6EZDFY77IVN24QQMNMDAGGZ6JNLXOFC/

I am closing this as a duplicate of the OpenSSH bug 2002734.

There is no action for the glibc team here, but if you have any questions please don't hesitate to reach out.

*** This bug has been marked as a duplicate of bug 2002734 ***

Hi Ondřej,
  I reopened this bug since the problem still exists, the openssh fix could help to solve some of the scenario, but it seems the root cause is that we only have minimal installed but we set the default to en_US.UTF-8 which is not available.

# cat /etc/locale.conf 
LANG=en_US.UTF-8
# locale -a
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_COLLATE to default locale: No such file or directory
C
C.utf8
POSIX

Note that for anaconda, this was fixed in bug 1958876. Does osbuild-composer use anaconda, too?

No, I think osbuild-composer doesn't use anaconda.

And this might be the related code snippet
https://github.com/osbuild/osbuild-composer/blob/main/internal/distro/rhel90/distro.go#L60

Hi,

Ondřej is out for a while, so I am taking this on.

(In reply to Wei Shi from comment #22)
> No, I think osbuild-composer doesn't use anaconda.
> 
> And this might be the related code snippet
> https://github.com/osbuild/osbuild-composer/blob/main/internal/distro/rhel90/
> distro.go#L60

Wei, you are exactly right and the location you pointed out is indeed the correct place. 

I read up on the discussion and as per what Florian said:

> We (the glibc team) have been working on the assumption that C.UTF-8 will be the locale used by small images, and focused our efforts on reducing its size. We have not investigated size reduction of en_US.UTF-8.

I think it is the right call to change the aforementioned line from "en_US.UTF-8" to "C.UTF-8" for all RHEL 9 based images that only have "glibc-minimal-langpack".

(In reply to Christian Kellner from comment #23)
> I think it is the right call to change the aforementioned line from
> "en_US.UTF-8" to "C.UTF-8" for all RHEL 9 based images that only have
> "glibc-minimal-langpack".

As the glibc team lead I can confirm this: Yes, C.UTF-8 is *always*
installed in RHEL9, and is minimally sized (360KiB). You cannot remove
C.UTF-8 with a package manager, it is designed to be always available
for application use as the default UTF-8 compliant locale that you can
use. Even if you remove glibc-minimal-langpack you will still have C.UTF-8.
C.UTF-8 is part of glibc-common and can not be removed.

Hi Christian,
  I don't see any updates on upstream issue: osbuild/osbuild-composer/issues/2206, do you need any help to move it forward?

Tested with rhel-guest-image-9.0-20220304.3.x86_64.qcow2, and I can confirm that osbuild/osbuild-composer/issues/2206 is fixed.

However, cloud-init will reset /etc/locale.conf back to "en_US.UTF-8" after the initial launch of instance, because of we set locale in cloud.cfg by default and its default behavior:

/etc/cloud/cloud.cfg
cloud_config_modules:
 - locale

https://cloudinit.readthedocs.io/en/latest/topics/modules.html#locale

https://github.com/canonical/cloud-init/blob/main/cloudinit/sources/__init__.py#L155
https://github.com/canonical/cloud-init/blob/main/cloudinit/sources/__init__.py#L663

Christian, Eduardo,
  I'm not quite sure if remove "locale" in /etc/cloud/cloud.cfg just in rhel-guest-image would be a good way to fix this issue.

(In reply to Wei Shi from comment #30)
> Tested with rhel-guest-image-9.0-20220304.3.x86_64.qcow2, and I can confirm
> that osbuild/osbuild-composer/issues/2206 is fixed.
> 
> However, cloud-init will reset /etc/locale.conf back to "en_US.UTF-8" after
> the initial launch of instance, because of we set locale in cloud.cfg by
> default and its default behavior:
> 
> /etc/cloud/cloud.cfg
> cloud_config_modules:
>  - locale
> 
> https://cloudinit.readthedocs.io/en/latest/topics/modules.html#locale
> 
> https://github.com/canonical/cloud-init/blob/main/cloudinit/sources/__init__.
> py#L155
> https://github.com/canonical/cloud-init/blob/main/cloudinit/sources/__init__.
> py#L663
> 
> Christian, Eduardo,
>   I'm not quite sure if remove "locale" in /etc/cloud/cloud.cfg just in
> rhel-guest-image would be a good way to fix this issue.

I don't think setting locale is actually required there. Can you remove that line and run some tests? I just reviewed our commit log / Bugzilla history and didn't find anything related. If your early tests are ok, we can file a new BZ to remove that from 9.0 and have exception approved.

Thanks!

Eduardo,
  My early test is PASS for removing locale in /etc/cloud/cloud.cfg, and I've created a new bz#2061604 for tracking against osbuild-composer, please move it to cloud-init if I read it wrong.

*** Bug 2022363 has been marked as a duplicate of this bug. ***

(In reply to Wei Shi from comment #32)
> Eduardo,
>   My early test is PASS for removing locale in /etc/cloud/cloud.cfg, and
> I've created a new bz#2061604 for tracking against osbuild-composer, please
> move it to cloud-init if I read it wrong.

Thanks a lot, just saw the new BZ.
This BZ can carry on and be closed.

We'll handle cloud-init bug on bz#2061604

Change the status to Verified per c30, and the new problem is tracking in the other bz#2061604.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: osbuild-composer), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2522