The installation was succeeding with RHEL-9.0.0-20210707.2 but is failing with RHEL-9.0.0-20210708.3

The updated packages between the 2 versions are:
crypto-policies
dnf
filesystem
glibc
libnl3
libtirpc
numactl-libs
perl-Text-Tabs+Wrap
python3-dnf
python3-dnf-plugins-core
slapi-nis
yum
yum-utils

Hope it helps narrowing the investigations.

The issue comes from the latest glibc version (glibc-2.33.9000-36.el9.x86_64).
Downgrading to glibc-common-2.33-14.el9.x86_64 fixes the installation issue.

Could you provide more information on what actually fails? How can we reproduce this? Thanks.

The failure happens during IPA server installation. In order to reproduce:
dnf install ipa-server ipa-server-dns
ipa-server-install —domain ipa.test —realm IPA.TEST —setup-dns —auto-forwarder -a Secret123 -p Secret123 -U

The installation fails while configuring the Certificate server. ipa-server-install is internally calling pkispawn, that sets up the Certificate server and restarts it with systemctl start pki-tomcatd@pki-tomcat.

Edewata may provide more details on the exact step failing when starting pki, based on the journal logs:
Jul 09 10:46:12 master.testrealm.test systemd[24185]: pki-tomcatd: Failed to determine user credentials: No such process
Jul 09 10:46:12 master.testrealm.test systemd[24185]: pki-tomcatd: Failed at step USER spawning /usr/sbin/pki-server: No such process

Unfortunately I’m not familiar enough with pki internals to point to the exact call that is failing.

I don't have any info about this particular installation scenario.
Could someone provide the pkispawn log and CA/KRA debug log?

(In reply to Florence Blanc-Renaud from comment #4)
> The failure happens during IPA server installation. In order to reproduce:
> dnf install ipa-server ipa-server-dns
> ipa-server-install —domain ipa.test —realm IPA.TEST —setup-dns
> —auto-forwarder -a Secret123 -p Secret123 -U
> 
> The installation fails while configuring the Certificate server.
> ipa-server-install is internally calling pkispawn, that sets up the
> Certificate server and restarts it with systemctl start
> pki-tomcatd@pki-tomcat.

With the -36 glibc...

rpm -qa | grep glibc
glibc-langpack-en-2.33.9000-36.el9.x86_64
glibc-common-2.33.9000-36.el9.x86_64
glibc-2.33.9000-36.el9.x86_64
glibc-gconv-extra-2.33.9000-36.el9.x86_64

And with compose RHEL-9.0.0-20210708.3 the commands you state work.

ipa-server-install --domain ipa.test --realm IPA.TEST --setup-dns --auto-forwarder -a Secret123 -p Secret123 -U
...
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/28]: configuring certificate server instance
  [2/28]: stopping certificate server instance to update CS.cfg
  [3/28]: backing up CS.cfg
  [4/28]: Add ipa-pki-wait-running
  [5/28]: secure AJP connector
  [6/28]: reindex attributes
  [7/28]: exporting Dogtag certificate store pin
  [8/28]: disabling nonces
  [9/28]: set up CRL publishing
  [10/28]: enable PKIX certificate path discovery and validation
  [11/28]: authorizing RA to modify profiles
  [12/28]: authorizing RA to manage lightweight CAs
  [13/28]: Ensure lightweight CAs container exists
  [14/28]: starting certificate server instance
  [15/28]: configure certmonger for renewals
  [16/28]: requesting RA certificate from CA
  [17/28]: publishing the CA certificate
  [18/28]: adding RA agent as a trusted user
  [19/28]: configure certificate renewals
  [20/28]: Configure HTTP to proxy connections
  [21/28]: updating IPA configuration
  [22/28]: enabling CA instance
  [23/28]: importing IPA certificate profiles
  [24/28]: migrating certificate profiles to LDAP
  [25/28]: adding default CA ACL
  [26/28]: adding 'ipa' CA entry
  [27/28]: configuring certmonger renewal for lightweight CAs
  [28/28]: deploying ACME service
Done configuring certificate server (pki-tomcatd).
...
==============================================================================
Setup complete

Next steps:
	1. You must make sure these network ports are open:
		TCP Ports:
		  * 80, 443: HTTP/HTTPS
		  * 389, 636: LDAP/LDAPS
		  * 88, 464: kerberos
		  * 53: bind
		UDP Ports:
		  * 88, 464: kerberos
		  * 53: bind
		  * 123: ntp

	2. You can now obtain a kerberos ticket using the command: 'kinit admin'
	   This ticket will allow you to use the IPA tools (e.g., ipa user-add)
	   and the web user interface.

Be sure to back up the CA certificates stored in /root/cacert.p12
These files are required to create replicas. The password for these
files is the Directory Manager password
The ipa-server-install command was successful
[root@intel-chiefriver-04 ~]# echo $?
0

I don't have an experience here with this software so we're relying on you to provide guidance.

Actual command line: ipa-server-install --domain ipa.test --realm IPA.TEST --setup-dns --auto-forwarder -a Secret123 -p Secret123 -U

Same for me: Installation succeeded for me on s390x. And three times on x86-64 (on different systems).

Here are my steps:
1minutetip 1MT-RHEL-9.0.0-20210706.3-Beta

# dnf install -y ipa-server ipa-server-dns
# ipa-server-install --domain ipa.test --realm IPA.TEST --setup-dns --auto-forwarder -a Secret123 -p Secret123 -U
=> installation was successful

# ipa-server-install --uninstall -U

Add repo for RHEL-9.0.0-20210708.3-Beta:
# cat /etc/yum.repos.d/faulty.repo
[rhel-latest]
name=rhel-latest
baseurl=http://download.eng.rdu2.redhat.com/rhel-9/nightly/RHEL-9-Beta/RHEL-9.0.0-20210708.3/compose/BaseOS/x86_64/os/
enabled=1
gpgcheck=0

[rhel-AppStream-latest]
name=rhel-AppStream-latest
baseurl=http://download.eng.rdu2.redhat.com/rhel-9/nightly/RHEL-9-Beta/RHEL-9.0.0-20210708.3/compose/AppStream/x86_64/os/
enabled=1
gpgcheck=0

# dnf update -y glibc
[...]
Upgraded:
  glibc-2.33.9000-36.el9.x86_64                          glibc-common-2.33.9000-36.el9.x86_64                          glibc-langpack-en-2.33.9000-36.el9.x86_64                         
Installed:
  glibc-gconv-extra-2.33.9000-36.el9.x86_64                                                                                                                                                

Complete!
# ipa-server-install --domain ipa.test --realm IPA.TEST --setup-dns --auto-forwarder -a Secret123 -p Secret123 -U
[...]
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/28]: configuring certificate server instance
Failed to configure CA instance
See the installation logs and the following files/directories for more information:
  /var/log/pki/pki-tomcat
  [error] RuntimeError: CA configuration failed.
CA configuration failed.
The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information


Uploading the logs files (ipaserver-install.log, pki-ca-spawn.log and journal)

(In reply to Florence Blanc-Renaud from comment #8)
> # dnf update -y glibc

> # ipa-server-install --domain ipa.test --realm IPA.TEST --setup-dns

Is there a reboot between the two steps?

If not, this is probably the cause. It's conceptually similar to bug 1927040. However, we are not treating this instance here as a bug because you are upgrading across different major releases of glibc, which is currently not supportable without a reboot.

We are seeing a significant number of tests failing in CI across our team with similar systemd errors to comment 4 at the moment.

(In reply to Florian Weimer from comment #10)
> (In reply to Florence Blanc-Renaud from comment #8)
> > # dnf update -y glibc
> 
> > # ipa-server-install --domain ipa.test --realm IPA.TEST --setup-dns
> 
> Is there a reboot between the two steps?
> 
> If not, this is probably the cause. It's conceptually similar to bug
> 1927040. However, we are not treating this instance here as a bug because
> you are upgrading across different major releases of glibc, which is
> currently not supportable without a reboot.

The problem is glibc is pulled by packages as a dependencies in CI without reboot which breaks all CI tests (for example cyrus-imapd).

I was under the impression that buildroot and compose were now in sync, so if the buildroot contains the new glibc, the compose should as well.

The new glibc is actually in some of the composes (I checked a nightly compose recently and it was already included). Which CI systems suffer from this, and how can we make sure they use a current compose?

Thanks for reporting back. I'm closing this bug because it's not a glibc issue, it's an infrastructure problem.

We can call systemctl daemon-reexec during package updates. This should help a little bit.

Note to QE: Successful re-exec can be verified by checking the inode number in “lsof -p 1” against “ls -li /usr/lib64/libc.so.6”. They should match.