Bug 1871397

*** Bug 1793577 has been marked as a duplicate of this bug. ***

Thank you for working on this. The fix broke systemd service start, if the service uses USER=.
We run CentOS 8 stream snapshot taken on 2020-12-01. That time, the glibc version/release was glibc-2.28-129.el8.x86_64. I started testing a new version, with this fix, which is glibc-2.28-145.el8.x86_64 (it also includes other fixes). Testing it, I found that some services fail to start. Here is the error from the systemd log:
```
Feb 01 18:47:17 <hostname was here> systemd[1]: Starting TPM2 Access Broker and Resource Management Daemon...
░░ Subject: A start job for unit tpm2-abrmd.service has begun execution
░░ Defined-By: systemd
░░
░░ A start job for unit tpm2-abrmd.service has begun execution.
░░
░░ The job identifier is 716412.
Feb 01 18:47:17 <hostname was here> systemd[1084136]: tpm2-abrmd.service: Failed to determine user credentials: No such process
Feb 01 18:47:17 <hostname was here> systemd[1084136]: tpm2-abrmd.service: Failed at step USER spawning /usr/sbin/tpm2-abrmd: No such process
░░ Subject: Process /usr/sbin/tpm2-abrmd could not be executed
░░ Defined-By: systemd
░░
░░ The process /usr/sbin/tpm2-abrmd could not be executed and failed.
░░
░░ The error number returned by this process is ERRNO.
Feb 01 18:47:17 <hostname was here> systemd[1]: tpm2-abrmd.service: Main process exited, code=exited, status=217/USER
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░
░░ An ExecStart= process belonging to unit tpm2-abrmd.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 217.
Feb 01 18:47:17 <hostname was here> systemd[1]: tpm2-abrmd.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░
░░ The unit tpm2-abrmd.service has entered the 'failed' state with result 'exit-code'.
Feb 01 18:47:17 <hostname was here> systemd[1]: Failed to start TPM2 Access Broker and Resource Management Daemon.
░░ Subject: A start job for unit tpm2-abrmd.service has failed
░░ Defined-By: systemd
░░
░░ A start job for unit tpm2-abrmd.service has finished with a failure.
░░
░░ The job identifier is 716412 and the job result is failed.
```

Here is the service definition (note this is not the only service fails this way, it seems all of the having USER=xxx do):
```
# /usr/lib/systemd/system/tpm2-abrmd.service
[Unit]
Description=TPM2 Access Broker and Resource Management Daemon

[Service]
Type=dbus
Restart=always
RestartSec=5
EnvironmentFile=-/etc/default/tpm2-abrmd
BusName=com.intel.tss2.Tabrmd
StandardOutput=syslog
ExecStart=/usr/sbin/tpm2-abrmd
User=tss
Environment=G_MESSAGES_DEBUG=all
MemoryMax=200M

[Install]
WantedBy=multi-user.target
```

I pulled the sources from https://git.centos.org/rpms/glibc/tree/c8s, and first just rebuilt it as it is, installed on the test machine and tried to restart the service. It failed with the same error.
Then, I commented out 11 patch lines from this bug - Patch370: glibc-rh1871397-1.patch to Patch380: glibc-rh1871397-11.patch and built it again, installed the packages on to the test host and tried to restart the service. It restarted without errors. So, it looks like the fix breaks these services.

Would you please check into this regression? If you need a new bug to be opened, LMK, I will open one.

(In reply to ifelmail from comment #14)
> Would you please check into this regression? If you need a new bug to be
> opened, LMK, I will open one.

I cannot reproduce this. Please open a new bug, include the systemd version and the contents of /etc/nsswitch.conf. Thanks.

Florian, thank you for checking into this.

nsswitch.conf is pretty much harmless:
```
aliases: files
automount: files
ethers: files
group: files mymachines systemd
hosts: files dns mymachines myhostname dns
initgroups: files
netgroup: files
netmasks: files
networks: files
passwd: files mymachines systemd
protocols: files
publickey: files
rpc: files
services: files
shadow: files
```

Systemd on the machine is not the one coming with C8s, but newer one. We use the systemd version 246.1-1, with these changes:
- Backport PR #17495 to fix program leak
- Backport PR #17495 to fix BPF program lifecycle
- Backport PR #17422 to clean up cgroups more reliably after exit
- Backport PR #17497 to add FixedRandomDelay= support
- Backport PR #16838 and #16857 to improve $PATH handling
- Backport PR #16940 to fix ECONN handling in sockets
- Backport PR #17031 to fix rate limiting on units in restart loop
- Backport PR #17082 to get nspawn TTY tweaks

- Don't compile in systemd-repart (needs libfdisk >= 2.33 and C8 has 2.32)
- Remove unused systemd-journal-remote.xml and systemd-journal-gatewayd.xml files since we never used firewalld

I'm trying to repro this with the version of systemd comes with the OS now. Another thing I found, if machine is rebooted, the issue is gone (at least for that service). This happens on provisioning time.

Please hold on, I will get back, once I have more data.

I provisioned a test host twice. First time with systemd and glibc coming from the latest version of the repo, the service failed there with the error. Here is strace output: https://gist.githubusercontent.com/ifel/3e8fa070ad70c87e07a4610051ec1cf1/raw/320fdf46e07600e665aed3f1b4440209173e8956/gistfile1.txt
Versions:
[root@<hostname> ~]# rpm -qa | grep ^systemd
systemd-pam-239-43.el8.x86_64
systemd-udev-239-43.el8.x86_64
systemd-container-239-43.el8.x86_64
systemd-libs-239-43.el8.x86_64
systemd-239-43.el8.x86_64
[root@<hostname> ~]# rpm -qa |grep ^glibc
glibc-all-langpacks-2.28-145.el8.x86_64
glibc-common-2.28-145.el8.x86_64
glibc-devel-2.28-145.el8.x86_64
glibc-headers-2.28-145.el8.x86_64
glibc-2.28-145.el8.x86_64
[root@<hostname> ~]#

Second time I provisioned the box with systemd coming from the latest version of the repo, but glibc was the latest version but rebuilt without glibc-rh1871397-1.patch to glibc-rh1871397-11.patch patches, the rest was the same. And the service started. Here is strace output: https://gist.githubusercontent.com/ifel/248c6e820462b4aeb2a470c289b1ebc7/raw/0c3622379eb78c2f9574a25672fabc1c4d3f387b/gistfile1.txt (note I cut parts of passwd, to show root and tss accounts only).

As you can see, when it works (without the patches), it reads /etc/passwd, and when it fails, it does not read the file. I'm under impression it's either uses cached version, or it tries to use other means, in spite nsswitch.conf instructs to use files.

Hope this is helpful.

Versions used on the second test (when the service worked):
[root@sparefullten25865 ~]# rpm -qa | grep ^systemd
systemd-pam-239-43.el8.x86_64
systemd-udev-239-43.el8.x86_64
systemd-container-239-43.el8.x86_64
systemd-libs-239-43.el8.x86_64
systemd-239-43.el8.x86_64
[root@sparefullten25865 ~]# rpm -qa |grep ^glibc
glibc-2.28-145.fb2.el8.x86_64
glibc-headers-2.28-145.fb2.el8.x86_64
glibc-devel-2.28-145.fb2.el8.x86_64
glibc-all-langpacks-2.28-145.fb2.el8.x86_64
glibc-common-2.28-145.fb2.el8.x86_64
[root@sparefullten25865 ~]#

(In reply to ifelmail from comment #16)
> I'm trying to repro this with the version of systemd comes with the OS now.
> Another thing I found, if machine is rebooted, the issue is gone (at least
> for that service). This happens on provisioning time.

This update cannot be installed with a reboot because the internal GLIBC_PRIVATE ABI changes. As a result, the nss_files module cannot be loaded, and the system behaves as if the information is not present.

I will check if we can make this experience smoother.

(In reply to Florian Weimer from comment #19)
> (In reply to ifelmail from comment #16)
> > I'm trying to repro this with the version of systemd comes with the OS now.
> > Another thing I found, if machine is rebooted, the issue is gone (at least
> > for that service). This happens on provisioning time.
> 
> This update cannot be installed with a reboot because the internal
> GLIBC_PRIVATE ABI changes. As a result, the nss_files module cannot be
> loaded, and the system behaves as if the information is not present.
> 
> I will check if we can make this experience smoother.

Sorry, I meant to write “cannot be installed *without* a reboot”.

Thank you. Let me know if you need any help with this - like test or something.

(In reply to ifelmail from comment #21)
> Thank you. Let me know if you need any help with this - like test or
> something.

Thanks for the offer. The reboot requirement is tracked in bug 1927040.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: glibc security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1585