Bug 2082845
| Summary: | Unable to manage RHEL for Edge hosts with IDM | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Benjamin Holmes <bholmes> |
| Component: | authselect | Assignee: | Pavel Březina <pbrezina> |
| Status: | CLOSED ERRATA | QA Contact: | Dan Lavu <dlavu> |
| Severity: | low | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 8.5 | CC: | abokovoy, aboscatt, atikhono, dlavu, dornelas, elpereir, ikke, jlebon, mschibli, pbrezina, rcritten, rdonca, sgadekar, travier, tscherf, walters |
| Target Milestone: | rc | Keywords: | Triaged |
| Target Release: | --- | Flags: | pm-rhel:
mirror+
|
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | authselect-1.2.6-1.el8 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-05-16 09:10:47 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 2142804 | ||
| Bug Blocks: | |||
| Attachments: | |||
Created attachment 1877831 [details]
ipa-client-install output
Output logs
Created attachment 1877832 [details]
sshd log output
Created attachment 1877833 [details]
sssd log output
Created attachment 1877834 [details]
ssh key permissions errors
@(In reply to Rob Crittenden from comment #5) > Is there a reason this is private? I see no customer information listed. Force of habit. Have rectified. > > Please provide more detailed reproduction steps for 1 and 2. We have no > experience with Edge. The more detailed the better. Aside from working round a configuration issue - https://access.redhat.com/solutions/5773421 - I simply followed our docs to build and deploy the image: For step 1: Set-up - https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/composing_installing_and_managing_rhel_for_edge_images/setting-up-image-builder_composing-installing-managing-rhel-for-edge-images Creating - https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/composing_installing_and_managing_rhel_for_edge_images/composing-rhel-for-edge-images-using-image-builder-in-rhel-web-console_composing-installing-managing-rhel-for-edge-images I explicitly added the packages seen in the attachments 'image-builder-packages.png' and 'image-builder-full-package-list.txt' Downloading - https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/composing_installing_and_managing_rhel_for_edge_images/composing-a-rhel-for-edge-image-using-image-builder-command-line_composing-installing-managing-rhel-for-edge-images#downloading-a-rhel-for-edge-image-using-the-command-line_composing-a-rhel-for-edge-image-using-image-builder-command-line Deploying - https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/composing_installing_and_managing_rhel_for_edge_images/installing-rpm-ostree-images_composing-installing-managing-rhel-for-edge-images Image was deployed as a VM in vSphere. It has 1 vCPU and 4GB RAM associated with it. Created attachment 1878435 [details]
packages added explicitly to RHEL for Edge commit with image builder.
Created attachment 1878436 [details]
full list of packages and dependencies added to RHEL for Edge commit with image builder
My first attempt at reproducing this resulted in a working image. sshd and sssd are functional after ipa-client-install (and a reboot). I believe I followed the steps, the image very much helped. - I started with an 8.5 image - Installed all the builder software per the docs - registered using subscription manager (required) - fired up cockpit and created a blueprint - I added the packages per your image - I created a user with system powers. - I created a qcow2 image and used virt-manager to import that - After first boot I ran ipa-client-install - id admin works and sshd works - reboot and id and sshd continue to work I ended up with a RHEL 8.6 client image as that is the repos I got via RHSM. I don't know if this is an 8.5 vs 8.6 issue. (In reply to Rob Crittenden from comment #9) > My first attempt at reproducing this resulted in a working image. sshd and > sssd are functional after ipa-client-install (and a reboot). > > I believe I followed the steps, the image very much helped. > > - I started with an 8.5 image > - Installed all the builder software per the docs > - registered using subscription manager (required) > - fired up cockpit and created a blueprint > - I added the packages per your image > - I created a user with system powers. > - I created a qcow2 image and used virt-manager to import that > - After first boot I ran ipa-client-install > - id admin works and sshd works > - reboot and id and sshd continue to work > > I ended up with a RHEL 8.6 client image as that is the repos I got via RHSM. > > I don't know if this is an 8.5 vs 8.6 issue. Thanks for picking this up, Rob. Generating a qcow2 will unfortunately not give you a RHEL for Edge image. It just gives you a 'full fat' RHEL image containing the packages you added and configurations you selected in Image Builder. I double checked this by creating and downloading the QEMU qcow2 for my image. I'm sorry I didn't go into more detail on that, I should've been more specific. As per [1], the valid image formats for deploying Edge hosts are: * RHEL for Edge Commit * RHEL for Edge Container * RHEL for Edge Installer * RHEL for Edge Raw * RHEL for Edge Simplified Installer With that in mind, I set up an http server, and generated a RHEL for Edge Commit with the packages I wanted - I believe it's the top option in the dropdown when selecting an image type to create. I downloaded the commit using the composer-cli as per [2], and moved the edge commit .tar file to my http host. I unpacked the RHEL for Edge Commit (the .tar file) onto the http server as per [3]. Sections 7.1, 7.3, and 7.4 are pertinent to me - I didn't do the containerised http host in 7.2 as I already had a friendly apache instance that I used. I created a kickstart file (will attach to BZ) and hosted it alongside the edge commit. So I had a directory structure in /var/www/html/edge which looked like: [bholmes@ignite edge]$ ll total 928548 -rw-r--r--. 1 root root 554 May 10 21:52 compose.json <-- extracted from from tar -rw-r--r--. 1 bholmes bholmes 950824960 May 11 20:50 e7ce11c6-d3dc-41f0-b740-3a3e8e0cc0b9-commit.tar <-- the edge commit, downloaded form image builder -rw-r--r--. 1 root root 883 May 5 16:28 kickstart.cfg <-- self-explanatory, but hand-crafted drwxr-xr-x. 7 root root 102 May 10 21:52 repo <-- extracted from tar I then used the RHEL 8 boot iso, added the 'inst.ks=http://<my http host>/edge/kickstart.cfg' parameter to the boot options, and left it alone to do its thing. After this had deployed, I ran the ipa-client-install process as per my original comments. [1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/composing_a_customized_rhel_system_image/composer-description_composing-a-customized-rhel-system-image#composer-output-formats_composer-description [2] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/composing_installing_and_managing_rhel_for_edge_images/composing-a-rhel-for-edge-image-using-image-builder-command-line_composing-installing-managing-rhel-for-edge-images#downloading-a-rhel-for-edge-image-using-the-command-line_composing-a-rhel-for-edge-image-using-image-builder-command-line [3] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/composing_installing_and_managing_rhel_for_edge_images/installing-rpm-ostree-images_composing-installing-managing-rhel-for-edge-images Created attachment 1879353 [details]
kickstart.cfg for edge host
The issue with sssd is that the sssd user and group are missing so the sssd_nss process can't start. This KCS contains part of the fix, adding the user https://access.redhat.com/solutions/6021931 In addition you need: groupadd -g 993 sssd Restart sssd and this should work: getent passwd admin They key error message for sshd is: Privilege separation user sshd does not exist. So similar to sssd entries are missing, in this case two users and a group. As a result the ownership of /etc/ssh/*_key is incorrect. You need to create two groups and a user, then fix the group ownership of /etc/ssh/*_key I'm not sure why the sshd and sssd users and groups aren't being created on installation. The sssd-common package is responsible for adding the entries. Its preinstall rpm script is: getent group sssd >/dev/null || groupadd -r sssd getent passwd sssd >/dev/null || useradd -r -g sssd -d / -s /sbin/nologin -c "User for sssd" sssd Similarly openssh-server has this as preinstall: getent group sshd >/dev/null || groupadd -g 74 -r sshd || : getent passwd sshd >/dev/null || \ useradd -c "Privilege-separated SSH" -u 74 -g sshd \ -s /sbin/nologin -r -d /var/empty/sshd sshd 2> /dev/null || : And finally openssh has this: getent group ssh_keys >/dev/null || groupadd -r ssh_keys || : So if you run these user/groupadd commands to create the passwd/group entries then you just need to fixup the permissions and start the service: chgrp ssh_keys /etc/ssh/*_key systemctl start sshd I think it is common problem for container-alike images which built from RPMs without running the post-install scripts. Any of the scripts that create resources to be present in the final image would not be run as expected. See https://github.com/coreos/rpm-ostree/issues/1884 for a long standing issue. Nobody took the stack of packages that comprises FreeIPA dependencies to convert their post-install scripts into something that runs as a systemd unit yet. Technically, this should be possible just fine with RPM because %post and similar macros take '-p' argument to specify the shell to process the script (/bin/sh by default). It could be changed be some wrapper that creates a one-off unit to run. However, the bigger problem is that it doesn't solve the problem at all. CoreOS claims to run post-install scripts and triggers during image build phase: https://coreos.github.io/rpm-ostree/architecture-core/#overall-architecture, so it should not be required to create these systemd units, which contradicts what cgwalters said in the github issue above. Creating systemd unit would also not help as it would be run on the system and effectively would require those scripts to be arranged to run before anything useful to be done with IPA on the system itself. We definitely need to maintain some coherency with the set of UIDs/GIDs for system users (not just names) to survive across multiple boots. Changing component to rpm-ostree. This is not something individual package owners can manage. IIUC, we're saying that RPM preinstall/postinstall scripts are not being run when Image Builder/osbuild does an rpm-ostree compose? To test, I have created the sshd / sssd users and groups as per Rob's guidance: [root@jumpbox ~]# cat /etc/passwd root:x:0:0:root:/root:/bin/bash sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin sssd:x:996:993:User for sssd:/:/sbin/nologin [root@jumpbox ~]# cat /etc/group root:x:0: wheel:x:10:root sshd:x:74: sssd:x:993: And fixed the ssh keys: [root@jumpbox ~]# ls -la /etc/ssh/ total 612 drwxr-xr-x. 3 root root 245 May 17 14:32 . drwxr-xr-x. 92 root root 8192 May 17 14:48 .. -rw-r--r--. 1 root root 577388 May 17 14:31 moduli -rw-r--r--. 1 root root 1770 May 17 14:31 ssh_config drwxr-xr-x. 2 root root 28 May 17 14:31 ssh_config.d -rw-------. 1 root ssh_keys 480 May 17 14:32 ssh_host_ecdsa_key -rw-r--r--. 1 root root 162 May 17 14:32 ssh_host_ecdsa_key.pub -rw-------. 1 root ssh_keys 387 May 17 14:32 ssh_host_ed25519_key -rw-r--r--. 1 root root 82 May 17 14:32 ssh_host_ed25519_key.pub -rw-------. 1 root ssh_keys 2578 May 17 14:32 ssh_host_rsa_key -rw-r--r--. 1 root root 554 May 17 14:32 ssh_host_rsa_key.pub -rw-------. 1 root root 4269 May 17 14:31 sshd_config I ran the ipa-client-install: ipa-client-install --mkhomedir -U -w <credential> -p <credential> --no-sudo --no-ntp This allowed me to complete the installation of the ipa client succesfully, and log into the host as a user managed by IDM which is great. However, when rebooting the host, the host fails to restart cleanly / successfully due to a number of failed services (screenshot attached). When the host finally reaches the login prompt, it is inaccessible. I'll try and get access to extract logs. Did you experience this Rob? Created attachment 1880481 [details]
boot failures after successfull ipa-client-install
As per previous comment, even after a successful installation of the ipa client, the host becomes inaccessible after a reboot.
I tried just now and get similar behavior. I booted into single-user mode and the dbus user is missing which is probably causing cascading failures. I think this is just another side-effect of rpm %post not being executed, in this case dbus-daemon. Given the initial description, I suspect that the IPA installation is removing the `altfiles` modules from `/etc/nsswitch.conf` which breaks user/group resolution on the system. From an RHCOS node (might be slightly different on RHEL Edge): ``` $ grep altfiles /etc/nsswitch.conf passwd: sss files altfiles systemd group: sss files altfiles systemd ``` Workaround, if this is the issue, is to re-add `altfiles` to `/etc/nsswitch.conf`. This is likely a bug / missing backport in RHEL authselect. See: - https://github.com/authselect/authselect/pull/273 - https://discussion.fedoraproject.org/t/systemd-resolved-service-failed-to-determine-user-credentials-no-such-process/33240/4 - https://pagure.io/fedora-kde/SIG/issue/152 IPA is not touching anything in /etc/nsswitch.conf itself. As far as I can see, authselect in RHEL does not have support for rpm-ostree, so the pull request you talk about is not present in any RHEL version. Thimothee, could you please open a separate bug for authselect and link it here? I don't want to move this bug to authselect because there are might be more changes needed in other components as well and it is better to have a tracker for those (this bug). Given the reference to SSSD, I'd like to bring up https://github.com/SSSD/sssd/issues/6107 which got fixed in 8.6 but may still be at play here. By default, SSSD runs as root. When you enroll a host into IPA deployment, ipa-client-install does not configure SSSD to run as unprivileged user, so this does not apply at all. Fresh 8.6-based RHEL for Edge deployment
# rpm-ostree status
State: idle
Deployments:
● rhel:rhel/8/x86_64/edge
Version: 8.6 (2022-05-19T20:55:20Z)
Commit: 921b293d7f9a0dea5660e5daca925352f8d84caa6af9f8106dd313c5d998b590
# rpm -q ostree rpm-ostree sssd-client ipa-client glibc nss-altfiles
ostree-2022.1-2.el8.x86_64
rpm-ostree-2022.2-2.el8.x86_64
sssd-client-2.6.2-4.el8_6.x86_64
ipa-client-4.9.8-6.module+el8.6.0+14300+0c339766.x86_64
glibc-2.28-189.1.el8.x86_64
nss-altfiles-2.18.1-12.el8.x86_64
# id sssd
uid=995(sssd) gid=992(sssd) groups=992(sssd)
# id sshd
uid=74(sshd) gid=74(sshd) groups=74(sshd)
# grep alt/etc/nsswitch.conf
# Commonly used alternative service providers (may need installation):
passwd: sss files altfiles systemd
group: sss files altfiles systemd
I have no IDM domain to test the ipa-client-install against, but it looks like everything is configured as expected before joining a domain.
You can try to enroll it to the public demo we have: https://www.freeipa.org/page/Demo (In reply to Alexander Bokovoy from comment #22) > Thimothee, could you please open a separate bug for authselect and link it > here? I don't want to move this bug to authselect because there are might be > more changes needed in other components as well and it is better to have a > tracker for those (this bug). Can you verify that my guess is correct? If it is, then this is not a bug in rpm-ostree and we can move this one to authselect. We will answer other bugs as they come in: one issue per bug. post-run of ipa-client-install passwd and group lack altfiles in nsswitch.conf. Restoring them allows lookup of the missing entities and sshd and sssd start successfully. Thanks. Can you also verify if it affects RHEL Edge 8.6 like Derrick tried? Is using RHEL 8.6 an option for you? 8.5 should be EOL now if I'm not mistaken. authselect folks: see https://bugzilla.redhat.com/show_bug.cgi?id=2082845#c21 for the summary Do I understand it correctly that RHE Edge is equivalent to Fedora Silverblue? I can provide patch for authselect that we use on Fedora https://src.fedoraproject.org/rpms/authselect/blob/rawhide/f/authselect.spec#_315. However, as far as I can tell there is also some override in Silverblue upstream. (In reply to Pavel Březina from comment #31) > Do I understand it correctly that RHE Edge is equivalent to Fedora > Silverblue? > Yes, it is a rpm-ostree-based RHEL variant. Fedora Silverblue is a rpm-ostree variant of Fedora. (In reply to Pavel Březina from comment #31) > I can provide patch for authselect that we use on Fedora > https://src.fedoraproject.org/rpms/authselect/blob/rawhide/f/authselect. > spec#_315. However, as far as I can tell there is also some override in > Silverblue upstream. I think this should work. To test it: 1. Do the setup as before and stop before the IPA setup. 2. Use `rpm-ostree override replace ./authselect*.rpm` and reboot 3. Do the IPA setup 4. Confirm that altfiles is still in `/etc/nsswitch.conf` As per request from @aboscatt, updating the ticket to keep it alive. Currently waiting for SaaS Image Builder to support RHEL 8.6 before rebuilding my currently affected hosts. The scratch build is for 8.5 so you should be able to test it without 8.6 support. It was not yet pushed to dist git, it is *just a scratch build*. I am waiting for positive feedback to know that the issue has been resolved. Hi Pavel,
I was talking to Ben earlier today and, he shared the feedback below, maybe it is worth it to let it registered here.
==============================================
It's not just the users for sssd / sshd that are missing. Any rpm that is installed against and Edge 8.5 host is missing its users
I've had to add the following users to my edge host in order for it to function correctly:
```
- "sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin"
- "sssd:x:996:993:User for sssd:/:/sbin/nologin"
- "dbus:x:81:81:System message bus:/:/sbin/nologin"
- "polkitd:x:998:996:User for polkitd:/:/sbin/nologin"
- "chrony:x:992:989::/var/lib/chrony:/sbin/nologin"
```
(extract from a playbook I'm using to fix up my Edge 8.5 hosts)
==============================================
Do you have any guesses if those are related to authselect itself, or it should be addressed somewhere else? I will leave it to Ben to add more details if needed, and provide feedback from the scratch build.
Kindly
The problem in authselect is that is is missing nss-altfiles module which is required on rpm-ostree systems. If these users are in altfiles location then yes, it is authselect issue which should be fixed by given scratch build. If these users should be in /etc/passwd then no, it is not related to authselect. New scratch build link: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=46455806 Please, provide feedback. Once I get positive feedback, I will push the build to rhel. Please follow the instructions from https://bugzilla.redhat.com/show_bug.cgi?id=2082845#c34 to test this package so that we can move forward. Can you try following the instructions from https://bugzilla.redhat.com/show_bug.cgi?id=2082845#c34 with the temporary package build to verify the fix? Hi, I face this problem on my test lab. I can test this for you, given that I get provided rhel9 rpm for it. Or I can also just patch the system manually if given what to do. If you want, I can also build you a test edge image for vmware or kvm, if it helps. I have now rhel9-edge image running in our lab on vmware, which faces this altfiles problem. Log is telling sssd can't find users that exist in /usr/lib/passwd. I have also IdM which I can test against. Unfortunately it still fails getting service users. I tested this now. This time it ends like this: Enrolled in IPA realm COOL.LAB Created /etc/ipa/default.conf Configured /etc/sssd/sssd.conf Configured /etc/krb5.conf for IPA realm COOL.LAB Systemwide CA database updated. Hostname (rh-edge-01.cool.lab) does not have A/AAAA record. Missing reverse record(s) for address(es): 10.128.1.137. Adding SSH public key from /etc/ssh/ssh_host_ed25519_key.pub Adding SSH public key from /etc/ssh/ssh_host_ecdsa_key.pub Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub SSSD enabled SSSD service restart was unsuccessful. Configured /etc/openldap/ldap.conf Unable to find 'admin' user with 'getent passwd admin'! Unable to reliably detect configuration. Check NSS setup manually. Configured /etc/ssh/ssh_config Configured /etc/ssh/sshd_config.d/04-ipa.conf sshd failed to restart: CalledProcessError(Command ['/bin/systemctl', 'restart', 'sshd.service'] returned non-zero exit status 1: 'Job for sshd.service failed because the control process exited with error code.\nSee "systemctl status sshd.service" and "journalctl -xeu sshd.service" for details.\n') Configuring cool.lab as NIS domain. Client configuration complete. The ipa-client-install command was successful [root@localhost ~]# rpm -qi authselect Name : authselect Version : 1.2.3 Release : 7.el9_0.1 Architecture: x86_64 Install Date: Tue Sep 27 13:12:56 2022 Group : Unspecified Size : 141465 License : GPLv3+ Signature : (none) Source RPM : authselect-1.2.3-7.el9_0.1.src.rpm Build Date : Mon Sep 26 14:23:41 2022 Build Host : x86-64-01.build.eng.rdu2.redhat.com Packager : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla> Vendor : Red Hat, Inc. URL : https://github.com/authselect/authselect Summary : Configures authentication and identity sources from supported profiles Description : Authselect is designed to be a replacement for authconfig but it takes a different approach to configure the system. Instead of letting the administrator build the PAM stack with a tool (which may potentially end up with a broken configuration), it would ship several tested stacks (profiles) that solve a use-case and are well tested and supported. At the same time, some obsolete features of authconfig are not supported by authselect. here is what chrony complains:
Continue to configure the system with these values? [no]: yes
Synchronizing time
Configuration of chrony was changed by installer.
CalledProcessError(Command ['/bin/systemctl', 'restart', 'chronyd.service'] returned non-zero exit status 1: 'Job for chronyd.service failed because the control process exited with error code.\nSee "systemctl status chronyd.service" and "journalctl -xeu chronyd.service" for details.\n')
The ipa-client-install command failed. See /var/log/ipaclient-install.log for more information
[root@localhost ~]# systemctl status chronyd
× chronyd.service - NTP client/server
Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2022-09-27 14:39:12 EEST; 20s ago
Docs: man:chronyd(8)
man:chrony.conf(5)
Process: 9141 ExecStart=/usr/sbin/chronyd $OPTIONS (code=exited, status=1/FAILURE)
CPU: 18ms
Sep 27 14:39:12 rh-edge-01.cool.lab systemd[1]: Starting NTP client/server...
Sep 27 14:39:12 rh-edge-01.cool.lab chronyd[9143]: chronyd version 4.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SI>
Sep 27 14:39:12 rh-edge-01.cool.lab chronyd[9143]: Fatal error : Could not get user/group ID of chrony
Sep 27 14:39:12 rh-edge-01.cool.lab chronyd[9141]: Could not get user/group ID of chrony
Sep 27 14:39:12 rh-edge-01.cool.lab systemd[1]: chronyd.service: Control process exited, code=exited, status=1/FAILURE
Sep 27 14:39:12 rh-edge-01.cool.lab systemd[1]: chronyd.service: Failed with result 'exit-code'.
Sep 27 14:39:12 rh-edge-01.cool.lab systemd[1]: Failed to start NTP client/server.
It works for me. I used the image you given me and downloaded packages from http://brew-task-repos.usersys.redhat.com/repos/scratch/pbrezina/authselect/1.2.3/7.el9_0.1/x86_64/ (without srpm). Then: $ rpm-ostree override replace *.rpm $ systemctl reboot After successful reboot bash-5.1# ipa-client-install --mkhomedir -U -w Secret123 -p admin --no-sudo --no-ntp --domain ipa.test This program will set up IPA client. Version 4.9.8 Discovery was successful! Client hostname: mytest2.ipa.test Realm: IPA.TEST DNS Domain: ipa.test IPA Server: master.ipa.test BaseDN: dc=ipa,dc=test Skipping chrony configuration Successfully retrieved CA cert Subject: CN=ca,OU=sssd,O=test Issuer: CN=ca,OU=sssd,O=test Valid From: 2022-03-07 10:44:20 Valid Until: 2041-11-22 10:44:20 Subject: CN=Certificate Authority,O=IPA.TEST Issuer: CN=ca,OU=sssd,O=test Valid From: 2022-09-27 03:42:25 Valid Until: 2042-06-14 03:42:25 Enrolled in IPA realm IPA.TEST Created /etc/ipa/default.conf Configured /etc/sssd/sssd.conf Configured /etc/krb5.conf for IPA realm IPA.TEST Systemwide CA database updated. Hostname (mytest2.ipa.test) does not have A/AAAA record. Missing A/AAAA record(s) for host mytest2.ipa.test: 192.168.122.33. Missing reverse record(s) for address(es): 192.168.122.33. Adding SSH public key from /etc/ssh/ssh_host_ed25519_key.pub Adding SSH public key from /etc/ssh/ssh_host_ecdsa_key.pub Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub SSSD enabled Configured /etc/openldap/ldap.conf Configured /etc/ssh/ssh_config Configured /etc/ssh/sshd_config.d/04-ipa.conf Configuring ipa.test as NIS domain. Client configuration complete. The ipa-client-install command was successful bash-5.1# grep -E '^(passwd|group)' /etc/nsswitch.conf passwd: files sss systemd altfiles group: files sss systemd altfiles Are you sure, you had the correct authselect version installed? You can check by looking into /usr/shared/authselect/vendor - it should not be empty. bash-5.1# rpm -q authselect authselect-1.2.3-7.el9_0.1.x86_64 bash-5.1# ll /usr/share/authselect/vendor/ total 8 drwxr-xr-x. 2 root root 175 Jan 1 1970 minimal drwxr-xr-x. 2 root root 4096 Jan 1 1970 sssd drwxr-xr-x. 2 root root 4096 Jan 1 1970 winbind I bet the post install isn't the same as built time install. This is how it looks: -------------------------------------------- [root@localhost ~]# ll /usr/share/authselect/vendor/ total 0 [root@localhost ~]# rpm -qV authselect .......T. /etc/bash_completion.d/authselect-completion.sh .......T. /usr/bin/authselect .......T. d /usr/share/man/cs/man7/authselect-migration.7.gz .......T. d /usr/share/man/cs/man8/authselect.8.gz .......T. d /usr/share/man/de/man8/authselect.8.gz .......T. d /usr/share/man/es/man7/authselect-migration.7.gz .......T. d /usr/share/man/fr/man7/authselect-migration.7.gz .......T. d /usr/share/man/fr/man8/authselect.8.gz .......T. d /usr/share/man/ja/man7/authselect-migration.7.gz .......T. d /usr/share/man/man7/authselect-migration.7.gz .......T. d /usr/share/man/man8/authselect.8.gz .......T. d /usr/share/man/nl/man7/authselect-migration.7.gz .......T. d /usr/share/man/nl/man8/authselect.8.gz .......T. d /usr/share/man/ru/man7/authselect-migration.7.gz .......T. d /usr/share/man/ru/man8/authselect.8.gz .......T. d /usr/share/man/sv/man7/authselect-migration.7.gz .......T. d /usr/share/man/sv/man8/authselect.8.gz .......T. d /usr/share/man/tr/man7/authselect-migration.7.gz .......T. d /usr/share/man/tr/man8/authselect.8.gz .......T. d /usr/share/man/uk/man7/authselect-migration.7.gz .......T. d /usr/share/man/uk/man8/authselect.8.gz [root@localhost ~]# ll /usr/share/authselect/vendor/ total 0 -------------------------------------------- end of ipa-client install -------------------------------------------- SSSD enabled SSSD service restart was unsuccessful. Configured /etc/openldap/ldap.conf Unable to find 'admin' user with 'getent passwd admin'! Unable to reliably detect configuration. Check NSS setup manually. Configured /etc/ssh/ssh_config Configured /etc/ssh/sshd_config.d/04-ipa.conf sshd failed to restart: CalledProcessError(Command ['/bin/systemctl', 'restart', 'sshd.service'] returned non-zero exit status 1: 'Job for sshd.service failed because the control process exited with error code.\nSee "systemctl status sshd.service" and "journalctl -xeu sshd.service" for details.\n') Configuring cool.lab as NIS domain. Client configuration complete. The ipa-client-install command was successful -------------------------------------------- Seems the directory you mention doesn't come with rpm files listing, so likely something you create in some post script, which then possibibly doesn't get applied in edge install? oh, forgot this, it's the same version: [root@localhost ~]# rpm -qi authselect Name : authselect Version : 1.2.3 Release : 7.el9_0.1 Architecture: x86_64 Install Date: Tue Sep 27 13:12:56 2022 Group : Unspecified Size : 141465 License : GPLv3+ Signature : (none) Source RPM : authselect-1.2.3-7.el9_0.1.src.rpm Build Date : Mon Sep 26 14:23:41 2022 Build Host : x86-64-01.build.eng.rdu2.redhat.com Packager : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla> Vendor : Red Hat, Inc. URL : https://github.com/authselect/authselect Summary : Configures authentication and identity sources from supported profiles Description : Hmm. I'm using the same code that works on Silverblue, that is: # Keep nss-altfiles for all rpm-ostree based systems. # See https://github.com/authselect/authselect/issues/48 if test -e /run/ostree-booted; then for PROFILE in `ls %{_datadir}/authselect/default`; do %{_bindir}/authselect create-profile $PROFILE --vendor --base-on $PROFILE --symlink-pam --symlink-dconf --symlink=REQUIREMENTS --symlink=README &> /dev/null %__sed -ie "s/^\(passwd\|group\):\(.*\)systemd\(.*\)/\1:\2systemd altfiles\3/g" %{_datadir}/authselect/vendor/$PROFILE/nsswitch.conf &> /dev/null done fi Since it works for me during 'rpm-ostree override replace' but not for you during built time then perhaps /run/ostree-booted is not yet available during Edge compose? I'm setting need info from Colin. Today Image Builder/osbuild only uses the very last stage of rpm-ostree's build side, so it needs to reimplement https://github.com/coreos/rpm-ostree/pull/1750 to have that work as is. So should I push the authselect change and open a bug against Edge to include rpm-ostree commit? Or do you propose something else? I think you should push this fix as it is the right one. The next step is to have osbuild create /run/ostree during image builds to make that work there. Sorry, it's `/run/ostree-booted` Hi Pavel, How should we proceed with this one? Should we plan for 8.8 (kind late) or next release? What are your suggestions here? Let's do it in 8.8 when rebasing to latest version. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (authselect bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2023:3022 |
Created attachment 1877830 [details] on boot failures after ipa-client-install Description of problem: Installing the Red Hat-supported FreeIPA client onto a RHEL for Edge host results in several misconfiguration, and a host that cannot be accessed after reboot Version-Release number of selected component (if applicable): RHEL for Edge 8.5 ipa-client 4.9.6, 12.module+el8.5.0+14526+983b221b How reproducible: Every time Steps to Reproduce: 1.Create RHEL for Edge Blueprint with ipa-client and associated dependencies as added packages. 2. Create the RHEL for Edge commit, download the .tar, and follow the steps for a network-based deployment in a domain managed by IDM. 3. Install the ipa-client with ipa-client-install --mkhomedir -U -w <credential> -p <join user> --no-sudo --no-ntp 4.Note that this command works fine on a vanilla RHEL 8 host in the same domain. Actual results: 1. ipa-client installation informs the user it has completed succesfully, but the restarts are sshd and sssd fail due to missing users and groups for sshd and sssd. sssd excerpt #1: (2022-05-06 14:38:31): [nss] [sss_user_by_name_or_uid] (0x0040): [sssd] is neither a valid UID nor a user name which could be resolved by getpwnam(). sshd excerpt #1: May 06 14:21:41 jumpbox.demolab.local sshd[2597]: Privilege separation user sshd does not exist 2. private keys in /etc/ssh/ have the wrong permissions - presented as 0640. sshd excerpt (repeated for all private keys in that location) #2 May 06 14:30:16 jumpbox.demolab.local sshd[3689]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ May 06 14:30:16 jumpbox.demolab.local sshd[3689]: Permissions 0640 for '/etc/ssh/ssh_host_rsa_key' are too open. May 06 14:30:16 jumpbox.demolab.local sshd[3689]: It is required that your private key files are NOT accessible by others. May 06 14:30:16 jumpbox.demolab.local sshd[3689]: This private key will be ignored. May 06 14:30:16 jumpbox.demolab.local sshd[3689]: Unable to load host key "/etc/ssh/ssh_host_rsa_key": bad permissions May 06 14:30:16 jumpbox.demolab.local sshd[3689]: Unable to load host key: /etc/ssh/ssh_host_rsa_key 3. On host restart, several services fail on boot, and it is no longe possible to access the host either by ssh or directly via a console as the login service also fails as a result of earlier failures. Expected results: 1. ipa-client-install completes succesfully, and registers the host with the IDM infrastructure. Additional info: RHEL for Edge is of interest to my public sector customers who have very stringent access requirements in restricted / air gapped environments. Hosts built on RHEL for Edge images will make ideal on-premise / cloud utility / jumpbox hosts due to their robustness, immutability, and tightly controlled operational model. Whilst not a core use-case for the technology, it's an interesting niche for this approach to OS management. Being able to utilise IDM to manage access control on RHEL for Edge hosts would go a long way to enabling this use case.