Created attachment 1877831 [details]
ipa-client-install output

Output logs

Created attachment 1877832 [details]
sshd log output

Created attachment 1877833 [details]
sssd log output

Created attachment 1877834 [details]
ssh key permissions errors

@(In reply to Rob Crittenden from comment #5)
> Is there a reason this is private? I see no customer information listed.

Force of habit. Have rectified.

> 
> Please provide more detailed reproduction steps for 1 and 2. We have no
> experience with Edge. The more detailed the better.

Aside from working round a configuration issue - https://access.redhat.com/solutions/5773421 - I simply followed our docs to build and deploy the image:

For step 1:

Set-up -
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/composing_installing_and_managing_rhel_for_edge_images/setting-up-image-builder_composing-installing-managing-rhel-for-edge-images

Creating -
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/composing_installing_and_managing_rhel_for_edge_images/composing-rhel-for-edge-images-using-image-builder-in-rhel-web-console_composing-installing-managing-rhel-for-edge-images

I explicitly added the packages seen in the attachments 'image-builder-packages.png' and 'image-builder-full-package-list.txt'

Downloading - 
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/composing_installing_and_managing_rhel_for_edge_images/composing-a-rhel-for-edge-image-using-image-builder-command-line_composing-installing-managing-rhel-for-edge-images#downloading-a-rhel-for-edge-image-using-the-command-line_composing-a-rhel-for-edge-image-using-image-builder-command-line

Deploying - 
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/composing_installing_and_managing_rhel_for_edge_images/installing-rpm-ostree-images_composing-installing-managing-rhel-for-edge-images

Image was deployed as a VM in vSphere. It has 1 vCPU and 4GB RAM associated with it.

Created attachment 1878435 [details]
packages added explicitly to RHEL for Edge commit with image builder.

Created attachment 1878436 [details]
full list of packages and dependencies added to RHEL for Edge commit with image builder

My first attempt at reproducing this resulted in a working image. sshd and sssd are functional after ipa-client-install (and a reboot).

I believe I followed the steps, the image very much helped.

- I started with an 8.5 image
- Installed all the builder software per the docs
- registered using subscription manager (required)
- fired up cockpit and created a blueprint
- I added the packages per your image
- I created a user with system powers.
- I created a qcow2 image and used virt-manager to import that
- After first boot I ran ipa-client-install
- id admin works and sshd works
- reboot and id and sshd continue to work

I ended up with a RHEL 8.6 client image as that is the repos I got via RHSM.

I don't know if this is an 8.5 vs 8.6 issue.

(In reply to Rob Crittenden from comment #9)
> My first attempt at reproducing this resulted in a working image. sshd and
> sssd are functional after ipa-client-install (and a reboot).
> 
> I believe I followed the steps, the image very much helped.
> 
> - I started with an 8.5 image
> - Installed all the builder software per the docs
> - registered using subscription manager (required)
> - fired up cockpit and created a blueprint
> - I added the packages per your image
> - I created a user with system powers.
> - I created a qcow2 image and used virt-manager to import that
> - After first boot I ran ipa-client-install
> - id admin works and sshd works
> - reboot and id and sshd continue to work
> 
> I ended up with a RHEL 8.6 client image as that is the repos I got via RHSM.
> 
> I don't know if this is an 8.5 vs 8.6 issue.

Thanks for picking this up, Rob. Generating a qcow2 will unfortunately not give you a RHEL for Edge image. It just gives you a 'full fat' RHEL image containing the packages you added and configurations you selected in Image Builder. I double checked this by creating and downloading the QEMU qcow2 for my image. I'm sorry I didn't go into more detail on that, I should've been more specific.

As per [1], the valid image formats for deploying Edge hosts are:

* RHEL for Edge Commit
* RHEL for Edge Container
* RHEL for Edge Installer
* RHEL for Edge Raw
* RHEL for Edge Simplified Installer

With that in mind, I set up an http server, and generated a RHEL for Edge Commit with the packages I wanted - I believe it's the top option in the dropdown when selecting an image type to create.

I downloaded the commit using the composer-cli as per [2], and moved the edge commit .tar file to my http host.

I unpacked the RHEL for Edge Commit (the .tar file) onto the http server as per [3]. Sections 7.1, 7.3, and 7.4 are pertinent to me - I didn't do the containerised http host in 7.2 as I already had a friendly apache instance that I used.

I created a kickstart file (will attach to BZ) and hosted it alongside the edge commit. So I had a directory structure in /var/www/html/edge which looked like:

[bholmes@ignite edge]$ ll
total 928548
-rw-r--r--. 1 root    root          554 May 10 21:52 compose.json <-- extracted from from tar
-rw-r--r--. 1 bholmes bholmes 950824960 May 11 20:50 e7ce11c6-d3dc-41f0-b740-3a3e8e0cc0b9-commit.tar <-- the edge commit, downloaded form image builder
-rw-r--r--. 1 root    root          883 May  5 16:28 kickstart.cfg <-- self-explanatory, but hand-crafted
drwxr-xr-x. 7 root    root          102 May 10 21:52 repo <-- extracted from tar

I then used the RHEL 8 boot iso, added the 'inst.ks=http://<my http host>/edge/kickstart.cfg' parameter to the boot options, and left it alone to do its thing.

After this had deployed, I ran the ipa-client-install process as per my original comments.

[1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/composing_a_customized_rhel_system_image/composer-description_composing-a-customized-rhel-system-image#composer-output-formats_composer-description
[2] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/composing_installing_and_managing_rhel_for_edge_images/composing-a-rhel-for-edge-image-using-image-builder-command-line_composing-installing-managing-rhel-for-edge-images#downloading-a-rhel-for-edge-image-using-the-command-line_composing-a-rhel-for-edge-image-using-image-builder-command-line
[3] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/composing_installing_and_managing_rhel_for_edge_images/installing-rpm-ostree-images_composing-installing-managing-rhel-for-edge-images

Created attachment 1879353 [details]
kickstart.cfg for edge host

The issue with sssd is that the sssd user and group are missing so the sssd_nss process can't start.

This KCS contains part of the fix, adding the user https://access.redhat.com/solutions/6021931

In addition you need: groupadd -g 993 sssd

Restart sssd and this should work: getent passwd admin

They key error message for sshd is:

Privilege separation user sshd does not exist.

So similar to sssd entries are missing, in this case two users and a group. As a result the ownership of /etc/ssh/*_key is incorrect.

You need to create two groups and a user, then fix the group ownership of /etc/ssh/*_key

I'm not sure why the sshd and sssd users and groups aren't being created on installation. The sssd-common package is responsible for adding the entries. Its preinstall rpm script is:

getent group sssd >/dev/null || groupadd -r sssd
getent passwd sssd >/dev/null || useradd -r -g sssd -d / -s /sbin/nologin -c "User for sssd" sssd

Similarly openssh-server has this as preinstall:

getent group sshd >/dev/null || groupadd -g 74 -r sshd || :
getent passwd sshd >/dev/null || \
  useradd -c "Privilege-separated SSH" -u 74 -g sshd \
  -s /sbin/nologin -r -d /var/empty/sshd sshd 2> /dev/null || :

And finally openssh has this:

getent group ssh_keys >/dev/null || groupadd -r ssh_keys || :

So if you run these user/groupadd commands to create the passwd/group entries then you just need to fixup the permissions and start the service:

chgrp ssh_keys /etc/ssh/*_key
systemctl start sshd

I think it is common problem for container-alike images which built from RPMs without running the post-install scripts. Any of the scripts that create resources to be present in the final image would not be run as expected.
See https://github.com/coreos/rpm-ostree/issues/1884 for a long standing issue. Nobody took the stack of packages that comprises FreeIPA dependencies to convert their post-install scripts into something that runs as a systemd unit yet.

Technically, this should be possible just fine with RPM because %post and similar macros take '-p' argument to specify the shell to process the script (/bin/sh by default). It could be changed be some wrapper that creates a one-off unit to run.

However, the bigger problem is that it doesn't solve the problem at all. CoreOS claims to run post-install scripts and triggers during image build phase: https://coreos.github.io/rpm-ostree/architecture-core/#overall-architecture, so it should not be required to create these systemd units, which contradicts what cgwalters said in the github issue above. Creating systemd unit would also not help as it would be run on the system and effectively would require those scripts to be arranged to run before anything useful to be done with IPA on the system itself. We definitely need to maintain some coherency with the set of UIDs/GIDs for system users (not just names) to survive across multiple boots.

Changing component to rpm-ostree. This is not something individual package owners can manage.

IIUC, we're saying that RPM preinstall/postinstall scripts are not being run when Image Builder/osbuild does an rpm-ostree compose?

To test, I have created the sshd / sssd users and groups as per Rob's guidance:

[root@jumpbox ~]# cat /etc/passwd 
root:x:0:0:root:/root:/bin/bash
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
sssd:x:996:993:User for sssd:/:/sbin/nologin

[root@jumpbox ~]# cat /etc/group
root:x:0:
wheel:x:10:root
sshd:x:74:
sssd:x:993:

And fixed the ssh keys:

[root@jumpbox ~]# ls -la /etc/ssh/
total 612
drwxr-xr-x.  3 root root        245 May 17 14:32 .
drwxr-xr-x. 92 root root       8192 May 17 14:48 ..
-rw-r--r--.  1 root root     577388 May 17 14:31 moduli
-rw-r--r--.  1 root root       1770 May 17 14:31 ssh_config
drwxr-xr-x.  2 root root         28 May 17 14:31 ssh_config.d
-rw-------.  1 root ssh_keys    480 May 17 14:32 ssh_host_ecdsa_key
-rw-r--r--.  1 root root        162 May 17 14:32 ssh_host_ecdsa_key.pub
-rw-------.  1 root ssh_keys    387 May 17 14:32 ssh_host_ed25519_key
-rw-r--r--.  1 root root         82 May 17 14:32 ssh_host_ed25519_key.pub
-rw-------.  1 root ssh_keys   2578 May 17 14:32 ssh_host_rsa_key
-rw-r--r--.  1 root root        554 May 17 14:32 ssh_host_rsa_key.pub
-rw-------.  1 root root       4269 May 17 14:31 sshd_config

I ran the ipa-client-install:

ipa-client-install --mkhomedir -U -w <credential> -p <credential> --no-sudo --no-ntp

This allowed me to complete the installation of the ipa client succesfully, and log into the host as a user managed by IDM which is great.

However, when rebooting the host, the host fails to restart cleanly / successfully due to a number of failed services (screenshot attached). When the host finally reaches the login prompt, it is inaccessible. I'll try and get access to extract logs.

Did you experience this Rob?

Created attachment 1880481 [details]
boot failures after successfull ipa-client-install

As per previous comment, even after a successful installation of the ipa client, the host becomes inaccessible after a reboot.

I tried just now and get similar behavior. I booted into single-user mode and the dbus user is missing which is probably causing cascading failures.

I think this is just another side-effect of rpm %post not being executed, in this case dbus-daemon.

Given the initial description, I suspect that the IPA installation is removing the `altfiles` modules from `/etc/nsswitch.conf` which breaks user/group resolution on the system.

From an RHCOS node (might be slightly different on RHEL Edge):

```
$ grep altfiles /etc/nsswitch.conf
passwd: sss files altfiles systemd
group: sss files altfiles systemd
```

Workaround, if this is the issue, is to re-add `altfiles` to `/etc/nsswitch.conf`.

This is likely a bug / missing backport in RHEL authselect. See:
- https://github.com/authselect/authselect/pull/273
- https://discussion.fedoraproject.org/t/systemd-resolved-service-failed-to-determine-user-credentials-no-such-process/33240/4
- https://pagure.io/fedora-kde/SIG/issue/152

IPA is not touching anything in /etc/nsswitch.conf itself.

As far as I can see, authselect in RHEL does not have support for rpm-ostree, so the pull request you talk about is not present in any RHEL version.

Thimothee, could you please open a separate bug for authselect and link it here? I don't want to move this bug to authselect because there are might be more changes needed in other components as well and it is better to have a tracker for those (this bug).

Given the reference to SSSD, I'd like to bring up https://github.com/SSSD/sssd/issues/6107 which got fixed in 8.6 but may still be at play here.

By default, SSSD runs as root. When you enroll a host into IPA deployment, ipa-client-install does not configure SSSD to run as unprivileged user, so this does not apply at all.

Fresh 8.6-based RHEL for Edge deployment

# rpm-ostree status
State: idle
Deployments:
● rhel:rhel/8/x86_64/edge
                   Version: 8.6 (2022-05-19T20:55:20Z)
                    Commit: 921b293d7f9a0dea5660e5daca925352f8d84caa6af9f8106dd313c5d998b590


# rpm -q ostree rpm-ostree sssd-client ipa-client glibc nss-altfiles
ostree-2022.1-2.el8.x86_64
rpm-ostree-2022.2-2.el8.x86_64
sssd-client-2.6.2-4.el8_6.x86_64
ipa-client-4.9.8-6.module+el8.6.0+14300+0c339766.x86_64
glibc-2.28-189.1.el8.x86_64
nss-altfiles-2.18.1-12.el8.x86_64

# id sssd
uid=995(sssd) gid=992(sssd) groups=992(sssd)

# id sshd
uid=74(sshd) gid=74(sshd) groups=74(sshd)


# grep alt/etc/nsswitch.conf
# Commonly used alternative service providers (may need installation):
passwd: sss files altfiles systemd
group: sss files altfiles systemd


I have no IDM domain to test the ipa-client-install against, but it looks like everything is configured as expected before joining a domain.

You can try to enroll it to the public demo we have: https://www.freeipa.org/page/Demo

(In reply to Alexander Bokovoy from comment #22)
> Thimothee, could you please open a separate bug for authselect and link it
> here? I don't want to move this bug to authselect because there are might be
> more changes needed in other components as well and it is better to have a
> tracker for those (this bug).

Can you verify that my guess is correct? If it is, then this is not a bug in rpm-ostree and we can move this one to authselect.

We will answer other bugs as they come in: one issue per bug.

post-run of ipa-client-install passwd and group lack altfiles in nsswitch.conf.

Restoring them allows lookup of the missing entities and sshd and sssd start successfully.

Thanks. Can you also verify if it affects RHEL Edge 8.6 like Derrick tried? Is using RHEL 8.6 an option for you? 8.5 should be EOL now if I'm not mistaken.

authselect folks: see https://bugzilla.redhat.com/show_bug.cgi?id=2082845#c21 for the summary

Do I understand it correctly that RHE Edge is equivalent to Fedora Silverblue?

I can provide patch for authselect that we use on Fedora https://src.fedoraproject.org/rpms/authselect/blob/rawhide/f/authselect.spec#_315. However, as far as I can tell there is also some override in Silverblue upstream.

(In reply to Pavel Březina from comment #31)
> Do I understand it correctly that RHE Edge is equivalent to Fedora
> Silverblue?
> 

Yes, it is a rpm-ostree-based RHEL variant.  Fedora Silverblue is a rpm-ostree variant of Fedora.

(In reply to Pavel Březina from comment #31)
> I can provide patch for authselect that we use on Fedora
> https://src.fedoraproject.org/rpms/authselect/blob/rawhide/f/authselect.
> spec#_315. However, as far as I can tell there is also some override in
> Silverblue upstream.

I think this should work.

To test it:
1. Do the setup as before and stop before the IPA setup.
2. Use `rpm-ostree override replace ./authselect*.rpm` and reboot
3. Do the IPA setup
4. Confirm that altfiles is still in `/etc/nsswitch.conf`

As per request from @aboscatt, updating the ticket to keep it alive.

Currently waiting for SaaS Image Builder to support RHEL 8.6 before rebuilding my currently affected hosts.

The scratch build is for 8.5 so you should be able to test it without 8.6 support. It was not yet pushed to dist git, it is *just a scratch build*. I am waiting for positive feedback to know that the issue has been resolved.

Hi Pavel,

I was talking to Ben earlier today and, he shared the feedback below, maybe it is worth it to let it registered here.

==============================================
It's not just the users for sssd / sshd that are missing. Any rpm that is installed against and Edge 8.5 host is missing its users

I've had to add the following users to my edge host in order for it to function correctly:

```
        - "sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin"
        - "sssd:x:996:993:User for sssd:/:/sbin/nologin" 
        - "dbus:x:81:81:System message bus:/:/sbin/nologin" 
        - "polkitd:x:998:996:User for polkitd:/:/sbin/nologin"
        - "chrony:x:992:989::/var/lib/chrony:/sbin/nologin"
```
(extract from a playbook I'm using to fix up my Edge 8.5 hosts)
==============================================

Do you have any guesses if those are related to authselect itself, or it should be addressed somewhere else? I will leave it to Ben to add more details if needed, and provide feedback from the scratch build.

Kindly

The problem in authselect is that is is missing nss-altfiles module which is required on rpm-ostree systems. If these users are in altfiles location then yes, it is authselect issue which should be fixed by given scratch build. If these users should be in /etc/passwd then no, it is not related to authselect.

New scratch build link: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=46455806

Please, provide feedback. Once I get positive feedback, I will push the build to rhel.

Please follow the instructions from https://bugzilla.redhat.com/show_bug.cgi?id=2082845#c34 to test this package so that we can move forward.

Can you try following the instructions from https://bugzilla.redhat.com/show_bug.cgi?id=2082845#c34 with the temporary package build to verify the fix?

Hi,

I face this problem on my test lab. I can test this for you, given that I get provided rhel9 rpm for it. Or I can also just patch the system manually if given what to do. If you want, I can also build you a test edge image for vmware or kvm, if it helps.

I have now rhel9-edge image running in our lab on vmware, which faces this altfiles problem. Log is telling sssd can't find users that exist in /usr/lib/passwd.

I have also IdM which I can test against.

Unfortunately it still fails getting service users.

I tested this now. This time it ends like this:

Enrolled in IPA realm COOL.LAB
Created /etc/ipa/default.conf
Configured /etc/sssd/sssd.conf
Configured /etc/krb5.conf for IPA realm COOL.LAB
Systemwide CA database updated.
Hostname (rh-edge-01.cool.lab) does not have A/AAAA record.
Missing reverse record(s) for address(es): 10.128.1.137.
Adding SSH public key from /etc/ssh/ssh_host_ed25519_key.pub
Adding SSH public key from /etc/ssh/ssh_host_ecdsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub
SSSD enabled
SSSD service restart was unsuccessful.
Configured /etc/openldap/ldap.conf
Unable to find 'admin' user with 'getent passwd admin'!
Unable to reliably detect configuration. Check NSS setup manually.
Configured /etc/ssh/ssh_config
Configured /etc/ssh/sshd_config.d/04-ipa.conf
sshd failed to restart: CalledProcessError(Command ['/bin/systemctl', 'restart', 'sshd.service'] returned non-zero exit status 1: 'Job for sshd.service failed because the control process exited with error code.\nSee "systemctl status sshd.service" and "journalctl -xeu sshd.service" for details.\n')
Configuring cool.lab as NIS domain.
Client configuration complete.
The ipa-client-install command was successful
[root@localhost ~]# rpm -qi authselect
Name        : authselect
Version     : 1.2.3
Release     : 7.el9_0.1
Architecture: x86_64
Install Date: Tue Sep 27 13:12:56 2022
Group       : Unspecified
Size        : 141465
License     : GPLv3+
Signature   : (none)
Source RPM  : authselect-1.2.3-7.el9_0.1.src.rpm
Build Date  : Mon Sep 26 14:23:41 2022
Build Host  : x86-64-01.build.eng.rdu2.redhat.com
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
Vendor      : Red Hat, Inc.
URL         : https://github.com/authselect/authselect
Summary     : Configures authentication and identity sources from supported profiles
Description :
Authselect is designed to be a replacement for authconfig but it takes
a different approach to configure the system. Instead of letting
the administrator build the PAM stack with a tool (which may potentially
end up with a broken configuration), it would ship several tested stacks
(profiles) that solve a use-case and are well tested and supported.
At the same time, some obsolete features of authconfig are not
supported by authselect.

here is what chrony complains:

Continue to configure the system with these values? [no]: yes
Synchronizing time
Configuration of chrony was changed by installer.
CalledProcessError(Command ['/bin/systemctl', 'restart', 'chronyd.service'] returned non-zero exit status 1: 'Job for chronyd.service failed because the control process exited with error code.\nSee "systemctl status chronyd.service" and "journalctl -xeu chronyd.service" for details.\n')
The ipa-client-install command failed. See /var/log/ipaclient-install.log for more information
[root@localhost ~]# systemctl status chronyd
× chronyd.service - NTP client/server
     Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Tue 2022-09-27 14:39:12 EEST; 20s ago
       Docs: man:chronyd(8)
             man:chrony.conf(5)
    Process: 9141 ExecStart=/usr/sbin/chronyd $OPTIONS (code=exited, status=1/FAILURE)
        CPU: 18ms

Sep 27 14:39:12 rh-edge-01.cool.lab systemd[1]: Starting NTP client/server...
Sep 27 14:39:12 rh-edge-01.cool.lab chronyd[9143]: chronyd version 4.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SI>
Sep 27 14:39:12 rh-edge-01.cool.lab chronyd[9143]: Fatal error : Could not get user/group ID of chrony
Sep 27 14:39:12 rh-edge-01.cool.lab chronyd[9141]: Could not get user/group ID of chrony
Sep 27 14:39:12 rh-edge-01.cool.lab systemd[1]: chronyd.service: Control process exited, code=exited, status=1/FAILURE
Sep 27 14:39:12 rh-edge-01.cool.lab systemd[1]: chronyd.service: Failed with result 'exit-code'.
Sep 27 14:39:12 rh-edge-01.cool.lab systemd[1]: Failed to start NTP client/server.

It works for me.

I used the image you given me and downloaded packages from http://brew-task-repos.usersys.redhat.com/repos/scratch/pbrezina/authselect/1.2.3/7.el9_0.1/x86_64/ (without srpm).
Then:
$ rpm-ostree override replace *.rpm
$ systemctl reboot

After successful reboot

bash-5.1# ipa-client-install --mkhomedir -U -w Secret123 -p admin --no-sudo --no-ntp --domain ipa.test
This program will set up IPA client.
Version 4.9.8

Discovery was successful!
Client hostname: mytest2.ipa.test
Realm: IPA.TEST
DNS Domain: ipa.test
IPA Server: master.ipa.test
BaseDN: dc=ipa,dc=test

Skipping chrony configuration
Successfully retrieved CA cert
    Subject:     CN=ca,OU=sssd,O=test
    Issuer:      CN=ca,OU=sssd,O=test
    Valid From:  2022-03-07 10:44:20
    Valid Until: 2041-11-22 10:44:20

    Subject:     CN=Certificate Authority,O=IPA.TEST
    Issuer:      CN=ca,OU=sssd,O=test
    Valid From:  2022-09-27 03:42:25
    Valid Until: 2042-06-14 03:42:25

Enrolled in IPA realm IPA.TEST
Created /etc/ipa/default.conf
Configured /etc/sssd/sssd.conf
Configured /etc/krb5.conf for IPA realm IPA.TEST
Systemwide CA database updated.
Hostname (mytest2.ipa.test) does not have A/AAAA record.
Missing A/AAAA record(s) for host mytest2.ipa.test: 192.168.122.33.
Missing reverse record(s) for address(es): 192.168.122.33.
Adding SSH public key from /etc/ssh/ssh_host_ed25519_key.pub
Adding SSH public key from /etc/ssh/ssh_host_ecdsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub
SSSD enabled
Configured /etc/openldap/ldap.conf
Configured /etc/ssh/ssh_config
Configured /etc/ssh/sshd_config.d/04-ipa.conf
Configuring ipa.test as NIS domain.
Client configuration complete.
The ipa-client-install command was successful

bash-5.1# grep -E '^(passwd|group)' /etc/nsswitch.conf
passwd:     files sss systemd altfiles
group:      files sss systemd altfiles


Are you sure, you had the correct authselect version installed? You can check by looking into /usr/shared/authselect/vendor - it should not be empty.

bash-5.1# rpm -q authselect
authselect-1.2.3-7.el9_0.1.x86_64

bash-5.1# ll /usr/share/authselect/vendor/
total 8
drwxr-xr-x. 2 root root  175 Jan  1  1970 minimal
drwxr-xr-x. 2 root root 4096 Jan  1  1970 sssd
drwxr-xr-x. 2 root root 4096 Jan  1  1970 winbind

I bet the post install isn't the same as built time install. This is how it looks:

--------------------------------------------
[root@localhost ~]# ll /usr/share/authselect/vendor/
total 0
[root@localhost ~]# rpm -qV authselect
.......T.    /etc/bash_completion.d/authselect-completion.sh
.......T.    /usr/bin/authselect
.......T.  d /usr/share/man/cs/man7/authselect-migration.7.gz
.......T.  d /usr/share/man/cs/man8/authselect.8.gz
.......T.  d /usr/share/man/de/man8/authselect.8.gz
.......T.  d /usr/share/man/es/man7/authselect-migration.7.gz
.......T.  d /usr/share/man/fr/man7/authselect-migration.7.gz
.......T.  d /usr/share/man/fr/man8/authselect.8.gz
.......T.  d /usr/share/man/ja/man7/authselect-migration.7.gz
.......T.  d /usr/share/man/man7/authselect-migration.7.gz
.......T.  d /usr/share/man/man8/authselect.8.gz
.......T.  d /usr/share/man/nl/man7/authselect-migration.7.gz
.......T.  d /usr/share/man/nl/man8/authselect.8.gz
.......T.  d /usr/share/man/ru/man7/authselect-migration.7.gz
.......T.  d /usr/share/man/ru/man8/authselect.8.gz
.......T.  d /usr/share/man/sv/man7/authselect-migration.7.gz
.......T.  d /usr/share/man/sv/man8/authselect.8.gz
.......T.  d /usr/share/man/tr/man7/authselect-migration.7.gz
.......T.  d /usr/share/man/tr/man8/authselect.8.gz
.......T.  d /usr/share/man/uk/man7/authselect-migration.7.gz
.......T.  d /usr/share/man/uk/man8/authselect.8.gz

[root@localhost ~]# ll /usr/share/authselect/vendor/
total 0
--------------------------------------------

end of ipa-client install
--------------------------------------------
SSSD enabled
SSSD service restart was unsuccessful.
Configured /etc/openldap/ldap.conf
Unable to find 'admin' user with 'getent passwd admin'!
Unable to reliably detect configuration. Check NSS setup manually.
Configured /etc/ssh/ssh_config
Configured /etc/ssh/sshd_config.d/04-ipa.conf
sshd failed to restart: CalledProcessError(Command ['/bin/systemctl', 'restart', 'sshd.service'] returned non-zero exit status 1: 'Job for sshd.service failed because the control process exited with error code.\nSee "systemctl status sshd.service" and "journalctl -xeu sshd.service" for details.\n')
Configuring cool.lab as NIS domain.
Client configuration complete.
The ipa-client-install command was successful

--------------------------------------------


Seems the directory you mention doesn't come with rpm files listing, so likely something you create in some post script, which then possibibly doesn't get applied in edge install?

oh, forgot this, it's the same version:

[root@localhost ~]# rpm -qi authselect
Name        : authselect
Version     : 1.2.3
Release     : 7.el9_0.1
Architecture: x86_64
Install Date: Tue Sep 27 13:12:56 2022
Group       : Unspecified
Size        : 141465
License     : GPLv3+
Signature   : (none)
Source RPM  : authselect-1.2.3-7.el9_0.1.src.rpm
Build Date  : Mon Sep 26 14:23:41 2022
Build Host  : x86-64-01.build.eng.rdu2.redhat.com
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
Vendor      : Red Hat, Inc.
URL         : https://github.com/authselect/authselect
Summary     : Configures authentication and identity sources from supported profiles
Description :

Hmm. I'm using the same code that works on Silverblue, that is:

# Keep nss-altfiles for all rpm-ostree based systems.
# See https://github.com/authselect/authselect/issues/48
if test -e /run/ostree-booted; then
    for PROFILE in `ls %{_datadir}/authselect/default`; do
        %{_bindir}/authselect create-profile $PROFILE --vendor --base-on $PROFILE --symlink-pam --symlink-dconf --symlink=REQUIREMENTS --symlink=README &> /dev/null
        %__sed -ie "s/^\(passwd\|group\):\(.*\)systemd\(.*\)/\1:\2systemd altfiles\3/g" %{_datadir}/authselect/vendor/$PROFILE/nsswitch.conf &> /dev/null
    done
fi

Since it works for me during 'rpm-ostree override replace' but not for you during built time then perhaps /run/ostree-booted is not yet available during Edge compose? I'm setting need info from Colin.

Today Image Builder/osbuild only uses the very last stage of rpm-ostree's build side, so it needs to reimplement https://github.com/coreos/rpm-ostree/pull/1750 to have that work as is.

So should I push the authselect change and open a bug against Edge to include rpm-ostree commit? Or do you propose something else?

I think you should push this fix as it is the right one.
The next step is to have osbuild create /run/ostree during image builds to make that work there.

Sorry, it's `/run/ostree-booted`

Filed https://github.com/osbuild/osbuild/issues/1154

Hi Pavel,

How should we proceed with this one? Should we plan for 8.8 (kind late) or next release? What are your suggestions here?

Let's do it in 8.8 when rebasing to latest version.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (authselect bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:3022