Bug 2291235 - FreeIPA upgrade test from Fedora 39 fails since Rawhide 20240608.n.1
Summary: FreeIPA upgrade test from Fedora 39 fails since Rawhide 20240608.n.1
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: pam
Version: rawhide
Hardware: Unspecified
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Iker Pedrosa
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: openqa AcceptedBlocker
Depends On:
Blocks: F41BetaBlocker
TreeView+ depends on / blocked
 
Reported: 2024-06-10 22:32 UTC by Adam Williamson
Modified: 2024-07-02 15:45 UTC (History)
21 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2024-07-02 09:33:35 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
log tarball (5.93 MB, application/octet-stream)
2024-06-10 22:34 UTC, Adam Williamson
no flags Details

Description Adam Williamson 2024-06-10 22:32:58 UTC
I noticed the openQA FreeIPA server upgrade2 test is failing since sssd-2.10.0~beta1-1.fc41 landed in 20240608.n.1 . The one-release upgrade test (F40 to Rawhide) is fine, but F39 to Rawhide fails.

After the upgrade process is complete, on the upgraded server, ipa.service doesn't start. ipaupgrade.log shows the sssd service failing. sssd journal messages just say:

Jun 10 11:31:15 ipa001.test.openqa.fedoraproject.org systemd[1]: Starting sssd.service - System Security Services Daemon...
Jun 10 11:31:15 ipa001.test.openqa.fedoraproject.org sssd[24779]: Starting up
Jun 10 11:31:15 ipa001.test.openqa.fedoraproject.org sssd_be[24781]: Starting up
Jun 10 11:31:15 ipa001.test.openqa.fedoraproject.org sssd_be[24783]: Starting up
Jun 10 11:31:17 ipa001.test.openqa.fedoraproject.org sssd_be[24785]: Starting up
Jun 10 11:31:22 ipa001.test.openqa.fedoraproject.org sssd_be[24787]: Starting up
Jun 10 11:31:22 ipa001.test.openqa.fedoraproject.org sssd[24779]: Exiting the SSSD. Could not restart critical service [test.openqa.fedoraproject.org].
Jun 10 11:31:22 ipa001.test.openqa.fedoraproject.org systemd[1]: sssd.service: Main process exited, code=exited, status=1/FAILURE
Jun 10 11:31:22 ipa001.test.openqa.fedoraproject.org systemd[1]: sssd.service: Failed with result 'exit-code'.
Jun 10 11:31:22 ipa001.test.openqa.fedoraproject.org systemd[1]: Failed to start sssd.service - System Security Services Daemon.

but the sssd log files themselves are very verbose, so much so that I can't see exactly what the problem is. Will attach a tarball of the var/log/sssd directory.

Reproducible: Always

Comment 1 Adam Williamson 2024-06-10 22:34:16 UTC
Created attachment 2036936 [details]
log tarball

This is a tarball of /var/log from an affected test. /var/log/sssd should have all the relevant info.

Comment 2 Adam Williamson 2024-06-10 22:35:27 UTC
Proposing as a Beta blocker as a violation of https://fedoraproject.org/wiki/Fedora_41_Beta_Release_Criteria#Server_upgrade_requirements - "It must be possible to successfully complete a direct upgrade from a fully updated installation of each of the last two stable Fedora Server releases with the system configured as a FreeIPA domain controller or postgresql server as specified in the relevant criteria. The upgraded system must meet all relevant release criteria, including criteria relating to functionality of the server software."

Comment 3 Andre Robatino 2024-06-10 23:10:26 UTC
Noticed that it was a F40 Beta Blocker and assumed that was a mistake, but then saw this is concerning updates. If it was correct then please set it back to the original blocker.

Comment 4 Andre Robatino 2024-06-10 23:14:34 UTC
Sorry, I meant upgrades, not updates.

Comment 5 Adam Williamson 2024-06-11 06:16:53 UTC
no, the issue is that somehow I didn't remember to update the trackers when f40 was released, so 'betablocker' is still f40...will fix that now.

Comment 6 Adam Williamson 2024-06-11 06:28:01 UTC
...I forgot to say, thanks for spotting it!

Comment 7 Alexander Bokovoy 2024-06-11 06:31:38 UTC
Looks like it wasn't able to read the keytab:

   *  (2024-06-10 14:31:15): [be[test.openqa.fedoraproject.org]] [ipa_get_id_options] (0x0400): Option ldap_search_base set to cn=accounts,dc=test,dc=openqa,dc=fedoraproject,dc=org
   *  (2024-06-10 14:31:15): [be[test.openqa.fedoraproject.org]] [common_parse_search_base] (0x0100): Search base added: [DEFAULT][cn=accounts,dc=test,dc=openqa,dc=fedoraproject,dc=org][SUBTREE][]
   *  (2024-06-10 14:31:15): [be[test.openqa.fedoraproject.org]] [ipa_get_id_options] (0x0400): Option krb5_realm set to TEST.OPENQA.FEDORAPROJECT.ORG
   *  (2024-06-10 14:31:15): [be[test.openqa.fedoraproject.org]] [sdap_set_sasl_options] (0x0100): Will look for ipa001.test.openqa.fedoraproject.org.FEDORAPROJECT.ORG in default keytab
   *  (2024-06-10 14:31:15): [be[test.openqa.fedoraproject.org]] [create_child_req_send_buffer] (0x0400): buffer size: 93
   *  (2024-06-10 14:31:15): [be[test.openqa.fedoraproject.org]] [sdap_select_principal_from_keytab_sync] (0x0020): Failed to get principal from keytab (sss_atomic_read_s() failed), see ldap_child.log (pid = 24782) for details.
********************** BACKTRACE DUMP ENDS HERE *********************************
   
(2024-06-10 14:31:15): [be[test.openqa.fedoraproject.org]] [ipa_get_id_options] (0x0040): Cannot set the SASL-related options
(2024-06-10 14:31:15): [be[test.openqa.fedoraproject.org]] [ipa_init_id_ctx] (0x0020): Unable to init id context [5]: Input/output error
********************** PREVIOUS MESSAGE WAS TRIGGERED BY THE FOLLOWING BACKTRACE:
   *  (2024-06-10 14:31:15): [be[test.openqa.fedoraproject.org]] [ipa_get_id_options] (0x0040): Cannot set the SASL-related options
   *  (2024-06-10 14:31:15): [be[test.openqa.fedoraproject.org]] [ipa_init_id_ctx] (0x0020): Unable to init id context [5]: Input/output error
********************** BACKTRACE DUMP ENDS HERE *********************************
   
(2024-06-10 14:31:15): [be[test.openqa.fedoraproject.org]] [sssm_ipa_init] (0x0020): Unable to init IPA ID context [5]: Input/output error
(2024-06-10 14:31:15): [be[test.openqa.fedoraproject.org]] [dp_module_run_constructor] (0x0010): Module [ipa] constructor failed [5]: Input/output error
********************** PREVIOUS MESSAGE WAS TRIGGERED BY THE FOLLOWING BACKTRACE:
   *  (2024-06-10 14:31:15): [be[test.openqa.fedoraproject.org]] [sssm_ipa_init] (0x0020): Unable to init IPA ID context [5]: Input/output error
   *  (2024-06-10 14:31:15): [be[test.openqa.fedoraproject.org]] [dp_module_run_constructor] (0x0010): Module [ipa] constructor failed [5]: Input/output error
********************** BACKTRACE DUMP ENDS HERE *********************************

ldap_child.log is simply empty.

Comment 8 Alexander Bokovoy 2024-06-11 06:58:28 UTC
It was OK at 14:14:16:

Jun 10 14:14:16 ipa001 systemd[1]: Starting sssd.service - System Security Services Daemon...
Jun 10 14:14:16 ipa001 sssd[9257]: Starting up
Jun 10 14:14:16 ipa001 sssd_be[9258]: Starting up
Jun 10 14:14:16 ipa001 sssd_ssh[9263]: Starting up
Jun 10 14:14:16 ipa001 sssd_sudo[9264]: Starting up
Jun 10 14:14:16 ipa001 sssd_nss[9259]: Starting up
Jun 10 14:14:16 ipa001 sssd_ifp[9262]: Starting up
Jun 10 14:14:16 ipa001 sssd_pam[9260]: Starting up
Jun 10 14:14:16 ipa001 sssd_pac[9265]: Starting up
Jun 10 14:14:16 ipa001 systemd[1]: Started sssd.service - System Security Services Daemon.

Stopped during system upgrade at 14:21:00:

Jun 10 14:21:00 ipa001 sssd_ifp[9262]: Shutting down (status = 0)
Jun 10 14:21:00 ipa001 sssd_pam[9260]: Shutting down (status = 0)
Jun 10 14:21:00 ipa001 sssd_ssh[9263]: Shutting down (status = 0)
Jun 10 14:21:00 ipa001 sssd_sudo[9264]: Shutting down (status = 0)
Jun 10 14:21:00 ipa001 sssd_pac[9265]: Shutting down (status = 0)
Jun 10 14:21:00 ipa001 sssd_nss[9259]: Shutting down (status = 0)
Jun 10 14:21:00 ipa001 systemd[1]: Stopping systemd-homed-activate.service - Home Area Activation...
Jun 10 14:21:00 ipa001 sssd_be[9258]: Shutting down (status = 0)
Jun 10 14:21:00 ipa001 systemd[1]: Started plymouth-reboot.service - Show Plymouth Reboot Screen.
Jun 10 14:21:00 ipa001 systemd[1]: systemd-homed-activate.service: Deactivated successfully.
Jun 10 14:21:00 ipa001 systemd[1]: Stopped systemd-homed-activate.service - Home Area Activation.
Jun 10 14:21:00 ipa001 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=plymouth-reboot comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=suc
Jun 10 14:21:00 ipa001 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-homed-activate comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? r
Jun 10 14:21:00 ipa001 systemd[1]: sssd.service: Deactivated successfully.
Jun 10 14:21:00 ipa001 systemd[1]: Stopped sssd.service - System Security Services Daemon.

Upgrade happened at 14:23:

Jun 10 14:23:12 ipa001 dnf-3[716]:  Upgrading        : sssd-common-pac-2.10.0~beta1-1.fc41.x86_64        430/2052
Jun 10 14:23:12 ipa001 dnf-3[716]:  Upgrading        : sssd-krb5-common-2.10.0~beta1-1.fc41.x86_64       431/2052
Jun 10 14:23:12 ipa001 dnf-3[716]: warning: group sssd does not exist - using root
Jun 10 14:23:12 ipa001 dnf-3[716]: warning: group sssd does not exist - using root
Jun 10 14:23:12 ipa001 dnf-3[716]: warning: user sssd does not exist - using root
Jun 10 14:23:12 ipa001 dnf-3[716]: warning: group sssd does not exist - using root
Jun 10 14:23:12 ipa001 dbus-broker-launch[715]: Noticed file-system modification, trigger reload.
Jun 10 14:23:12 ipa001 dbus-broker-launch[715]: Noticed file-system modification, trigger reload.
Jun 10 14:23:13 ipa001 dbus-broker-launch[715]: Noticed file-system modification, trigger reload.
Jun 10 14:23:13 ipa001 dnf-3[716]:  Upgrading        : libfprint-1.94.6-1.fc40.x86_64                    432/2052
Jun 10 14:23:13 ipa001 dnf-3[716]:  Upgrading        : fprintd-1.94.2-11.fc40.x86_64                     433/2052
Jun 10 14:23:13 ipa001 dnf-3[716]:  Upgrading        : python-pip-wheel-24.0-2.fc41.noarch               434/2052
Jun 10 14:23:13 ipa001 dnf-3[716]:  Upgrading        : libssh-0.10.6-6.fc41.x86_64                       435/2052
Jun 10 14:23:13 ipa001 dnf-3[716]:  Upgrading        : python3-sssdconfig-2.10.0~beta1-1.fc41.noarch     436/2052
Jun 10 14:23:13 ipa001 dnf-3[716]:  Upgrading        : sssd-ad-2.10.0~beta1-1.fc41.x86_64                437/2052
Jun 10 14:23:13 ipa001 dnf-3[716]:  Upgrading        : sssd-ipa-2.10.0~beta1-1.fc41.x86_64               438/2052
Jun 10 14:23:13 ipa001 dnf-3[716]: warning: group sssd does not exist - using root
Jun 10 14:23:13 ipa001 dnf-3[716]: warning: user sssd does not exist - using root
Jun 10 14:23:13 ipa001 dnf-3[716]: warning: group sssd does not exist - using root
Jun 10 14:23:13 ipa001 dnf-3[716]:  Upgrading        : sssd-krb5-2.10.0~beta1-1.fc41.x86_64              439/2052
Jun 10 14:23:13 ipa001 dnf-3[716]:  Upgrading        : sssd-ldap-2.10.0~beta1-1.fc41.x86_64              440/2052
Jun 10 14:23:13 ipa001 dnf-3[716]:  Upgrading        : sssd-proxy-2.10.0~beta1-1.fc41.x86_64             441/2052

upgrade continues, so by 14:31:15 we were still in the upgrade path and at 14:32:55 we reboot second time:

Jun 10 14:32:55 ipa001 kernel: Linux version 6.10.0-0.rc2.20240607git8a92980606e3.28.fc41.x86_64 (mockbuild@7a76e6232d034e82aa1b8e85988de8da) (gcc (GCC) 14.1.1 20240522 (Red Hat 14.1.1-4), GNU ld version 2.42.50.20240513) #1 SMP PR

SSSD start is initiated at 14:33:01:

Jun 10 14:33:01 ipa001 systemd[1]: Starting sssd.service - System Security Services Daemon...
Jun 10 14:33:02 ipa001 sssd[802]: Starting up
Jun 10 14:33:02 ipa001 sssd_be[817]: Starting up
Jun 10 14:33:02 ipa001 sssd_be[832]: Starting up
Jun 10 14:33:04 ipa001 sssd_be[1003]: Starting up
Jun 10 14:33:08 ipa001 sssd_be[1065]: Starting up
Jun 10 14:33:08 ipa001 sssd[802]: Exiting the SSSD. Could not restart critical service [test.openqa.fedoraproject.org].
Jun 10 14:33:08 ipa001 systemd[1]: sssd.service: Main process exited, code=exited, status=1/FAILURE
Jun 10 14:33:08 ipa001 systemd[1]: sssd.service: Failed with result 'exit-code'.
Jun 10 14:33:08 ipa001 systemd[1]: Failed to start sssd.service - System Security Services Daemon.

It looks like something has happened to the /etc/krb5.keytab, at the very least...

Comment 9 Alexander Bokovoy 2024-06-11 07:18:39 UTC
It looks like SSSD switched to unprivileged user by default?

{
        "__SEQNUM" : "27000",
        "_EXE" : "/usr/sbin/sssd",
        "_MACHINE_ID" : "ac6ac1dda4124e289a775a455994e591",
        "_CMDLINE" : "/usr/sbin/sssd -i --logger=files",
        "__CURSOR" : "s=6550fc39cd77421c8237913b111b15a0;i=6978;b=f2480929821e4562bbd4d6853c840d1a;m=51d1044;t=61a8d65b3d553;x=3bcbd9ed0fc28e74",
        "PRIORITY" : "3",
        "_SYSTEMD_UNIT" : "sssd.service",
        "_TRANSPORT" : "journal",
        "_BOOT_ID" : "f2480929821e4562bbd4d6853c840d1a",
        "_SELINUX_CONTEXT" : "system_u:system_r:sssd_t:s0",
        "_GID" : "388",
        "MESSAGE" : "Exiting the SSSD. Could not restart critical service [test.openqa.fedoraproject.org].",
        "__MONOTONIC_TIMESTAMP" : "85790788",
        "_RUNTIME_SCOPE" : "system",
        "CODE_FILE" : "src/util/sss_log.c",
        "_HOSTNAME" : "ipa001.test.openqa.fedoraproject.org",
        "_SOURCE_REALTIME_TIMESTAMP" : "1718044459258886",
        "CODE_FUNC" : "sss_log_internal",
        "SSSD_PRG_NAME" : "sssd[sssd]",
        "__REALTIME_TIMESTAMP" : "1718044459259219",
        "_SYSTEMD_SLICE" : "system.slice",
        "_CAP_EFFECTIVE" : "0",
        "SYSLOG_IDENTIFIER" : "sssd",
        "_SYSTEMD_INVOCATION_ID" : "98aba6c15b4d45eaaade4f3d7b408417",
        "CODE_LINE" : "106",
        "SYSLOG_FACILITY" : "3",
        "SSSD_DOMAIN" : "",
        "_UID" : "388",
        "_SYSTEMD_CGROUP" : "/system.slice/sssd.service",
        "__SEQNUM_ID" : "6550fc39cd77421c8237913b111b15a0",
        "_PID" : "2661",
        "_COMM" : "sssd"
}

Comment 10 Alexey Tikhonov 2024-06-11 20:37:47 UTC
I was able to reproduce this in a local VM.

The reason is a wrong ownership 'ldap_child' binary (among others):
```
-rwxr-x---. 1 root root  53160 Jun  7 02:00 ldap_child
```
so 'sssd_be' running under 'sssd' user can't execute it.

It should be:
```
-rwxr-x---.  1 root sssd  52K Jun  5 20:00 ldap_child
```

In turn, the reason for this is that sssd user/group aren't created properly during package upgrade:
```
Jun 10 14:23:12 ipa001 dnf-3[716]:  Upgrading        : sssd-krb5-common-2.10.0~beta1-1.fc41.x86_64       431/2052
Jun 10 14:23:12 ipa001 dnf-3[716]: warning: group sssd does not exist - using root
Jun 10 14:23:12 ipa001 dnf-3[716]: warning: group sssd does not exist - using root
```

This means `%sysusers_create_compat %{SOURCE1}` doesn't work as expected during upgrade f39->f41 (but works f40->f41)
https://src.fedoraproject.org/rpms/sssd/blob/rawhide/f/sssd.spec#_1052

At this point I have no idea why.

Comment 11 Sumit Bose 2024-06-12 08:04:12 UTC
Hi,

was the `sssd-common` package which has:


    preinstall scriptlet (using /bin/sh):

    # generated from sssd.sysusers
    getent group 'sssd' >/dev/null || groupadd -r 'sssd' || :
    getent passwd 'sssd' >/dev/null || \
        useradd -r -g 'sssd' -d '/run/sssd/' -s '/sbin/nologin' -c 'User for sssd' 'sssd' || :

updated before `sssd-krb5-common`? 

If not, there might be an issue with ordering the dependencies during update?

If it was updated before, are there any errors in the logs?

bye,
Sumit

Comment 12 Alexey Tikhonov 2024-06-12 08:13:09 UTC
(In reply to Sumit Bose from comment #11)
> 
> If not, there might be an issue with ordering the dependencies during update?

```
%package krb5-common
...	
Requires: sssd-common = %{version}-%{release}
```

But actual order seems to be wrong?
```
$ grep Upgrading messages | grep sssd
Jun 10 14:22:49 ipa001 dnf-3[716]:  Upgrading        : sssd-winbind-idmap-2.10.0~beta1-1.fc41.x86_64     204/2052
Jun 10 14:23:12 ipa001 dnf-3[716]:  Upgrading        : sssd-common-pac-2.10.0~beta1-1.fc41.x86_64        430/2052
Jun 10 14:23:12 ipa001 dnf-3[716]:  Upgrading        : sssd-krb5-common-2.10.0~beta1-1.fc41.x86_64       431/2052
Jun 10 14:23:13 ipa001 dnf-3[716]:  Upgrading        : python3-sssdconfig-2.10.0~beta1-1.fc41.noarch     436/2052
Jun 10 14:23:13 ipa001 dnf-3[716]:  Upgrading        : sssd-ad-2.10.0~beta1-1.fc41.x86_64                437/2052
Jun 10 14:23:13 ipa001 dnf-3[716]:  Upgrading        : sssd-ipa-2.10.0~beta1-1.fc41.x86_64               438/2052
Jun 10 14:23:13 ipa001 dnf-3[716]:  Upgrading        : sssd-krb5-2.10.0~beta1-1.fc41.x86_64              439/2052
Jun 10 14:23:13 ipa001 dnf-3[716]:  Upgrading        : sssd-ldap-2.10.0~beta1-1.fc41.x86_64              440/2052
Jun 10 14:23:13 ipa001 dnf-3[716]:  Upgrading        : sssd-proxy-2.10.0~beta1-1.fc41.x86_64             441/2052
Jun 10 14:23:16 ipa001 dnf-3[716]:  Upgrading        : sssd-2.10.0~beta1-1.fc41.x86_64                   449/2052
Jun 10 14:23:17 ipa001 dnf-3[716]:  Upgrading        : sssd-nfs-idmap-2.10.0~beta1-1.fc41.x86_64         469/2052
Jun 10 14:23:29 ipa001 dnf-3[716]:  Upgrading        : sssd-client-2.10.0~beta1-1.fc41.x86_64            494/2052
Jun 10 14:23:32 ipa001 dnf-3[716]:  Upgrading        : sssd-common-2.10.0~beta1-1.fc41.x86_64            508/2052
Jun 10 14:23:34 ipa001 dnf-3[716]:  Upgrading        : sssd-dbus-2.10.0~beta1-1.fc41.x86_64              510/2052
Jun 10 14:24:16 ipa001 dnf-3[716]:  Upgrading        : sssd-tools-2.10.0~beta1-1.fc41.x86_64             724/2052
Jun 10 14:24:19 ipa001 dnf-3[716]:  Upgrading        : sssd-idp-2.10.0~beta1-1.fc41.x86_64               735/2052
Jun 10 14:24:42 ipa001 dnf-3[716]:  Upgrading        : sssd-passkey-2.10.0~beta1-1.fc41.x86_64           786/2052
Jun 10 14:27:09 ipa001 dnf-3[716]:  Upgrading        : sssd-kcm-2.10.0~beta1-1.fc41.x86_64              1066/2052
```


What puzzles me is that according to the report it works fine for f40->f41 upgrades (I didn't check).

Comment 13 Alexey Tikhonov 2024-06-12 08:21:07 UTC
Looks like (almost) all SSSD packages have 
```
Requires: sssd-common = %{version}-%{release}
```

Comment 14 Alexey Tikhonov 2024-06-12 08:29:53 UTC
There is also:
```
Jun 10 14:23:32 ipa001 dnf-3[716]:  Upgrading        : systemd-256~rc4-2.fc41.x86_64                     507/2052
Jun 10 14:23:32 ipa001 dnf-3[716]:  Running scriptlet: systemd-256~rc4-2.fc41.x86_64                     507/2052
Jun 10 14:23:32 ipa001 audit[1994]: ADD_GROUP pid=1994 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:groupadd_t:s0 msg='op=add-group id=388 exe="/usr/sbin/groupadd" hostname=? addr=? terminal=? res=success'
Jun 10 14:23:32 ipa001 kernel: kauditd_printk_skb: 52 callbacks suppressed
Jun 10 14:23:32 ipa001 kernel: audit: type=1116 audit(1718043812.172:126): pid=1994 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:groupadd_t:s0 msg='op=add-group id=388 exe="/usr/sbin/groupadd" hostname=? addr=? terminal=? res=success'
Jun 10 14:23:32 ipa001 audit[1994]: GRP_MGMT pid=1994 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:groupadd_t:s0 msg='op=add-shadow-group id=388 exe="/usr/sbin/groupadd" hostname=? addr=? terminal=? res=success'
Jun 10 14:23:32 ipa001 kernel: audit: type=1132 audit(1718043812.175:127): pid=1994 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:groupadd_t:s0 msg='op=add-shadow-group id=388 exe="/usr/sbin/groupadd" hostname=? addr=? terminal=? res=success'
Jun 10 14:23:32 ipa001 audit[1999]: ADD_USER pid=1999 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:useradd_t:s0 msg='op=add-user acct="sssd" exe="/usr/sbin/useradd" hostname=? addr=? terminal=? res=success'
Jun 10 14:23:32 ipa001 kernel: audit: type=1114 audit(1718043812.236:128): pid=1999 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:useradd_t:s0 msg='op=add-user acct="sssd" exe="/usr/sbin/useradd" hostname=? addr=? terminal=? res=success'
Jun 10 14:23:32 ipa001 dnf-3[716]:  Running scriptlet: sssd-common-2.10.0~beta1-1.fc41.x86_64            508/2052
Jun 10 14:23:32 ipa001 dnf-3[716]:  Upgrading        : sssd-common-2.10.0~beta1-1.fc41.x86_64            508/2052
```

Comment 15 Alexey Tikhonov 2024-06-12 08:49:17 UTC
Hi dnf maintainers

Looks like
```
%package krb5-common
...	
Requires: sssd-common = %{version}-%{release}
```
isn't enough to make sure that 'sssd-common' is upgraded and its pre-script is execute *before* 'sssd-krb5-common' is upgraded.

Could you please suggest how to express this dependency in the spec-file?

Comment 16 Panu Matilainen 2024-06-13 05:42:46 UTC
If a plain 'Requires' doesn't get the dependency installed first, it's because there's a dependency loop that will need to be dealt with.

Comment 17 Alexey Tikhonov 2024-06-13 08:48:57 UTC
(In reply to Panu Matilainen from comment #16)
> If a plain 'Requires' doesn't get the dependency installed first, it's
> because there's a dependency loop that will need to be dealt with.


If I read this correctly then "Requires:" is expected to do the work.

There is no evident loop in sssd spec-file.

Besides, it looks there is a similar issue with 'passim' in the attached log:
```
Jun 10 14:26:19 ipa001 dnf-3[716]:  Installing       : passim-0.1.8-1.fc41.x86_64                        987/2052
Jun 10 14:26:19 ipa001 dbus-broker-launch[715]: Noticed file-system modification, trigger reload.
Jun 10 14:26:19 ipa001 dbus-broker-launch[715]: Looking up NSS user entry for 'passim'...
Jun 10 14:26:19 ipa001 dbus-broker-launch[715]: NSS returned no entry for 'passim'
Jun 10 14:26:19 ipa001 dbus-broker-launch[715]: Invalid user-name in /usr/share/dbus-1/system.d/org.freedesktop.Passim.conf +12: user="passim"
Jun 10 14:26:19 ipa001 dbus-broker-launch[715]: Invalid user name 'passim' in service file '/usr/share/dbus-1/system-services/org.freedesktop.Passim.service'
Jun 10 14:26:19 ipa001 dnf-3[716]:  Running scriptlet: passim-0.1.8-1.fc41.x86_64                        987/2052
```

So this looks like a bug in rpm/dnf.

Shall I change component to 'rpm' or is 'dnf' ok?

Comment 18 Panu Matilainen 2024-06-13 09:24:26 UTC
There's a dependency loop somewhere in the package set, and those are often not immediately obvious.
It's neither an rpm or dnf bug.

Comment 19 Alexey Tikhonov 2024-06-13 09:29:30 UTC
Could you please help / suggest how to debug it?

Comment 20 Sumit Bose 2024-06-13 10:48:54 UTC
Hi,

can `Conflicts:` cause a loop?

We have e.g.:


%package common
Summary: Common files for the SSSD
License: GPL-3.0-or-later
...
Requires: libsss_certmap = %{version}-%{release}


%package -n libsss_certmap
Summary: SSSD Certificate Mapping Library
License: LGPL-3.0-or-later
Conflicts: sssd-common < %{version}-%{release}


Thanks for your help.

bye,
Sumit

Comment 21 Panu Matilainen 2024-06-14 11:22:40 UTC
Conflicts don't take part in ordering, so that's not an issue.

Basically one needs the --deploops output from the transaction where loops occur to analyze them (the other option, aka desperation, is manually chasing them)

With rpm itself, it's just --deploops on the command line. dnf4 in f39 supports --setopts=tsflags=deploops, dnf5 upstream does too but not the version in f39. So it depends on how exactly the test is performed. I have no idea what test we're talking about here exactly (much less where to find it)

Comment 22 Alexey Tikhonov 2024-06-14 11:47:49 UTC
(In reply to Panu Matilainen from comment #21)
> what test we're talking about here exactly

Test isn't relevant.

The issue happens during plain upgrade from F39 to Rawhide.

Will it work to provide "--setopts=tsflags=deploops" to "dnf system-upgrade reboot"?

Comment 23 Panu Matilainen 2024-06-14 12:10:46 UTC
The test is all that counts, because we need to know the exact package set involved. There's no such thing as "plain upgrade from f39 to rawhide", the possible combinations of packages in Fedora are in the millions or more and more often than not only a specific subset will trip up stuff like this.

I have no idea if setopts is passed through system-upgrade. Somebody from the dnf team can maybe answer that.

Comment 24 Adam Williamson 2024-06-14 19:37:14 UTC
The test starts from a stock Fedora 39 Server install, deploys it as a FreeIPA server, then upgrades it. The full set of packages installed should be apparent from the logs in the log tarball.

Comment 25 Alexey Tikhonov 2024-06-16 11:38:19 UTC
(In reply to Adam Williamson from comment #24)
> The test starts from a stock Fedora 39 Server install, deploys it as a
> FreeIPA server, then upgrades it.

Another reproducer: stock Fedora 39 Workstation, install "sssd-*", upgrade to Rawhide
(i.e. no need to deploy FreeIPA server)

Comment 26 Panu Matilainen 2024-06-17 06:39:09 UTC
# dnf --releasever=39 --installroot=/srv/test install "sssd-*"
...
# dnf --releasever=41 --installroot=/srv/test --setopt=tsflags=deploops,test update

...gives me the following:

Running transaction test
warning: 8 Strongly Connected Components
warning: SCC #1: 2 members (18 external dependencies)
warning: 	python3-libs-3.13.0~b2-3.fc41.x86_64
warning: 		-> python3-3.13.0~b2-3.fc41.x86_64
warning: 	python3-3.13.0~b2-3.fc41.x86_64
warning: 		-> python3-libs-3.13.0~b2-3.fc41.x86_64
warning: SCC #2: 3 members (31 external dependencies)
warning: 	samba-common-libs-2:4.20.1-4.fc41.x86_64
warning: 		-> libwbclient-2:4.20.1-4.fc41.x86_64
warning: 		-> samba-client-libs-2:4.20.1-4.fc41.x86_64
warning: 	samba-client-libs-2:4.20.1-4.fc41.x86_64
warning: 		-> samba-common-libs-2:4.20.1-4.fc41.x86_64
warning: 		-> libwbclient-2:4.20.1-4.fc41.x86_64
warning: 	libwbclient-2:4.20.1-4.fc41.x86_64
warning: 		-> samba-client-libs-2:4.20.1-4.fc41.x86_64
warning: SCC #3: 2 members (29 external dependencies)
warning: 	systemd-256-1.fc41.x86_64
warning: 		-> systemd-pam-256-1.fc41.x86_64
warning: 	systemd-pam-256-1.fc41.x86_64
warning: 		-> systemd-256-1.fc41.x86_64
warning: SCC #4: 3 members (15 external dependencies)
warning: 	ca-certificates-2023.2.62_v7.0.401-6.fc40.noarch
warning: 		=> coreutils-9.5-2.fc41.x86_64
warning: 	openssl-libs-1:3.2.2-1.fc41.x86_64
warning: 		-> ca-certificates-2023.2.62_v7.0.401-6.fc40.noarch
warning: 	coreutils-9.5-2.fc41.x86_64
warning: 		-> openssl-libs-1:3.2.2-1.fc41.x86_64
warning: SCC #5: 6 members (5 external dependencies)
warning: 	bash-5.2.26-3.fc40.x86_64
warning: 		-> ncurses-libs-6.4-12.20240127.fc40.x86_64
warning: 		-> glibc-2.39.9000-26.fc41.x86_64
warning: 	glibc-common-2.39.9000-26.fc41.x86_64
warning: 		-> glibc-2.39.9000-26.fc41.x86_64
warning: 		-> bash-5.2.26-3.fc40.x86_64
warning: 	glibc-minimal-langpack-2.39.9000-26.fc41.x86_64
warning: 		-> glibc-common-2.39.9000-26.fc41.x86_64
warning: 		-> glibc-2.39.9000-26.fc41.x86_64
warning: 	glibc-2.39.9000-26.fc41.x86_64
warning: 		-> glibc-minimal-langpack-2.39.9000-26.fc41.x86_64
warning: 		-> glibc-common-2.39.9000-26.fc41.x86_64
warning: 		-> glibc-gconv-extra-2.39.9000-26.fc41.x86_64
warning: 	ncurses-libs-6.4-12.20240127.fc40.x86_64
warning: 		-> glibc-2.39.9000-26.fc41.x86_64
warning: 	glibc-gconv-extra-2.39.9000-26.fc41.x86_64
warning: 		-> glibc-common-2.39.9000-26.fc41.x86_64
warning: 		-> glibc-2.39.9000-26.fc41.x86_64
warning: SCC #6: 7 members (2 external dependencies)
warning: 	fedora-repos-rawhide-41-0.2.noarch
warning: 		-> fedora-repos-41-0.2.noarch
warning: 	fedora-repos-41-0.2.noarch
warning: 		-> fedora-release-41-0.13.noarch
warning: 		-> fedora-repos-rawhide-41-0.2.noarch
warning: 		-> fedora-gpg-keys-41-0.2.noarch
warning: 	fedora-release-common-41-0.13.noarch
warning: 		-> fedora-repos-41-0.2.noarch
warning: 		-> fedora-release-41-0.13.noarch
warning: 	fedora-release-41-0.13.noarch
warning: 		-> fedora-release-common-41-0.13.noarch
warning: 	setup-2.15.0-4.fc41.noarch
warning: 		-> fedora-release-41-0.13.noarch
warning: 	filesystem-3.18-9.fc41.x86_64
warning: 		=> setup-2.15.0-4.fc41.noarch
warning: 	fedora-gpg-keys-41-0.2.noarch
warning: 		-> filesystem-3.18-9.fc41.x86_64
warning: SCC #7: 6 members (5 external dependencies)
warning: 	filesystem-3.18-6.fc39.x86_64
warning: 		-> fedora-gpg-keys-39-2.noarch
warning: 	setup-2.14.4-1.fc39.noarch
warning: 		-> filesystem-3.18-6.fc39.x86_64
warning: 		-> filesystem-3.18-6.fc39.x86_64
warning: 	fedora-release-39-36.noarch
warning: 		-> setup-2.14.4-1.fc39.noarch
warning: 		-> fedora-repos-39-2.noarch
warning: 	fedora-release-common-39-36.noarch
warning: 		-> fedora-release-39-36.noarch
warning: 	fedora-repos-39-2.noarch
warning: 		-> fedora-release-common-39-36.noarch
warning: 	fedora-gpg-keys-39-2.noarch
warning: 		-> fedora-repos-39-2.noarch
warning: SCC #8: 6 members (1362 external dependencies)
warning: 	glibc-2.38-18.fc39.x86_64
warning: 		-> ncurses-libs-6.4-7.20230520.fc39.1.x86_64
warning: 		-> ncurses-libs-6.4-7.20230520.fc39.1.x86_64
warning: 		-> ncurses-libs-6.4-7.20230520.fc39.1.x86_64
warning: 		-> ncurses-libs-6.4-7.20230520.fc39.1.x86_64
warning: 		-> ncurses-libs-6.4-7.20230520.fc39.1.x86_64
warning: 		-> ncurses-libs-6.4-7.20230520.fc39.1.x86_64
warning: 		-> ncurses-libs-6.4-7.20230520.fc39.1.x86_64
warning: 		-> ncurses-libs-6.4-7.20230520.fc39.1.x86_64
warning: 		-> ncurses-libs-6.4-7.20230520.fc39.1.x86_64
warning: 		-> ncurses-libs-6.4-7.20230520.fc39.1.x86_64
warning: 		-> ncurses-libs-6.4-7.20230520.fc39.1.x86_64
warning: 		-> ncurses-libs-6.4-7.20230520.fc39.1.x86_64
warning: 		-> bash-5.2.26-1.fc39.x86_64
warning: 		-> bash-5.2.26-1.fc39.x86_64
warning: 		-> bash-5.2.26-1.fc39.x86_64
warning: 		-> bash-5.2.26-1.fc39.x86_64
warning: 		-> bash-5.2.26-1.fc39.x86_64
warning: 		-> bash-5.2.26-1.fc39.x86_64
warning: 		-> bash-5.2.26-1.fc39.x86_64
warning: 		-> bash-5.2.26-1.fc39.x86_64
warning: 		-> bash-5.2.26-1.fc39.x86_64
warning: 		-> bash-5.2.26-1.fc39.x86_64
warning: 		-> bash-5.2.26-1.fc39.x86_64
warning: 		-> bash-5.2.26-1.fc39.x86_64
warning: 		-> bash-5.2.26-1.fc39.x86_64
warning: 		-> bash-5.2.26-1.fc39.x86_64
warning: 		-> bash-5.2.26-1.fc39.x86_64
warning: 	glibc-minimal-langpack-2.38-18.fc39.x86_64
warning: 		-> glibc-2.38-18.fc39.x86_64
warning: 		-> glibc-2.38-18.fc39.x86_64
warning: 	glibc-common-2.38-18.fc39.x86_64
warning: 		-> glibc-minimal-langpack-2.38-18.fc39.x86_64
warning: 		-> glibc-gconv-extra-2.38-18.fc39.x86_64
warning: 		-> glibc-2.38-18.fc39.x86_64
warning: 	bash-5.2.26-1.fc39.x86_64
warning: 		-> glibc-common-2.38-18.fc39.x86_64
warning: 	ncurses-libs-6.4-7.20230520.fc39.1.x86_64
warning: 		-> bash-5.2.26-1.fc39.x86_64
warning: 	glibc-gconv-extra-2.38-18.fc39.x86_64
warning: 		-> glibc-2.38-18.fc39.x86_64
warning: 		-> glibc-2.38-18.fc39.x86_64
Transaction test succeeded.

No sssd-related dependency loops there, and if I let that complete the order appears correct:

  Upgrading        : sssd-krb5-common-2.10.0~beta1-2.fc41.x86_64        129/296 
  Running scriptlet: samba-common-2:4.20.1-4.fc41.noarch                130/296 
  [...]
  Upgrading        : samba-common-libs-2:4.20.1-4.fc41.x86_64           133/296 
  Upgrading        : sssd-common-pac-2.10.0~beta1-2.fc41.x86_64         134/296 

But okay looking closer the recipe was 'stock Fedora 39 Workstation, install "sssd-*"' so I was missing the "stock workstation" part there. Anyway the above shows how to get the deploops out of dnf, without actually performing the upgrade. I'm on a mobile broadband here so I'll leave heaving all those gigabytes around to somebody else.

Comment 27 Alexey Tikhonov 2024-06-17 12:27:19 UTC
(In reply to Panu Matilainen from comment #26)
> 
>   Upgrading        : sssd-krb5-common-2.10.0~beta1-2.fc41.x86_64       129/296 
>   Running scriptlet: samba-common-2:4.20.1-4.fc41.noarch               130/296 
>   [...]
>   Upgrading        : samba-common-libs-2:4.20.1-4.fc41.x86_64          133/296 
>   Upgrading        : sssd-common-pac-2.10.0~beta1-2.fc41.x86_64        134/296 

Reported issue is about "sssd-common" being installed not first (after "sssd-krb5-common", etc), not about "samba-common".

> But okay looking closer the recipe was 'stock Fedora 39 Workstation, install
> "sssd-*"' so I was missing the "stock workstation" part there. Anyway the
> above shows how to get the deploops out of dnf, without actually performing
> the upgrade.

Ok, thanks, I'll try this.

Comment 28 Panu Matilainen 2024-06-17 12:37:12 UTC
> Reported issue is about "sssd-common" being installed not first (after "sssd-krb5-common", etc), not about "samba-common".

I know, samba-common* was there for context but I mixed up sssd-common-pac for sssd-common. sssd-common was in fact installed just before this, so wasn't the most useful copy-paste ever, sorry about that :D

  Upgrading        : systemd-pam-256-1.fc41.x86_64                      126/296 
  Upgrading        : systemd-256-1.fc41.x86_64                          127/296 
  Running scriptlet: systemd-256-1.fc41.x86_64                          127/296 
  Running scriptlet: sssd-common-2.10.0~beta1-2.fc41.x86_64             128/296 
  Upgrading        : sssd-common-2.10.0~beta1-2.fc41.x86_64             128/296 
  Running scriptlet: sssd-common-2.10.0~beta1-2.fc41.x86_64             128/296 
  Upgrading        : sssd-krb5-common-2.10.0~beta1-2.fc41.x86_64        129/296

Comment 29 Alexey Tikhonov 2024-06-17 16:13:43 UTC
When I run above on a Workstation it's the same as initial issue:
```
...
  Upgrading        : python-pip-wheel-24.0-5.fc41.noarch                                                                                             830/3974 
  Upgrading        : bind-utils-32:9.18.26-1.fc41.x86_64                                                                                             831/3974 
  Upgrading        : adcli-0.9.2-6.fc40.x86_64                                                                                                       832/3974 
  Upgrading        : sssd-common-pac-2.10.0~beta1-2.fc41.x86_64                                                                                      833/3974 
  Upgrading        : sssd-krb5-common-2.10.0~beta1-2.fc41.x86_64                                                                                     834/3974 
warning: group sssd does not exist - using root
warning: group sssd does not exist - using root
warning: user sssd does not exist - using root
```

`--setopt=tsflags=deploops,test` reports "81 Strongly Connected Component"
There is no word "loop" in the log, so I guess I need to review the log "manually"? (it is 2880 lines)

Comment 30 Adam Williamson 2024-06-17 16:28:31 UTC
"Strongly connected components" create loops. Can you just attach the log?

Comment 31 Alexey Tikhonov 2024-06-17 17:49:19 UTC
How do I read "A -> B"  --  is it
(1) "A is a req for B"
or
(2) "B is a req for A"
?

Or does it depend on set (old/new)?

I have a really hard time understanding this log, it looks like there are tons of loops like:
```
warning:        authselect-libs-1.4.3-1.fc39.x86_64
warning:                -> util-linux-2.39.4-1.fc39.x86_64

warning:        util-linux-2.40.1-2.fc41.x86_64
warning:                -> authselect-libs-1.5.0-5.fc41.x86_64
```
but it's (always?) one way dependency for "old" set and other way dependency for "new" set.

Attaching full log.

Comment 33 Alexey Tikhonov 2024-06-17 17:54:33 UTC
```
warning:        openssl-libs-1:3.1.1-4.fc39.x86_64
warning:                -> sssd-proxy-2.9.5-1.fc39.x86_64
warning:                -> libsss_certmap-2.9.5-1.fc39.x86_64

warning:        libsss_certmap-2.9.5-1.fc39.x86_64
warning:                -> sssd-proxy-2.9.5-1.fc39.x86_64

warning:        libsss_certmap-2.10.0~beta1-2.fc41.x86_64
warning:                -> openssl-libs-1:3.2.2-1.fc41.x86_64

warning:        sssd-proxy-2.10.0~beta1-2.fc41.x86_64
warning:                -> libsss_certmap-2.10.0~beta1-2.fc41.x86_64

warning:        sssd-proxy-2.10.0~beta1-2.fc41.x86_64
warning:                -> libsss_certmap-2.10.0~beta1-2.fc41.x86_64
warning:                -> openssl-libs-1:3.2.2-1.fc41.x86_64
```

Comment 34 Panu Matilainen 2024-06-18 07:02:07 UTC
I totally agree the output could be (quite a bit) more reader friendly.

A -> B means A depends on B, and it has to be both ways for it to be a loop. The older packages are being uinstalled and for that the order is (roughly) reversed which is why they get logged the other way around, but you can ignore those because this is about install, not erase order.

> SCC #51: 88 members (492 external dependencies)

This is the kind of loop-of-doom that has close to zero chance of being cut up without "side-effects", and that's where sssd-common and friends are too. The output could be more helpful pointing out the dependencies that actually loop, it's far from obvious in exactly the kind of large loops that cause the actual problems.

A common source of unnecessary loops is "Requires: xxx = %{version}-%{release}" between sub-packages when the only reason is to keep the version-release locked to the same set. I'm not saying that's the problem here, but eliminating unnecessary ordered dependencies is always a good thing: dependencies whose only purpose is to lock sub-packages to the same version-release should be marked Requires(meta) to avoid ordering based on them.

Comment 35 Panu Matilainen 2024-06-18 07:09:34 UTC
(except that argh, I keep forgetting Requires(meta) only works in >= 4.20 just now, whereas it should've been supported for years)

Comment 36 Panu Matilainen 2024-06-18 07:23:08 UTC
...except that argh, scratch the above, I keep mixing up the issue. Requires(meta) does work for several releases now, it's (meta) on weak dependencies that was broken for a long time. 

/me goes to fetch more coffee, sorry about the noise

Comment 37 Alexey Tikhonov 2024-06-18 12:29:59 UTC
> A common source of unnecessary loops is "Requires: xxx = %{version}-%{release}" between sub-packages when the only reason is to keep the version-release locked to the same set. I'm not saying that's the problem here, but eliminating unnecessary ordered dependencies is always a good thing

But in this case 'sssd-common' creates 'sssd' service user that is used during installation of other sub-packages (to chown some files), that's why it has to be installed (upgraded) first...

Comment 38 Panu Matilainen 2024-06-18 12:46:33 UTC
Sure. The point was simply that there could be others that are there only for the version sync, and relaxing the ordering constraints where not actually relevant can help resolve loops elsewhere.

Comment 39 Alexey Tikhonov 2024-06-21 18:24:05 UTC
```
warning:                -> openssl-libs-1:3.2.2-1.fc41.x86_64
    warning:                -> pkcs11-provider-0.5-2.fc41.x86_64
        warning:                -> openssl-libs-1:3.2.2-1.fc41.x86_64
```

https://src.fedoraproject.org/rpms/openssl/c/84795a92474d367f6ae605f294a943094e87a860 :
```
+            Recommends: pkcs11-provider%{?_isa}
```

https://src.fedoraproject.org/rpms/pkcs11-provider/blob/rawhide/f/pkcs11-provider.spec#_16 :
```
BuildRequires: openssl-devel >= 3.0.7
```

Panu, can this be a reason?

And what is the difference between "->" and "=>"?

Comment 40 Alexey Tikhonov 2024-06-21 18:27:50 UTC
Or maybe:
```
warning:                -> openssl-libs-1:3.2.2-1.fc41.x86_64
    warning:                -> ca-certificates-2023.2.62_v7.0.401-6.fc40.noarch
        warning:                => coreutils-9.5-2.fc41.x86_64
            warning:                -> openssl-libs-1:3.2.2-1.fc41.x86_64
```

Comment 41 Alexey Tikhonov 2024-06-21 18:41:09 UTC
Other candidates:

```
warning:        fprintd-pam-1.94.2-11.fc40.x86_64
warning:                -> authselect-1.5.0-5.fc41.x86_64
warning:        authselect-1.5.0-5.fc41.x86_64
warning:                -> fprintd-pam-1.94.2-11.fc40.x86_64
```

And probably most "promising":
```
warning:        sssd-common-2.10.0~beta1-2.fc41.x86_64
warning:                => systemd-256-1.fc41.x86_64
    warning:                -> util-linux-2.40.1-2.fc41.x86_64
        warning:                -> pam-1.6.1-3.fc41.x86_64
            warning:                -> authselect-1.5.0-5.fc41.x86_64
                warning:                -> sssd-2.10.0~beta1-2.fc41.x86_64
                    warning:                -> sssd-common-2.10.0~beta1-2.fc41.x86_64
```

Comment 42 Panu Matilainen 2024-06-24 05:48:23 UTC
BuildRequires are only relevant for building packages, as per the name. Recommends and other weak dependencies may be involved in ordering depending on rpm version and various details. In rpm 4.19 they certainly are. And yeah the loop around sssd-common and systemd indeed looks like a good candidate. It's pretty common for systemd to be involved (which is not to say "guilty of" - it's more complicated than that) in these gigantic loops.

The difference between -> and => is that the former is a "plain" Requires and the latter is a "scriptlet" dependency such as Requires(post). The latter is favored when cutting loops and can help in specific situations but shouldn't be used lightly for that purpose as it quickly stops working at all if everybody just claims their dependency is more important :D

I see two dubious dependencies in that "promising" set:
1) pam -> authselect: surely pam doesn't require authselect to function, only for configuring? In that case it should be marked Requires(meta) in pam.spec and poof the loop is gone.
2) authselect -> sssd: this is only a Suggests and having it involved in ordering is probably not what the packager intended and thus unfortunate. Unfortunately in rpm 4.19 there's no way to flag weak dependencies as "meta" (a silly bug that needs fixing in rpm), and while 4.20 ignores such plain weak dependencies during ordering for reasons just like this, 4.19 does take them into account.

Due to the circumstances, 1) is probably the easier/faster one to address. 2) would not be an issue if the upgrade was done with rpm from F41, but I guess "system-upgrade" still runs with the version from the current release. We are considering backporting some related fixes to a new 4.19 release but that'll take some time.

Comment 43 Alexey Tikhonov 2024-06-24 11:30:43 UTC
Thank you, Panu.

(In reply to Panu Matilainen from comment #42)
> 
> I see two dubious dependencies in that "promising" set:
> 1) pam -> authselect: surely pam doesn't require authselect to function,
> only for configuring? In that case it should be marked Requires(meta) in
> pam.spec and poof the loop is gone.

Iker, is it possible?

Comment 44 Alexey Tikhonov 2024-06-24 11:37:42 UTC
> BuildRequires are only relevant for building packages

I guess in this case it also links against openssl libs, so as a result `pkcs11-provider-0.5-2.fc41` depends on `openssl-libs-1:3.2.2-1.fc41` implicitly.

Comment 45 Panu Matilainen 2024-06-24 12:48:15 UTC
Oh sure, BuildRequires may indirectly introduce additional runtime dependencies that will affect ordering, such as in case of libraries linked to.

Comment 47 Sumit Bose 2024-06-24 14:49:02 UTC
(In reply to Alexey Tikhonov from comment #41)
> Other candidates:
> 
> ```
> warning:        fprintd-pam-1.94.2-11.fc40.x86_64
> warning:                -> authselect-1.5.0-5.fc41.x86_64
> warning:        authselect-1.5.0-5.fc41.x86_64
> warning:                -> fprintd-pam-1.94.2-11.fc40.x86_64
> ```
> 
> And probably most "promising":
> ```
> warning:        sssd-common-2.10.0~beta1-2.fc41.x86_64
> warning:                => systemd-256-1.fc41.x86_64
>     warning:                -> util-linux-2.40.1-2.fc41.x86_64
>         warning:                -> pam-1.6.1-3.fc41.x86_64
>             warning:                -> authselect-1.5.0-5.fc41.x86_64
>                 warning:                -> sssd-2.10.0~beta1-2.fc41.x86_64

Hi,

why does authselect depend on the `sssd` meta package which will pull in everything and not only on `sssd-common`? Didn't we try to remove dependencies on the `sssd` meta package or was this only for RHEL?

Since the meta package will pull in everything it might be somewhat expected that the dependency resolver is lost and just start with some of the dependencies. If the authselect dependency would be only `sssd-common` then there is still a loop, but just with sssd-common and no other SSSD components at this point.

However, since it looks like any random component might introduce such loops I wonder what would be the alternatives to make this more reliable. One way might be using `%pretrans` another to add the `%sysusers_create_compat %{SOURCE1}` %pre macro or similar to all packages which need the SSSD user for installation. Both have the pro and cons and maybe there are others but from the given two I think I would prefer the latter.

bye,
Sumit


>                     warning:                ->
> sssd-common-2.10.0~beta1-2.fc41.x86_64
> ```

Comment 48 Adam Williamson 2024-06-24 15:42:21 UTC
pam dep on authselect: added in https://src.fedoraproject.org/rpms/pam/c/ff21ecd1
authselect dep on sssd (suggests): added in https://src.fedoraproject.org/rpms/authselect/c/4c5cb1a9aef1bb71c390f2f26a8fddbfbbffe8d4 , without explanation. That was quite early in the package's history.

Comment 49 Panu Matilainen 2024-06-25 05:29:30 UTC
> One way might be using `%pretrans` another to add the `%sysusers_create_compat %{SOURCE1}` %pre macro or similar to all packages which need the SSSD user for installation.

It's not possible to create users (or much anything else for that matter) from %pretrans because it runs in a complete void during initial install. You can't win this with dirty tricks, so please don't try. The best thing you can do is reduce ordering constraints in your package set: have a hard look at all your Requires and any that are there only to ensure version lock to subcomponents, turn to Requires(meta). If none are, then none are. But it's worth a look.

> pam dep on authselect: added in https://src.fedoraproject.org/rpms/pam/c/ff21ecd1

Thanks Adam for digging that up. That does look like a potential meta dependency to me: install scriptlets do not need pam services, only the running system does. So the order does not matter at all. AFAICS.

> authselect dep on sssd (suggests): added in https://src.fedoraproject.org/rpms/authselect/c/4c5cb1a9aef1bb71c390f2f26a8fddbfbbffe8d4 , without explanation. That was quite early in the package's history.

Yup, it has the air of "having this around seems like useful" dependency, never intended to affect ordering or anything like that. Although, by 2018 it would have already done so.

Comment 50 Pavel Březina 2024-06-25 09:07:38 UTC
(In reply to Adam Williamson from comment #48)
> pam dep on authselect: added in
> https://src.fedoraproject.org/rpms/pam/c/ff21ecd1
> authselect dep on sssd (suggests): added in
> https://src.fedoraproject.org/rpms/authselect/c/
> 4c5cb1a9aef1bb71c390f2f26a8fddbfbbffe8d4 , without explanation. That was
> quite early in the package's history.

I'm pretty sure there was a bugzilla requesting to add that, but it is ancient history now. I think it can be safely omitted now, since sssd is not involved in the default installation at all now.

What does Requires(meta) do?

I will follow you recommendations in authselect.

Comment 51 Panu Matilainen 2024-06-25 09:49:19 UTC
> What does Requires(meta) do?

https://rpm-software-management.github.io/rpm/manual/spec.html#dependencies

For practical purposes, it's just like Requires except that meta-dependencies do not participate in install/erase ordering at all. It's raison d'être is to help cutting unnecessary ordering loops without sacrificing the actual dependency, for the cases where applicable.

Comment 52 Pavel Březina 2024-06-25 11:36:47 UTC
And what would be the recommendation? Drop all suggests in authselect, use requires(meta) in PAM or perhaps even both? Or is there Suggests(meta)?

Comment 53 Panu Matilainen 2024-06-25 12:09:39 UTC
Please see my earlier comments, c#42 in particular. I don't know these package sets anywhere near enough to be able to give you more precise recommendations than that.

Comment 54 Adam Williamson 2024-06-25 15:24:26 UTC
Based on c#42 I feel like the safest thing to try here would be to turn the pam dep on authselect into Requires(meta) . Dropping the suggests: sssd from authselect seems like a slightly bigger change, and also time-limited since rpm 4.20 will ignore it for ordering. so I'd suggest trying Requires(meta) in pam first, and we can see if that's enough to resolve the issue...

Comment 55 Adam Williamson 2024-06-25 15:24:51 UTC
Panu, would we need to change it on both ends (in F39 and Rawhide) or only on the Rawhide end?

Comment 56 Panu Matilainen 2024-06-26 05:22:59 UTC
Since it only happens on upgrade to rawhide, it should be enough to only update the rawhide package(s).

And, that made me realize you can actually change the "Suggests: sssd" in authselect into "Suggests(meta): authselect": 4.18-19 will honor an existing meta flag, they just cannot build packages with it due to a silly oversight/bug. The suggests seems like the safer change, but ideally we'd do both because this is a fairly tight bunch and every removed unnecessary constraint helps.

Comment 57 Panu Matilainen 2024-06-26 05:25:13 UTC
> change the "Suggests: sssd" in authselect into "Suggests(meta): authselect"

Argh, where's the edit button when you need it. That's of course a thinko and should be "Suggests(meta): sssd", *in* authselect.

Comment 59 Pavel Březina 2024-06-28 12:49:28 UTC
(In reply to Pavel Březina from comment #58)
> https://src.fedoraproject.org/rpms/authselect/pull-request/24

Scratch build in pull requests worked but I am not able to fedpkg build locally.

```
error: line 60: Unknown tag: Suggests(meta): sssd
error: query of specfile /home/pbrezina/workspace/packages/fedora/authselect/authselect.spec failed, can't parse

Could not get n-v-r-e from /home/pbrezina/workspace/packages/fedora/authselect/authselect.spec
Note: You can skip NVR construction & NVR check with --skip-nvr-check. See help for more info.
Could not execute build: Cannot continue without properly constructed NVR
```

Can someone do it for me?

Comment 60 Adam Williamson 2024-06-28 15:05:16 UTC
That's the thing Panu was talking about with older versions of RPM not understanding that tag.

If the scratch build in the pull request worked, why do you want to do a local build? What do you want exactly, a build for a specific arch or release? I can do it, just not sure what it is you're looking for.

Comment 61 Adam Williamson 2024-06-29 06:43:39 UTC
+3 in https://pagure.io/fedora-qa/blocker-review/issue/1608 , marking accepted.

Comment 62 Pavel Březina 2024-07-01 11:07:15 UTC
(In reply to Adam Williamson from comment #60)
> That's the thing Panu was talking about with older versions of RPM not
> understanding that tag.
> 
> If the scratch build in the pull request worked, why do you want to do a
> local build? What do you want exactly, a build for a specific arch or
> release? I can do it, just not sure what it is you're looking for.

I don't want to do a local build, I want to do koji build. But can't, since `fedpkg build` fails on my machine due to old rpm.

Comment 63 Iker Pedrosa 2024-07-02 09:33:35 UTC
This should be fixed by the recent update in PAM. If this is not the case, feel free to reopen the ticket.

Comment 64 Pavel Březina 2024-07-02 09:55:08 UTC
(In reply to Pavel Březina from comment #62)
> (In reply to Adam Williamson from comment #60)
> > That's the thing Panu was talking about with older versions of RPM not
> > understanding that tag.
> > 
> > If the scratch build in the pull request worked, why do you want to do a
> > local build? What do you want exactly, a build for a specific arch or
> > release? I can do it, just not sure what it is you're looking for.
> 
> I don't want to do a local build, I want to do koji build. But can't, since
> `fedpkg build` fails on my machine due to old rpm.

Nevermind, I used toolbox with rawhide and was able to build it. https://koji.fedoraproject.org/koji/taskinfo?taskID=119893267

Comment 65 Fedora Update System 2024-07-02 10:05:05 UTC
FEDORA-2024-b81c4f2a9a (authselect-1.5.0-6.fc41) has been submitted as an update to Fedora 41.
https://bodhi.fedoraproject.org/updates/FEDORA-2024-b81c4f2a9a

Comment 66 Fedora Update System 2024-07-02 12:02:27 UTC
FEDORA-2024-b81c4f2a9a (authselect-1.5.0-6.fc41) has been pushed to the Fedora 41 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 67 Adam Williamson 2024-07-02 15:45:45 UTC
The test failed again in today's Rawhide, but it looks like it didn't have the new pam or authselect builds yet. We'll see how tomorrow's compose goes.


Note You need to log in before you can comment on or make changes to this bug.