Fedora Account System
Red Hat Associate
Red Hat Customer
In openQA testing, FreeIPA server deployment periodically fails with an error: 2026-03-15T09:26:37Z DEBUG The ipa-client-install command failed, exception: ACIError: Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty) There doesn't seem to be any rhyme or reason as to when this happens, it just...happens sometimes. Most of the time the test is fine. The error occurs during the ipa-client-install part of the process. Here's a longer log extract: 2026-03-15T09:26:37Z DEBUG Starting external process 2026-03-15T09:26:37Z DEBUG args=['/usr/bin/certutil', '-d', 'sql:/tmp/tmp89o9isx9', '-A', '-n', 'CA certificate 1', '-t', 'C,,', '-a', '-f', '/tmp/tmp89o9isx9/pwdfile.tx t'] 2026-03-15T09:26:37Z DEBUG Process finished, return code=0 2026-03-15T09:26:37Z DEBUG stdout= 2026-03-15T09:26:37Z DEBUG stderr= 2026-03-15T09:26:37Z DEBUG failed to find session_cookie in persistent storage for principal 'host/ipa001.test.openqa.fedoraproject.org.FEDORAPROJECT.ORG' 2026-03-15T09:26:37Z DEBUG trying https://ipa001.test.openqa.fedoraproject.org/ipa/json 2026-03-15T09:26:37Z DEBUG Created connection context.rpcclient_281473231564352 2026-03-15T09:26:37Z DEBUG [try 1]: Forwarding 'schema' to json server 'https://ipa001.test.openqa.fedoraproject.org/ipa/json' 2026-03-15T09:26:37Z DEBUG New HTTP connection (ipa001.test.openqa.fedoraproject.org) 2026-03-15T09:26:37Z DEBUG received Set-Cookie (<class 'list'>)'['ipa_session=MagBearerToken=tOQOzYTYm3YG82Nz3I4P8ypWytW9NiOq42wvdZWgC8ixL6Bu9nlijvXXz%2biyEWdKmPFhQJ7tzM QIm05A%2fxBcbEfrhsv1Y83LdE5xkAsO4yxngpfiPqYrSLhpoU9mzrQRC0NGSAgSu%2fV7it4aNhiy2r1Ry%2f0jYmGRABGYbjFlwkm6CkL7FErnlSFeEgcLBDtqQmXEb%2bf9kw5Al9y9lYhakuFL%2bmJrm3HL0M%2fS74N OF6xkIWrZDoqb3Y9Ht7enG0KvLcB7eyPWCQywyIKE5JkzLNX8YMs4%2bm7Tr%2b7fouYP49ChVTFSls8XMSEdS898x7G94vIYMzHvEQ8%2fQxyVMbCRWNzeRvP94rrWMeeme1FIsJk%2fPre0Z2maBz8X6Vkpewrq5xFWAx9l 9ZaRCWzLx6LcpO2x4QwHX6peuMtEP3YM0mQ%3d;path=/ipa;httponly;secure;']' 2026-03-15T09:26:37Z DEBUG storing cookie 'ipa_session=MagBearerToken=tOQOzYTYm3YG82Nz3I4P8ypWytW9NiOq42wvdZWgC8ixL6Bu9nlijvXXz%2biyEWdKmPFhQJ7tzMQIm05A%2fxBcbEfrhsv1Y83 LdE5xkAsO4yxngpfiPqYrSLhpoU9mzrQRC0NGSAgSu%2fV7it4aNhiy2r1Ry%2f0jYmGRABGYbjFlwkm6CkL7FErnlSFeEgcLBDtqQmXEb%2bf9kw5Al9y9lYhakuFL%2bmJrm3HL0M%2fS74NOF6xkIWrZDoqb3Y9Ht7enG0 KvLcB7eyPWCQywyIKE5JkzLNX8YMs4%2bm7Tr%2b7fouYP49ChVTFSls8XMSEdS898x7G94vIYMzHvEQ8%2fQxyVMbCRWNzeRvP94rrWMeeme1FIsJk%2fPre0Z2maBz8X6Vkpewrq5xFWAx9l9ZaRCWzLx6LcpO2x4QwHX6p euMtEP3YM0mQ%3d;' for principal host/ipa001.test.openqa.fedoraproject.org.FEDORAPROJECT.ORG 2026-03-15T09:26:37Z DEBUG Destroyed connection context.rpcclient_281473231564352 and then there's the traceback. In the journal there's nothing more obviously useful, just: Mar 15 05:26:37 ipa001.test.openqa.fedoraproject.org [9612]: GSSAPI client step 1 Mar 15 05:26:37 ipa001.test.openqa.fedoraproject.org [9612]: GSSAPI client step 1 Mar 15 05:26:37 ipa001.test.openqa.fedoraproject.org [9612]: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty) Filing against 44 for now as the most recent failure is from 44, but I'm fairly sure I've been seeing this for some time so it likely affects 42 and 43 too. I'll attach the full /var/log tarball.
Created attachment 2133724 [details] tarball of /var/log from a failed test
There's also a similar case with a different error, "KerberosError: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (2529638972): KDC returned error string: HANDLE_AUTHDATA": 2026-02-28T09:29:11Z DEBUG Starting external process 2026-02-28T09:29:11Z DEBUG args=['/usr/bin/certutil', '-d', 'sql:/tmp/tmpohivo1hj', '-A', '-n', 'CA certificate 1', '-t', 'C,,', '-a', '-f', '/tmp/tmpohivo1hj/pwdfile.tx t'] 2026-02-28T09:29:11Z DEBUG Process finished, return code=0 2026-02-28T09:29:11Z DEBUG stdout= 2026-02-28T09:29:11Z DEBUG stderr= 2026-02-28T09:29:11Z DEBUG failed to find session_cookie in persistent storage for principal 'host/ipa001.test.openqa.fedoraproject.org.FEDORAPROJECT.ORG' 2026-02-28T09:29:11Z DEBUG trying https://ipa001.test.openqa.fedoraproject.org/ipa/json 2026-02-28T09:29:11Z DEBUG Created connection context.rpcclient_281473129197120 2026-02-28T09:29:11Z DEBUG [try 1]: Forwarding 'schema' to json server 'https://ipa001.test.openqa.fedoraproject.org/ipa/json' 2026-02-28T09:29:11Z DEBUG New HTTP connection (ipa001.test.openqa.fedoraproject.org) 2026-02-28T09:29:11Z DEBUG HTTP connection destroyed (ipa001.test.openqa.fedoraproject.org) Traceback (most recent call last): File "/usr/lib/python3.14/site-packages/ipaclient/remote_plugins/__init__.py", line 118, in get_package plugins = api._remote_plugins ^^^^^^^^^^^^^^^^^^^ AttributeError: 'API' object has no attribute '_remote_plugins' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.14/site-packages/ipalib/rpc.py", line 514, in get_auth_info response = self._sec_context.step() File "/usr/lib/python3.14/site-packages/decorator.py", line 235, in fun return caller(func, *(extras + args), **kw) File "/usr/lib64/python3.14/site-packages/gssapi/_utils.py", line 165, in check_last_err return func(self, *args, **kwargs) File "/usr/lib/python3.14/site-packages/decorator.py", line 235, in fun return caller(func, *(extras + args), **kw) File "/usr/lib64/python3.14/site-packages/gssapi/_utils.py", line 131, in catch_and_return_token return func(self, *args, **kwargs) File "/usr/lib64/python3.14/site-packages/gssapi/sec_contexts.py", line 584, in step return self._initiator_step(token=token) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^ File "/usr/lib64/python3.14/site-packages/gssapi/sec_contexts.py", line 606, in _initiator_step res = rsec_contexts.init_sec_context(self._target_name, self._creds, self, self._mech, ...<2 lines>... self._channel_bindings, token) File "gssapi/raw/sec_contexts.pyx", line 188, in gssapi.raw.sec_contexts.init_sec_context gssapi.raw.misc.GSSError: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (2529638972): KDC returned error string: HANDLE_AUTHDA TA
I've seen this in upstream Azure CI in cases there is a CPU or memory pressure. It is effectively leading to race condition between two separate components (gssproxy and mod_auth_gssapi). The HANDLE_AUTHDATA comes when a user has no SID assigned yet. This, again, an asynchronous operation: when user is created, its POSIX ids assigned and SID generation is kicked off. The latter is asynchronous and may end later than when you start using this user, especially if there is CPU shortage.
So is there anything we can do to fix it?
Something you can fix on the OpenQA tests side is to validate the account was configured before using it. E.g., something like $ kinit admin $ ipa user-add testuser --first test --last user $ while test -z "`ipa user-show --all --raw testuser|grep -i ipantsecurityidentifier`" ; do sleep 5; done $ ipa passwd testuser $ do work ...
But we're hitting this *just when running ipa-server-install*, not when doing any manual operations on a deployed server.
Ah. 2026-03-15T09:26:37Z DEBUG [try 1]: Forwarding 'schema' to json server 'https://ipa001.test.openqa.fedoraproject.org/ipa/json' 2026-03-15T09:26:37Z DEBUG New HTTP connection (ipa001.test.openqa.fedoraproject.org) 2026-03-15T09:26:37Z DEBUG received Set-Cookie (<class 'list'>)'['ipa_session=MagBearerToken=tOQOzYTYm3YG82Nz3I4P8ypWytW9NiOq42wvdZWgC8ixL6Bu9nlijvXXz%2biyEWdKmPFhQJ7tzMQIm05A%2fxBcbEfrhsv1Y83LdE5xkAsO4yxngpfiPqYrSLhpoU9mzrQRC0NGSA gSu%2fV7it4aNhiy2r1Ry%2f0jYmGRABGYbjFlwkm6CkL7FErnlSFeEgcLBDtqQmXEb%2bf9kw5Al9y9lYhakuFL%2bmJrm3HL0M%2fS74NOF6xkIWrZDoqb3Y9Ht7enG0KvLcB7eyPWCQywyIKE5JkzLNX8YMs4%2bm7Tr%2b7fouYP49ChVTFSls8XMSEdS898x7G94vIYMzHvEQ8%2fQxyVMbCRWNzeRvP94 rrWMeeme1FIsJk%2fPre0Z2maBz8X6Vkpewrq5xFWAx9l9ZaRCWzLx6LcpO2x4QwHX6peuMtEP3YM0mQ%3d;path=/ipa;httponly;secure;']' 2026-03-15T09:26:37Z DEBUG storing cookie 'ipa_session=MagBearerToken=tOQOzYTYm3YG82Nz3I4P8ypWytW9NiOq42wvdZWgC8ixL6Bu9nlijvXXz%2biyEWdKmPFhQJ7tzMQIm05A%2fxBcbEfrhsv1Y83LdE5xkAsO4yxngpfiPqYrSLhpoU9mzrQRC0NGSAgSu%2fV7it4aNhiy2r1Ry%2 f0jYmGRABGYbjFlwkm6CkL7FErnlSFeEgcLBDtqQmXEb%2bf9kw5Al9y9lYhakuFL%2bmJrm3HL0M%2fS74NOF6xkIWrZDoqb3Y9Ht7enG0KvLcB7eyPWCQywyIKE5JkzLNX8YMs4%2bm7Tr%2b7fouYP49ChVTFSls8XMSEdS898x7G94vIYMzHvEQ8%2fQxyVMbCRWNzeRvP94rrWMeeme1FIsJk%2fPre0Z2 maBz8X6Vkpewrq5xFWAx9l9ZaRCWzLx6LcpO2x4QwHX6peuMtEP3YM0mQ%3d;' for principal host/ipa001.test.openqa.fedoraproject.org.FEDORAPROJECT.ORG leads to the following in HTTP error_log: [Sun Mar 15 05:26:37.273150 2026] [wsgi:error] [pid 9612:tid 10064] [remote 172.16.2.100:49256] ipa: DEBUG: WSGI wsgi_dispatch.__call__: [Sun Mar 15 05:26:37.273273 2026] [wsgi:error] [pid 9612:tid 10064] [remote 172.16.2.100:49256] ipa: DEBUG: WSGI jsonserver.__call__: [Sun Mar 15 05:26:37.273322 2026] [wsgi:error] [pid 9612:tid 10064] [remote 172.16.2.100:49256] ipa: DEBUG: KerberosWSGIExecutioner.__call__: and in krb5kdc.log: Mar 15 05:26:37 ipa001.test.openqa.fedoraproject.org krb5kdc[9333](Warning): MS-PAC not available. This makes FreeIPA vulnerable to privilege escalation exploit (CVE-2025-7493). Please generate SIDs to enable PAC support. Mar 15 05:26:37 ipa001.test.openqa.fedoraproject.org krb5kdc[9333](info): TGS_REQ : handle_authdata (-1780008435) Mar 15 05:26:37 ipa001.test.openqa.fedoraproject.org krb5kdc[9333](info): TGS_REQ (6 etypes {aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), camellia256-cts- cmac(26), camellia128-cts-cmac(25)}) 172.16.2.100: HANDLE_AUTHDATA: authtime 1773566796, HTTP/ipa001.test.openqa.fedoraproject.org.FEDORAPROJECT.ORG for ldap/ipa001.test.openqa.fedoraproject.org.FEDORAPROJE CT.ORG, Database has not been initialized Mar 15 05:26:37 ipa001.test.openqa.fedoraproject.org krb5kdc[9333](info): ... CONSTRAINED-DELEGATION s4u-client=host/ipa001.test.openqa.fedoraproject.org.FEDORAPROJECT.ORG Ok, this looks like one of conditions during initialization phase is broken after CVE-2025-7493 fixes.
Cool. Does it make sense that that would only happen *occasionally*?
Yes, it stumbles upon initialization of MS-PAC which should happen before the client (master server itself) is enrolled but again happens asynchronously. Basically, there are two places where a check is being done for MS-PAC configuration being initialized. In both these places we supposed to return a code that will say 'no processing needed' at this moment but we return KRB5_KDB_DBNOTINITED, not ENOENT. I have a test fix, will work with it tomorrow. I opened https://pagure.io/freeipa/issue/9962 upstream.
FEDORA-2026-3615b328cc (freeipa-4.13.1-11.fc45 and samba-4.24.0-7.fc45) has been submitted as an update to Fedora 45. https://bodhi.fedoraproject.org/updates/FEDORA-2026-3615b328cc
FEDORA-2026-a80002c130 (freeipa-4.13.1-9.fc44 and samba-4.24.0-6.fc44) has been submitted as an update to Fedora 44. https://bodhi.fedoraproject.org/updates/FEDORA-2026-a80002c130
FEDORA-2026-3615b328cc (freeipa-4.13.1-11.fc45 and samba-4.24.0-7.fc45) has been pushed to the Fedora 45 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2026-a80002c130 has been pushed to the Fedora 44 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2026-a80002c130` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2026-a80002c130 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2026-a80002c130 (freeipa-4.13.1-9.fc44 and samba-4.24.0-6.fc44) has been pushed to the Fedora 44 stable repository. If problem still persists, please make note of it in this bug report.
Still seems to be happening: https://openqa.fedoraproject.org/tests/4535341#step/role_deploy_domain_controller/31 Log tarball at https://openqa.fedoraproject.org/tests/4535341/file/role_deploy_domain_controller-var_log.tar.gz
KDC logs show: Apr 02 05:47:43 ipa001.test.openqa.fedoraproject.org krb5kdc[9377](Warning): MS-PAC not available during ticket verification. This makes FreeIPA vulnerable to privilege escalation exploit (CVE-2025-7493). Please generate SIDs to enable PAC support. Apr 02 05:47:43 ipa001.test.openqa.fedoraproject.org krb5kdc[9377](info): TGS_REQ : handle_authdata (-1780008435) Apr 02 05:47:43 ipa001.test.openqa.fedoraproject.org krb5kdc[9377](info): TGS_REQ (6 etypes {aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), camellia256-cts-cmac(26), camellia128-cts-cmac(25)}) 172.16.2.100: HANDLE_AUTHDATA: authtime 1775123263, ipa-dnskeysyncd/ipa001.test.openqa.fedoraproject.org.FEDORAPROJECT.ORG for ldap/ipa001.test.openqa.fedoraproject.org.FEDORAPROJECT.ORG, Database has not been initialized Apr 02 05:47:43 ipa001.test.openqa.fedoraproject.org krb5kdc[9377](info): closing down fd 11 Apr 02 05:47:43 ipa001.test.openqa.fedoraproject.org krb5kdc[9376](info): TGS_REQ (6 etypes {aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), camellia256-cts-cmac(26), camellia128-cts-cmac(25)}) 172.16.2.100: ISSUE: authtime 1775123263, etypes {rep=aes256-cts-hmac-sha384-192(20), tkt=camellia256-cts-cmac(26), ses=aes256-cts-hmac-sha384-192(20)}, ipa-dnskeysyncd/ipa001.test.openqa.fedoraproject.org.FEDORAPROJECT.ORG for ldap/ipa001.test.openqa.fedoraproject.org.FEDORAPROJECT.ORG Apr 02 05:47:43 ipa001.test.openqa.fedoraproject.org krb5kdc[9376](info): closing down fd 11 Apr 02 05:47:45 ipa001.test.openqa.fedoraproject.org krb5kdc[9377](Warning): MS-PAC not available during ticket verification. This makes FreeIPA vulnerable to privilege escalation exploit (CVE-2025-7493). Please generate SIDs to enable PAC support. Apr 02 05:47:45 ipa001.test.openqa.fedoraproject.org krb5kdc[9377](info): TGS_REQ : handle_authdata (-1780008435) Apr 02 05:47:45 ipa001.test.openqa.fedoraproject.org krb5kdc[9377](info): TGS_REQ (6 etypes {aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), camellia256-cts-cmac(26), camellia128-cts-cmac(25)}) 172.16.2.100: HANDLE_AUTHDATA: authtime 1775123262, HTTP/ipa001.test.openqa.fedoraproject.org.FEDORAPROJECT.ORG for ldap/ipa001.test.openqa.fedoraproject.org.FEDORAPROJECT.ORG, Database has not been initialized Apr 02 05:47:45 ipa001.test.openqa.fedoraproject.org krb5kdc[9377](info): ... CONSTRAINED-DELEGATION s4u-client=host/ipa001.test.openqa.fedoraproject.org.FEDORAPROJECT.ORG Apr 02 05:47:45 ipa001.test.openqa.fedoraproject.org krb5kdc[9377](info): closing down fd 11 At the same time, directory server was restarting Apr 2 05:47:37 ipa001 ns-slapd[9330]: [02/Apr/2026:05:47:37.883191640 -0400] - INFO - main - slapd stopped. Apr 2 05:47:37 ipa001 systemd[1]: dirsrv: Deactivated successfully. Apr 2 05:47:37 ipa001 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=dirsrv@TEST-OPENQA-FEDORAPROJECT-ORG comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Apr 2 05:47:37 ipa001 systemd[1]: Stopped dirsrv - 389 Directory Server TEST-OPENQA-FEDORAPROJECT-ORG.. Apr 2 05:47:37 ipa001 systemd[1]: dirsrv: Consumed 2.270s CPU time over 27.558s wall clock time, 84.9M memory peak. Apr 2 05:47:37 ipa001 audit: BPF prog-id=987 op=UNLOAD Apr 2 05:47:37 ipa001 audit: BPF prog-id=999 op=LOAD Apr 2 05:47:37 ipa001 systemd[1]: Starting dirsrv - 389 Directory Server TEST-OPENQA-FEDORAPROJECT-ORG.... .... Apr 2 05:47:39 ipa001 systemd[1]: Started dirsrv - 389 Directory Server TEST-OPENQA-FEDORAPROJECT-ORG.. ... Apr 2 05:47:44 ipa001 ns-slapd[10052]: [02/Apr/2026:05:47:44.336153000 -0400] - ERR - schema-compat-plugin - schema-compat-plugin tree scan will start in about 5 seconds! Apr 2 05:47:45 ipa001 [9646]: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty) Apr 2 05:47:49 ipa001 ns-slapd[10052]: [02/Apr/2026:05:47:49.346290560 -0400] - ERR - schema-compat-plugin - warning: no entries set up under ou=sudoers,dc=test,dc=openqa,dc=fedoraproject,dc=org Apr 2 05:47:49 ipa001 ns-slapd[10052]: [02/Apr/2026:05:47:49.363312440 -0400] - ERR - schema-compat-plugin - warning: no entries set up under cn=ng, cn=compat,dc=test,dc=openqa,dc=fedoraproject,dc=org Apr 2 05:47:49 ipa001 ns-slapd[10052]: [02/Apr/2026:05:47:49.370680400 -0400] - ERR - schema-compat-plugin - warning: no entries set up under cn=computers, cn=compat,dc=test,dc=openqa,dc=fedoraproject,dc=org Apr 2 05:47:49 ipa001 ns-slapd[10052]: [02/Apr/2026:05:47:49.372663920 -0400] - ERR - schema-compat-plugin - Finished plugin initialization. So this is unfortunate attempt to connect at the 389-ds restart time, but the error handling in KDC is correct. This connect happens as part of ipa-client-install run from within ipa-server-install, to configure client side on IPA master. So we should probably wait to ensure the whole restart settles down first and then start ipa-client-install: 2026-04-02T09:47:42Z DEBUG [IPA Discovery] 2026-04-02T09:47:42Z DEBUG Starting IPA discovery with domain=test.openqa.fedoraproject.org, servers=['ipa001.test.openqa.fedoraproject.org'], hostname=ipa001.test.openqa.fedoraproject.org 2026-04-02T09:47:42Z DEBUG Server and domain forced 2026-04-02T09:47:42Z DEBUG [Kerberos realm search] 2026-04-02T09:47:42Z DEBUG Kerberos realm forced 2026-04-02T09:47:42Z DEBUG [LDAP server check] .. 2026-04-02T09:47:42Z DEBUG Created connection context.rpcclient_281473187507776 2026-04-02T09:47:42Z DEBUG [try 1]: Forwarding 'schema' to json server 'https://ipa001.test.openqa.fedoraproject.org/ipa/json' 2026-04-02T09:47:42Z DEBUG New HTTP connection (ipa001.test.openqa.fedoraproject.org) 2026-04-02T09:47:45Z DEBUG received Set-Cookie (<class 'list'>)'['ipa_session=MagBearerToken=LMTmp%2f44nUbfJDgfp2ETqJKtlxOxcBkts58fBKlg1A%2fOvDHfRmJoFmOMGt7XYEgH%2froeOpbeX%2fJ7bvZdf9VpJA5O9Zt2UKlLHp63FT4Fs7rgAwQKhGH8HDqeu84t82b53Ikf3mL5anCo5iyFUzYZfXtCK8iiynT3Sxg5j7ScWjgTubLUwjpCXi%2b2H59ZIv5pvF0FYEZmXSR72YQOFvPZuuTHSddUisx5%2fAY7hbRNSxpvBJR4TWqeYrsAsGLl3n5ZHLA9O9yi%2fOrQZ0YakGtPjCfAG6XELVNUzLirZvV4ovy9X5zoBVOTugXOpHdPf63qPT1U7bdFtdCaqJEaXf5gtlymDhaYNfLu%2f4IYVtcpLzCheTHqxLBtOnFsp0QbsvWkPj1fKKzbeVB7TPpll%2fooKTTq87gcv1tLyw7oCH7zMT4%3d;path=/ipa;httponly;secure;']' 2026-04-02T09:47:45Z DEBUG storing cookie 'ipa_session=MagBearerToken=LMTmp%2f44nUbfJDgfp2ETqJKtlxOxcBkts58fBKlg1A%2fOvDHfRmJoFmOMGt7XYEgH%2froeOpbeX%2fJ7bvZdf9VpJA5O9Zt2UKlLHp63FT4Fs7rgAwQKhGH8HDqeu84t82b53Ikf3mL5anCo5iyFUzYZfXtCK8iiynT3Sxg5j7ScWjgTubLUwjpCXi%2b2H59ZIv5pvF0FYEZmXSR72YQOFvPZuuTHSddUisx5%2fAY7hbRNSxpvBJR4TWqeYrsAsGLl3n5ZHLA9O9yi%2fOrQZ0YakGtPjCfAG6XELVNUzLirZvV4ovy9X5zoBVOTugXOpHdPf63qPT1U7bdFtdCaqJEaXf5gtlymDhaYNfLu%2f4IYVtcpLzCheTHqxLBtOnFsp0QbsvWkPj1fKKzbeVB7TPpll%2fooKTTq87gcv1tLyw7oCH7zMT4%3d;' for principal host/ipa001.test.openqa.fedoraproject.org.FEDORAPROJECT.ORG 2026-04-02T09:47:45Z DEBUG Destroyed connection context.rpcclient_281473187507776 2026-04-02T09:47:45Z DEBUG File "/usr/lib/python3.14/site-packages/ipapython/admintool.py", line 219, in execute return_value = self.run() File "/usr/lib/python3.14/site-packages/ipapython/install/cli.py", line 343, in run return cfgr.run() ~~~~~~~~^^ File "/usr/lib/python3.14/site-packages/ipapython/install/core.py", line 360, in run return self.execute() ~~~~~~~~~~~~^^ File "/usr/lib/python3.14/site-packages/ipapython/install/core.py", line 386, in execute for rval in self._executor(): ~~~~~~~~~~~~~~^^ File "/usr/lib/python3.14/site-packages/ipapython/install/core.py", line 435, in __runner exc_handler(exc_info) ~~~~~~~~~~~^^^^^^^^^^ File "/usr/lib/python3.14/site-packages/ipapython/install/core.py", line 468, in _handle_execute_exception self._handle_exception(exc_info) ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ File "/usr/lib/python3.14/site-packages/ipapython/install/core.py", line 458, in _handle_exception six.reraise(*exc_info) ~~~~~~~~~~~^^^^^^^^^^^ File "/usr/lib/python3.14/site-packages/six.py", line 724, in reraise raise value File "/usr/lib/python3.14/site-packages/ipapython/install/core.py", line 425, in __runner step() ~~~~^^ File "/usr/lib/python3.14/site-packages/ipapython/install/core.py", line 419, in step_next return next(self.__gen) File "/usr/lib/python3.14/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from six.reraise(*exc_info) ~~~~~~~~~~~^^^^^^^^^^^ File "/usr/lib/python3.14/site-packages/six.py", line 724, in reraise raise value File "/usr/lib/python3.14/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python3.14/site-packages/ipapython/install/core.py", line 663, in _configure next(executor) ~~~~^^^^^^^^^^ File "/usr/lib/python3.14/site-packages/ipapython/install/core.py", line 435, in __runner exc_handler(exc_info) ~~~~~~~~~~~^^^^^^^^^^ File "/usr/lib/python3.14/site-packages/ipapython/install/core.py", line 468, in _handle_execute_exception self._handle_exception(exc_info) ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ File "/usr/lib/python3.14/site-packages/ipapython/install/core.py", line 526, in _handle_exception self.__parent._handle_exception(exc_info) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ File "/usr/lib/python3.14/site-packages/ipapython/install/core.py", line 458, in _handle_exception six.reraise(*exc_info) ~~~~~~~~~~~^^^^^^^^^^^ File "/usr/lib/python3.14/site-packages/six.py", line 724, in reraise raise value File "/usr/lib/python3.14/site-packages/ipapython/install/core.py", line 523, in _handle_exception super(ComponentBase, self)._handle_exception(exc_info) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ File "/usr/lib/python3.14/site-packages/ipapython/install/core.py", line 458, in _handle_exception six.reraise(*exc_info) ~~~~~~~~~~~^^^^^^^^^^^ File "/usr/lib/python3.14/site-packages/six.py", line 724, in reraise raise value File "/usr/lib/python3.14/site-packages/ipapython/install/core.py", line 425, in __runner step() ~~~~^^ File "/usr/lib/python3.14/site-packages/ipapython/install/core.py", line 419, in step_next return next(self.__gen) File "/usr/lib/python3.14/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from six.reraise(*exc_info) ~~~~~~~~~~~^^^^^^^^^^^ File "/usr/lib/python3.14/site-packages/six.py", line 724, in reraise raise value File "/usr/lib/python3.14/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python3.14/site-packages/ipapython/install/common.py", line 65, in _install for unused in self._installer(self.parent): ~~~~~~~~~~~~~~~^^^^^^^^^^^^^ File "/usr/lib/python3.14/site-packages/ipaclient/install/client.py", line 4377, in main install(self) ~~~~~~~^^^^^^ File "/usr/lib/python3.14/site-packages/ipaclient/install/client.py", line 2903, in install _install(options, dict()) ~~~~~~~~^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.14/site-packages/ipaclient/install/client.py", line 117, in inner func(options, tdict) ~~~~^^^^^^^^^^^^^^^^ File "/usr/lib/python3.14/site-packages/ipaclient/install/client.py", line 3217, in _install api.finalize() ~~~~~~~~~~~~^^ File "/usr/lib/python3.14/site-packages/ipalib/plugable.py", line 765, in finalize self.__do_if_not_done('load_plugins') ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^ File "/usr/lib/python3.14/site-packages/ipalib/plugable.py", line 432, in __do_if_not_done getattr(self, name)() ~~~~~~~~~~~~~~~~~~~^^ File "/usr/lib/python3.14/site-packages/ipalib/plugable.py", line 644, in load_plugins for package in self.packages: ^^^^^^^^^^^^^ File "/usr/lib/python3.14/site-packages/ipalib/__init__.py", line 1024, in packages ipaclient.remote_plugins.get_package(self), ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/usr/lib/python3.14/site-packages/ipaclient/remote_plugins/__init__.py", line 126, in get_package plugins = schema.get_package(server_info, client) File "/usr/lib/python3.14/site-packages/ipaclient/remote_plugins/schema.py", line 545, in get_package schema = Schema(client) File "/usr/lib/python3.14/site-packages/ipaclient/remote_plugins/schema.py", line 394, in __init__ fingerprint, ttl = self._fetch(client, ignore_cache=read_failed) ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.14/site-packages/ipaclient/remote_plugins/schema.py", line 419, in _fetch schema = client.forward(u'schema', **kwargs)['result'] ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.14/site-packages/ipalib/rpc.py", line 1028, in forward return self._call_command(command, params) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.14/site-packages/ipalib/rpc.py", line 1004, in _call_command return command(*params) File "/usr/lib/python3.14/site-packages/ipalib/rpc.py", line 1158, in _call return self.__request(name, args) ~~~~~~~~~~~~~~^^^^^^^^^^^^ File "/usr/lib/python3.14/site-packages/ipalib/rpc.py", line 1152, in __request raise error_class(**kw) 2026-04-02T09:47:45Z DEBUG The ipa-client-install command failed, exception: ACIError: Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty) 2026-04-02T09:47:45Z ERROR Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty) 2026-04-02T09:47:45Z ERROR The ipa-client-install command failed. See /var/log/ipaclient-install.log for more information
Ultimately, I think the real issue here is a CPU usage. You have two not so fast CPUs, running multiple daemons in parallel causes CPU contention, so tasks in those daemons take longer than expected. It is hard to detect when a service is really ready for use (as opposed to what systemd tells us). On IPA installer side we need to wait until the planned restart is finished before starting 'ipa-client-install --on-master'.
> Ultimately, I think the real issue here is a CPU usage. You have two not so fast CPUs, running multiple daemons in parallel causes CPU contention, so tasks in those daemons take longer than expected. That's plausible, yeah. There is likely contention both within the VM and across all the many many VMs running on the openQA worker host (we run up to 60 tests simultaneously per host). Is there anything I can do to mitigate this? Would maybe setting the FreeIPA server test VMs to a single emulated CPU help?
If anything, giving a single CPU will make things worse. I'm going to look at this in couple weeks -- next week is SambaXP and I'll barely have any time.