Bug 2424132
| Summary: | Upgrade from freeipa-4.12.5-3 and 389-ds-base-3.1.3-10 to latest rawhide fails | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Jan Pazdziora <adelton> |
| Component: | 389-ds-base | Assignee: | Viktor Ashirov <vashirov> |
| Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | rawhide | CC: | abokovoy, awilliam, ftrivino, ipa-maint, jachapma, mhjacks, mreynolds, rcritten, robatino, spichugi, ssorce, tbordaz, twoerner, vashirov |
| Target Milestone: | --- | Keywords: | Regression |
| Target Release: | --- | Flags: | fedora-admin-xmlrpc:
mirror+
|
| Hardware: | Unspecified | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | 389-ds-base-3.2.0-5.fc44 | Doc Type: | --- |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2026-01-12 13:47:29 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 2362357 | ||
|
Description
Jan Pazdziora
2025-12-20 20:02:27 UTC
We also see new and similar "creating replica fails after master got upgraded" failure on Rocky Linux 8 where 389-ds-base got upgraded from 1.4.3.39-15.* to 1.4.3.39-19.*: https://github.com/freeipa/freeipa-container/issues/710. It is quite strange coincidence that things started to fail on the same day on two OSes that are that far apart. Is there a common patch which went to both that could help narrow down the investigation? This looks the same as https://bugzilla.redhat.com/show_bug.cgi?id=2424526 which already gained BetaBlocker status. @adelton, do you mind if I'll close this one as a duplicate of the other one? We need to clone 2424526 upstream and work on it anyway. Well, https://bugzilla.redhat.com/show_bug.cgi?id=2424526 says "F43 to F44 upgrade works" while here we describe a failure of upgrade from Fedora 43 to rawhide, and even from rawhide to rawhide. Plus we also note here as additional datapoint that upgrades of Rocky Linux 8 and AlmaLinux 8 (and I assume RHEL 8 as well) started to fail at the same time. I'm a bit worried worried that some fast bandaid will be done for https://bugzilla.redhat.com/show_bug.cgi?id=2424526 to unblock the Beta without really digging into the root cause and fix. I talked to Victor and he noted there is at least one customer case with a similar index-related issue. This would explain the problem appearing in different releases. Both bugs now moved to 389-ds-base component. Hello Viktor, thanks for the active investigation and already having a PR upstream. Is there a RHEL specific Jira issue as the engineering tracker for that customer case? Seeing that the parentId patch mentioned as the culprit in the upstream issue https://github.com/389ds/389-ds-base/issues/7172 is in Rocky Linux 8 (https://git.rockylinux.org/staging/rpms/389-ds-base/-/commit/179e7a97665e61b81b6bbfc68b30c3adf95ae51c#1ed7112a87892a56d49cab4c6afc319d06fcd64c) makes me believe that this bugzilla and upstream PR https://github.com/389ds/389-ds-base/pull/7173 might actually be the same as the Rocky Linux 8 and AlmaLinux 8 issue https://github.com/freeipa/freeipa-container/issues/710. So I'd like to be able to link the place where the backport might happen from that FreeIPA container's GitHub issue. I did https://issues.redhat.com/issues/?jql=project %3D RHEL and component %3D 389-ds-base order by created but neither of those publicly visible issues seems to ring a bell. Hello Jan, first of all, thank you for the detailed reproducer! RHEL downstream work is tracked in https://issues.redhat.com/browse/RHEL-137786. I've changed the visibility of RHEL ticket and you should be able to see it now. Thanks! Perfect, thank you. *** Bug 2424526 has been marked as a duplicate of this bug. *** FEDORA-2026-3f562e9007 (389-ds-base-3.2.0-3.fc44) has been submitted as an update to Fedora 44. https://bodhi.fedoraproject.org/updates/FEDORA-2026-3f562e9007 FEDORA-2026-54d5a579fe (389-ds-base-3.2.0-4.fc44) has been submitted as an update to Fedora 44. https://bodhi.fedoraproject.org/updates/FEDORA-2026-54d5a579fe I tried to take a Fedora rawhide machine with the old packages installed and ipa-server-install run, basically steps 1 - 3 from comment 0. I then run # dnf upgrade -y https://kojipkgs.fedoraproject.org//packages/389-ds-base/3.2.0/4.fc44/$( uname -m )/389-ds-base-3.2.0-4.fc44.$( uname -m ).rpm \ https://kojipkgs.fedoraproject.org//packages/389-ds-base/3.2.0/4.fc44/$( uname -m )/389-ds-base-libs-3.2.0-4.fc44.$( uname -m ).rpm \ https://kojipkgs.fedoraproject.org//packages/389-ds-base/3.2.0/4.fc44/$( uname -m )/389-ds-base-robdb-libs-3.2.0-4.fc44.$( uname -m ).rpm \ https://kojipkgs.fedoraproject.org//packages/389-ds-base/3.2.0/4.fc44/noarch/python3-lib389-3.2.0-4.fc44.noarch.rpm [ ... this passed ... ] # ipa-server-upgrade Upgrading IPA:. Estimated time: 1 minute 30 seconds [1/11]: stopping directory server [2/11]: saving configuration [3/11]: disabling listeners [4/11]: enabling DS global lock [5/11]: disabling Schema Compat [6/11]: starting directory server [7/11]: updating schema [8/11]: upgrading server [9/11]: stopping directory server [10/11]: restoring configuration [11/11]: starting directory server Done. Update complete Upgrading IPA services Upgrading the configuration of the IPA services Disabled p11-kit-proxy [Verifying that root certificate is published] [Migrate CRL publish directory] Publish directory already set to new location [Verifying that KDC configuration is using ipa-kdb backend] [Fix DS schema file syntax] [Removing RA cert from DS NSS database] [Enable sidgen and extdom plugins by default] [Updating HTTPD service IPA configuration] [Updating HTTPD service IPA WSGI configuration] [Moving HTTPD service keytab to gssproxy] [Removing self-signed CA] [Removing Dogtag 9 CA] [Set OpenSSL engine or provider for BIND] Restarting ipa-dnskeysyncd [Checking for deprecated KDC configuration files] [Checking for deprecated backups of Samba configuration files] dnssec-validation yes [Add missing CA DNS records] Updating DNS system records IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. Unexpected error - see /var/log/ipaupgrade.log for details: DatabaseError: Operations error: The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information The /var/log/ipaupgrade.log ends with 2026-01-10T11:16:16Z DEBUG dnsrecord_find(<DNS name example.test.>, 'ipa-ca', structured=False, all=False, raw=False, version='2.254', pkey_only=False) 2026-01-10T11:16:16Z DEBUG Updating DNS system records 2026-01-10T11:16:16Z DEBUG raw: server_find(None, version='2.254', no_members=False, servrole='IPA master') 2026-01-10T11:16:16Z DEBUG server_find(None, all=False, raw=False, version='2.254', no_members=False, pkey_only=False, servrole=('IPA master',)) 2026-01-10T11:16:16Z DEBUG raw: server_role_find(None, server_server=None, role_servrole='IPA master', status='enabled', include_master=True, version='2.254') 2026-01-10T11:16:16Z DEBUG server_role_find(None, server_server=None, role_servrole='IPA master', status='enabled', include_master=True, all=False, raw=False, version='2.254') 2026-01-10T11:16:16Z DEBUG raw: topologysuffix_find(None, all=True, raw=True, version='2.254') 2026-01-10T11:16:16Z DEBUG topologysuffix_find(None, all=True, raw=True, version='2.254', pkey_only=False) 2026-01-10T11:16:16Z DEBUG raw: server_role_find(None, server_server='ipa.example.test', status='enabled', include_master=True, version='2.254') 2026-01-10T11:16:16Z DEBUG server_role_find(None, server_server='ipa.example.test', status='enabled', include_master=True, all=False, raw=False, version='2.254') 2026-01-10T11:16:16Z DEBUG raw: dnszone_show(<DNS name example.test.>, version='2.254') 2026-01-10T11:16:16Z DEBUG dnszone_show(<DNS name example.test.>, rights=False, all=False, raw=False, version='2.254') 2026-01-10T11:16:16Z DEBUG raw: dnsrecord_del(<DNS name example.test.>, <DNS name ipa-ca.example.test.>, del_all=True, version='2.254') 2026-01-10T11:16:16Z DEBUG dnsrecord_del(<DNS name example.test.>, <DNS name ipa-ca.example.test.>, del_all=True, structured=False, raw=False, version='2.254') 2026-01-10T11:16:16Z DEBUG raw: dnsrecord_delentry(<DNS name example.test.>, (<DNS name ipa-ca.example.test.>,), version='2.254') 2026-01-10T11:16:16Z DEBUG dnsrecord_delentry(<DNS name example.test.>, (<DNS name ipa-ca.example.test.>,), continue=False, version='2.254') 2026-01-10T11:16:16Z DEBUG Unhandled LDAPError: OPERATIONS_ERROR: {'msgtype': 107, 'msgid': 36, 'result': 1, 'desc': 'Operations error', 'ctrls': []} 2026-01-10T11:16:16Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. 2026-01-10T11:16:16Z DEBUG File "/usr/lib/python3.14/site-packages/ipapython/admintool.py", line 219, in execute return_value = self.run() File "/usr/lib/python3.14/site-packages/ipaserver/install/ipa_server_upgrade.py", line 54, in run server.upgrade() ~~~~~~~~~~~~~~^^ File "/usr/lib/python3.14/site-packages/ipaserver/install/server/upgrade.py", line 2083, in upgrade upgrade_configuration() ~~~~~~~~~~~~~~~~~~~~~^^ File "/usr/lib/python3.14/site-packages/ipaserver/install/server/upgrade.py", line 1904, in upgrade_configuration upgrade_bind(fstore) ~~~~~~~~~~~~^^^^^^^^ File "/usr/lib/python3.14/site-packages/ipaserver/install/server/upgrade.py", line 1499, in upgrade_bind add_ca_dns_records(bind) ~~~~~~~~~~~~~~~~~~^^^^^^ File "/usr/lib/python3.14/site-packages/ipaserver/install/server/upgrade.py", line 882, in add_ca_dns_records bind.update_system_records() ~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/usr/lib/python3.14/site-packages/ipaserver/install/bindinstance.py", line 1315, in update_system_records ) = system_records.update_dns_records() ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/usr/lib/python3.14/site-packages/ipaserver/dns_data_management.py", line 523, in update_dns_records self.update_base_records(), ~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/usr/lib/python3.14/site-packages/ipaserver/dns_data_management.py", line 470, in update_base_records self.api_instance.Command.dnsrecord_del( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ self.domain_abs, r_name, del_all=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.14/site-packages/ipalib/frontend.py", line 477, in __call__ return self.__do_call(*args, **options) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.14/site-packages/ipalib/frontend.py", line 544, in __do_call ret = self.run(*args, **options) File "/usr/lib/python3.14/site-packages/ipalib/frontend.py", line 885, in run return self.execute(*args, **options) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.14/site-packages/ipaserver/plugins/dns.py", line 3955, in execute result = self.obj.methods.delentry(*keys, version=options['version']) File "/usr/lib/python3.14/site-packages/ipalib/frontend.py", line 477, in __call__ return self.__do_call(*args, **options) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.14/site-packages/ipalib/frontend.py", line 544, in __do_call ret = self.run(*args, **options) File "/usr/lib/python3.14/site-packages/ipalib/frontend.py", line 885, in run return self.execute(*args, **options) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.14/site-packages/ipaserver/plugins/baseldap.py", line 1690, in execute delete_entry(pkey) ~~~~~~~~~~~~^^^^^^ File "/usr/lib/python3.14/site-packages/ipaserver/plugins/baseldap.py", line 1665, in delete_entry self._exc_wrapper(nkeys, options, ldap.delete_entry)(dn) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^ File "/usr/lib/python3.14/site-packages/ipaserver/plugins/baseldap.py", line 1207, in wrapped return func(*call_args, **call_kwargs) File "/usr/lib/python3.14/site-packages/ipaserver/plugins/baseldap.py", line 1215, in exc_func return callback( self, keys, options, e, call_func, *args, **kwargs) File "/usr/lib/python3.14/site-packages/ipaserver/plugins/baseldap.py", line 1711, in exc_callback raise exc File "/usr/lib/python3.14/site-packages/ipaserver/plugins/baseldap.py", line 1207, in wrapped return func(*call_args, **call_kwargs) File "/usr/lib/python3.14/site-packages/ipapython/ipaldap.py", line 1929, in delete_entry super(LDAPCache, self).delete_entry(dn) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^ File "/usr/lib/python3.14/site-packages/ipapython/ipaldap.py", line 1740, in delete_entry with self.error_handler(): ~~~~~~~~~~~~~~~~~~^^ File "/usr/lib64/python3.14/contextlib.py", line 162, in __exit__ self.gen.throw(value) ~~~~~~~~~~~~~~^^^^^^^ File "/usr/lib/python3.14/site-packages/ipapython/ipaldap.py", line 1166, in error_handler raise errors.DatabaseError(desc=desc, info=info) 2026-01-10T11:16:16Z DEBUG The ipa-server-upgrade command failed, exception: DatabaseError: Operations error: 2026-01-10T11:16:16Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details: DatabaseError: Operations error: 2026-01-10T11:16:16Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information In /var/log/dirsrv/slapd-EXAMPLE-TEST/errors I can see [10/Jan/2026:11:16:13.365328116 +0000] - NOTICE - ldbm_back_search - Unindexed search: search base="ou=authorities,ou=ca,o=ipaca" scope=2 filter="(objectClass=*)" conn=4 op=1 [10/Jan/2026:11:16:13.388642260 +0000] - NOTICE - ldbm_back_search - Unindexed search: search base="ou=certificateProfiles,ou=ca,o=ipaca" scope=2 filter="(objectClass=*)" conn=5 op=2 [10/Jan/2026:11:16:16.182716375 +0000] - WARN - dbmdb_open_dbi_from_filename - Attempt to open to open dbi userRoot/.default while txn is already pending. Usually that means that the index must be reindex. Root cause is likely that last import of reindex failed or that the index was created but not yet reindexed). [10/Jan/2026:11:16:16.207766840 +0000] - WARN - slapi_log_backtrace - [0] /usr/lib64/dirsrv/libslapd.so.0(+0x6d148) [0x7f193666d148] [10/Jan/2026:11:16:16.223453599 +0000] - WARN - slapi_log_backtrace - [1] /usr/lib64/dirsrv/plugins/libback-ldbm.so(dbmdb_open_dbi_from_filename+0x36a) [0x7f1931ade84a] [10/Jan/2026:11:16:16.239376193 +0000] - WARN - slapi_log_backtrace - [2] /usr/lib64/dirsrv/plugins/libback-ldbm.so(dbmdb_get_db+0xbd) [0x7f1931ade92d] [10/Jan/2026:11:16:16.244379747 +0000] - WARN - slapi_log_backtrace - [3] /usr/lib64/dirsrv/plugins/libback-ldbm.so(dblayer_get_index_file+0xb1) [0x7f1931a61631] [10/Jan/2026:11:16:16.260432957 +0000] - WARN - slapi_log_backtrace - [4] /usr/lib64/dirsrv/plugins/libback-ldbm.so(+0x125af) [0x7f1931a635af] [10/Jan/2026:11:16:16.265241687 +0000] - WARN - slapi_log_backtrace - [5] /usr/lib64/dirsrv/plugins/libback-ldbm.so(ldbm_ancestorid_index_entry+0x58) [0x7f1931a638e8] [10/Jan/2026:11:16:16.279754023 +0000] - WARN - slapi_log_backtrace - [6] /usr/lib64/dirsrv/plugins/libback-ldbm.so(index_addordel_entry+0x300) [0x7f1931a75b10] [10/Jan/2026:11:16:16.283435512 +0000] - WARN - slapi_log_backtrace - [7] /usr/lib64/dirsrv/plugins/libback-ldbm.so(ldbm_back_delete+0x13a2) [0x7f1931a8e742] [10/Jan/2026:11:16:16.297919556 +0000] - WARN - slapi_log_backtrace - [8] /usr/lib64/dirsrv/libslapd.so.0(+0x270d0) [0x7f19366270d0] [10/Jan/2026:11:16:16.301746535 +0000] - WARN - slapi_log_backtrace - [9] /usr/lib64/dirsrv/libslapd.so.0(do_delete+0x10f) [0x7f193662748f] [10/Jan/2026:11:16:16.316020611 +0000] - WARN - slapi_log_backtrace - [10] /usr/bin/ns-slapd(+0x126a7) [0x561d7465e6a7] [10/Jan/2026:11:16:16.319789194 +0000] - WARN - slapi_log_backtrace - [11] /lib64/libnspr4.so(+0x24d13) [0x7f1936d31d13] [10/Jan/2026:11:16:16.334380832 +0000] - WARN - slapi_log_backtrace - [12] /lib64/libc.so.6(+0x7227a) [0x7f193647f27a] [10/Jan/2026:11:16:16.338131058 +0000] - WARN - slapi_log_backtrace - [13] /lib64/libc.so.6(+0xf4d5c) [0x7f1936501d5c] [10/Jan/2026:11:16:16.352379770 +0000] - ERR - ldbm_ancestorid_index_update - ancestorid.c BAD 13130, err=-30798 Unexpected dbimpl error code [10/Jan/2026:11:16:16.356188014 +0000] - ERR - ldbm_back_delete - index_addordel_entry(idnsname=ipa-ca,idnsname=example.test.,cn=dns,dc=example,dc=test, 0x26) failed (-30798) New PR: https://github.com/389ds/389-ds-base/pull/7180 I've prepared a copr repo with the latest patch, while the PR is on review: https://copr.fedorainfracloud.org/coprs/vashirov/bz2424132/ It passed your reproducer on my VM. Thanks. I confirm that in the containerized FreeIPA setup from which the original issue https://github.com/freeipa/freeipa-container/issues/709 comes, merely adding RUN dnf copr enable -y vashirov/bz2424132 to the Dockerfile makes upgrades (from fedora-42-4.12.2 and from fedora-43-4.12.5) pass again: https://github.com/adelton/freeipa-container/actions/runs/20882796465. FEDORA-2026-092b6c1b30 (389-ds-base-3.2.0-5.fc44) has been submitted as an update to Fedora 44. https://bodhi.fedoraproject.org/updates/FEDORA-2026-092b6c1b30 FEDORA-2026-092b6c1b30 (389-ds-base-3.2.0-5.fc44) has been pushed to the Fedora 44 stable repository. If problem still persists, please make note of it in this bug report. |