Bug 2225626

Summary: The ipa-server-upgrade failed, exception: EmptyResult: no matching entry found
Product: Red Hat Enterprise Linux 8 Reporter: Rakesh Kumar <rakkumar>
Component: ipaAssignee: Florence Blanc-Renaud <frenaud>
Status: NEW --- QA Contact: ipa-qe
Severity: high Docs Contact:
Priority: medium    
Version: 8.7CC: rcritten, tscherf
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Rakesh Kumar 2023-07-25 15:20:32 UTC
Description of problem: IPA-server-upgrade is failing and producing the error :The ipa-server-upgrade command failed, exception: EmptyResult: no matching entry found


OS Version: rhel8.7

IPA: ipa-server-4.9.10-9.module+el8.7.0+17437+cf46f77f.x86_64 

SSSD: sssd-ipa-2.7.3-4.el8_7.3.x86_64


How reproducible:



2023-07-05T14:40:37Z DEBUG Logging to /var/log/ipaupgrade.log
2023-07-05T14:40:37Z DEBUG ipa-server-upgrade was invoked with arguments [] and options: {'verbose': False, 'quiet': False, 'log_file': None, 'force': False, 'skip_version_c
heck': False}
2023-07-05T14:40:37Z DEBUG IPA version 4.9.10-9.module+el8.7.0+17437+cf46f77f
2023-07-05T14:40:37Z DEBUG IPA platform rhel
2023-07-05T14:40:37Z DEBUG IPA os-release Red Hat Enterprise Linux 8.7 (Ootpa)
2023-07-05T14:40:37Z DEBUG importing all plugin modules in ipaserver.plugins...
2023-07-05T14:40:37Z DEBUG importing plugin module ipaserver.plugins.aci
2023-07-05T14:40:37Z DEBUG importing plugin module ipaserver.plugins.automember
2023-07-05T14:40:37Z DEBUG importing plugin module ipaserver.plugins.automount
2023-07-05T14:40:37Z DEBUG importing plugin module ipaserver.plugins.baseldap
2023-07-05T14:40:37Z DEBUG ipaserver.plugins.baseldap is not a valid plugin module
2023-07-05T14:40:37Z DEBUG importing plugin module ipaserver.plugins.baseuser
2023-07-05T14:40:37Z DEBUG importing plugin module ipaserver.plugins.batch
2023-07-05T14:40:37Z DEBUG importing plugin module ipaserver.plugins.ca
2023-07-05T14:40:37Z DEBUG importing plugin module ipaserver.plugins.caacl
2023-07-05T14:40:37Z DEBUG importing plugin module ipaserver.plugins.cert
2023-07-05T14:40:37Z DEBUG importing plugin module ipaserver.plugins.certmap
2023-07-05T14:40:37Z DEBUG importing plugin module ipaserver.plugins.certprofile
2023-07-05T14:40:37Z DEBUG importing plugin module ipaserver.plugins.config
2023-07-05T14:40:37Z DEBUG importing plugin module ipaserver.plugins.delegation
2023-07-05T14:40:37Z DEBUG importing plugin module ipaserver.plugins.dns
2023-07-05T14:40:37Z DEBUG importing plugin module ipaserver.plugins.dnsserver
2023-07-05T14:40:37Z DEBUG importing plugin module ipaserver.plugins.dogtag
2023-07-05T14:40:37Z DEBUG importing plugin module ipaserver.plugins.domainlevel
2023-07-05T14:40:37Z DEBUG importing plugin module ipaserver.plugins.group
2023-07-05T14:40:37Z DEBUG importing plugin module ipaserver.plugins.hbac
2023-07-05T14:40:37Z DEBUG ipaserver.plugins.hbac is not a valid plugin module
2023-07-05T14:40:37Z DEBUG importing plugin module ipaserver.plugins.hbacrule
2023-07-05T14:40:37Z DEBUG importing plugin module ipaserver.plugins.hbacsvc
2023-07-05T14:40:37Z DEBUG importing plugin module ipaserver.plugins.hbacsvcgroup
2023-07-05T14:40:37Z DEBUG importing plugin module ipaserver.plugins.hbactest


2023-07-05T14:40:38Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2023-07-05T14:40:38Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index'
2023-07-05T14:40:38Z DEBUG httpd is configured
2023-07-05T14:40:38Z DEBUG kadmin is configured
2023-07-05T14:40:38Z DEBUG dirsrv is configured
2023-07-05T14:40:38Z DEBUG pki-tomcatd is configured
2023-07-05T14:40:38Z DEBUG install is not configured
2023-07-05T14:40:38Z DEBUG krb5kdc is configured
2023-07-05T14:40:38Z DEBUG named is not configured
2023-07-05T14:40:38Z DEBUG filestore has files
2023-07-05T14:40:38Z DEBUG Searching for an interface of IP address: ::1
2023-07-05T14:40:38Z DEBUG Testing local IP address: ::1/128 (interface: lo)
2023-07-05T14:40:38Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state'
2023-07-05T14:40:38Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state'
2023-07-05T14:40:38Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state'
2023-07-05T14:40:38Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2023-07-05T14:40:38Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index'
2023-07-05T14:40:38Z DEBUG Starting external process
2023-07-05T14:40:38Z DEBUG args=['/bin/systemctl', 'is-active', 'dirsrv.service']
2023-07-05T14:40:38Z DEBUG Process finished, return code=3
2023-07-05T14:40:38Z DEBUG stdout=inactive

2023-07-05T14:40:38Z DEBUG stderr=
2023-07-05T14:40:38Z DEBUG Upgrading IPA:. Estimated time: 1 minute 30 seconds
2023-07-05T14:40:38Z DEBUG   [1/9]: saving configuration
2023-07-05T14:40:38Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2023-07-05T14:40:38Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2023-07-05T14:40:38Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2023-07-05T14:40:38Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2023-07-05T14:40:38Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2023-07-05T14:40:38Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2023-07-05T14:40:38Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2023-07-05T14:40:38Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2023-07-05T14:40:38Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2023-07-05T14:40:38Z DEBUG step duration: dirsrv __save_config 0.06 sec
2023-07-05T14:40:38Z DEBUG   [2/9]: disabling listeners
2023-07-05T14:40:38Z DEBUG step duration: dirsrv __disable_listeners 0.06 sec

2023-07-05T14:40:41Z DEBUG   [6/9]: updating schema
2023-07-05T14:40:41Z DEBUG retrieving schema for SchemaCache url=ldapi://%2Frun%2Fslapd-EXAMPLE.COM.socket conn=<ldap.ldapobject.SimpleLDAPObject object at 0x7fcfe2b0ca58>
2023-07-05T14:40:42Z DEBUG Processing schema LDIF file /usr/share/ipa/60kerberos.ldif
2023-07-05T14:40:42Z DEBUG Replace: ( 2.16.840.1.113719.1.301.4.10.1 NAME 'krbMaxRenewableAge'  EQUALITY integerMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.27 SINGLE-VALUE X-ORI
GIN ( 'IPA v4.9.10' 'user defined' ) )
2023-07-05T14:40:42Z DEBUG    with: ( 2.16.840.1.113719.1.301.4.10.1 NAME 'krbMaxRenewableAge' EQUALITY integerMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.27 SINGLE-VALUE X-ORIG
IN 'IPA v4.9.10' )
2023-07-05T14:40:42Z DEBUG Replace: ( 2.16.840.1.113719.1.301.4.6.1 NAME 'krbPrincipalExpiration'  EQUALITY generalizedTimeMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.24 SINGLE-
VALUE X-ORIGIN ( 'IPA v4.9.10' 'user defined' ) )
2023-07-05T14:40:42Z DEBUG    with: ( 2.16.840.1.113719.1.301.4.6.1 NAME 'krbPrincipalExpiration' EQUALITY generalizedTimeMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.24 SINGLE-V
ALUE X-ORIGIN 'IPA v4.9.10' )
2023-07-05T14:40:42Z DEBUG Replace: ( 2.16.840.1.113719.1.301.4.3.1 NAME 'krbPrincipalType'  EQUALITY integerMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.27 SINGLE-VALUE X-ORIGIN
 ( 'IPA v4.9.10' 'user defined' ) )
2023-07-05T14:40:42Z DEBUG    with: ( 2.16.840.1.113719.1.301.4.3.1 NAME 'krbPrincipalType' EQUALITY integerMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.27 SINGLE-VALUE X-ORIGIN 
'IPA v4.9.10' )
2023-07-05T14:40:42Z DEBUG Replace: ( 1.2.840.113554.1.4.1.6.5 NAME 'krbPwdAllowedKeysalts'  EQUALITY caseIgnoreIA5Match SYNTAX 1.3.6.1.4.1.1466.115.121.1.26 SINGLE-VALUE X-
ORIGIN ( 'IPA v4.9.10' 'user defined' ) )


2023-07-05T14:42:40Z DEBUG stderr=
2023-07-05T14:42:40Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index'
2023-07-05T14:42:40Z DEBUG Starting external process
2023-07-05T14:42:40Z DEBUG args=['/usr/bin/certutil', '-d', 'sql:/etc/pki/pki-tomcat/alias', '-L', '-f', '/etc/pki/pki-tomcat/alias/pwdfile.txt']
2023-07-05T14:42:41Z DEBUG Process finished, return code=0
2023-07-05T14:42:41Z DEBUG stdout=
Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

caSigningCert cert-pki-ca                                    CTu,Cu,Cu
ocspSigningCert cert-pki-ca                                  u,u,u
auditSigningCert cert-pki-ca                                 u,u,Pu
subsystemCert cert-pki-ca                                    u,u,u
Server-Cert cert-pki-ca                                      u,u,u



2023-07-05T14:42:41Z DEBUG stderr=
2023-07-05T14:42:41Z INFO Certmonger certificate renewal configuration already up-to-date
2023-07-05T14:42:41Z INFO [Enable PKIX certificate path discovery and validation]
2023-07-05T14:42:41Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state'
2023-07-05T14:42:41Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state'
2023-07-05T14:42:41Z DEBUG Saving StateFile to '/var/lib/ipa/sysupgrade/sysupgrade.state'
2023-07-05T14:42:41Z INFO [Authorizing RA Agent to modify profiles]
2023-07-05T14:42:41Z INFO [Authorizing RA Agent to manage lightweight CAs]
2023-07-05T14:42:41Z INFO [Ensuring Lightweight CAs container exists in Dogtag database]
2023-07-05T14:42:41Z INFO [Adding default OCSP URI configuration]
2023-07-05T14:42:41Z INFO [Disabling cert publishing]
2023-07-05T14:42:41Z INFO pki-tomcat configuration changed, restart pki-tomcat
2023-07-05T14:42:41Z DEBUG Starting external process
2023-07-05T14:42:41Z DEBUG args=['/bin/systemctl', 'restart', 'pki-tomcatd']
2023-07-05T14:43:02Z DEBUG Process finished, return code=0
2023-07-05T14:43:02Z DEBUG stdout=
2023-07-05T14:43:02Z DEBUG stderr=
2023-07-05T14:43:02Z DEBUG Starting external process
2023-07-05T14:43:02Z DEBUG args=['/bin/systemctl', 'is-active', 'pki-tomcatd']
2023-07-05T14:43:03Z DEBUG Process finished, return code=0
2023-07-05T14:43:03Z DEBUG stdout=active



2023-07-05T14:43:03Z DEBUG stderr=
2023-07-05T14:43:03Z DEBUG wait_for_open_ports: localhost [8080, 8443] timeout 120
2023-07-05T14:43:03Z DEBUG waiting for port: 8080
2023-07-05T14:43:03Z DEBUG SUCCESS: port: 8080
2023-07-05T14:43:03Z DEBUG waiting for port: 8443
2023-07-05T14:43:03Z DEBUG SUCCESS: port: 8443
2023-07-05T14:43:03Z DEBUG Restart of pki-tomcatd complete
2023-07-05T14:43:03Z INFO [Ensuring CA is using LDAPProfileSubsystem]
2023-07-05T14:43:03Z INFO [Migrating certificate profiles to LDAP]
2023-07-05T14:43:03Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2023-07-05T14:43:03Z DEBUG   File "/usr/lib/python3.6/site-packages/ipapython/admintool.py", line 180, in execute
    return_value = self.run()
  File "/usr/lib/python3.6/site-packages/ipaserver/install/ipa_server_upgrade.py", line 54, in run
    server.upgrade()
  File "/usr/lib/python3.6/site-packages/ipaserver/install/server/upgrade.py", line 2055, in upgrade
    upgrade_configuration()
  File "/usr/lib/python3.6/site-packages/ipaserver/install/server/upgrade.py", line 1908, in upgrade_configuration
    ca_enable_ldap_profile_subsystem(ca)
  File "/usr/lib/python3.6/site-packages/ipaserver/install/server/upgrade.py", line 458, in ca_enable_ldap_profile_subsystem
    cainstance.migrate_profiles_to_ldap()
  File "/usr/lib/python3.6/site-packages/ipaserver/install/cainstance.py", line 2069, in migrate_profiles_to_ldap
    profile_states = _get_ldap_profile_states()
  File "/usr/lib/python3.6/site-packages/ipaserver/install/cainstance.py", line 2136, in _get_ldap_profile_states
    attrs_list=["cn", "certProfileConfig"]
  File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line 1454, in get_entries
    **kwargs)
  File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line 1595, in find_entries
    raise errors.EmptyResult(reason='no matching entry found')

2023-07-05T14:43:03Z DEBUG The ipa-server-upgrade command failed, exception: EmptyResult: no matching entry found
2023-07-05T14:43:03Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details:
EmptyResult: no matching entry found


Actual results:


Expected results:


Additional info:

Comment 5 Rob Crittenden 2023-07-25 18:56:31 UTC
I guess it's completely unexpected that all certificate profiles would be missing.

If they are adventurous and not afraid of touching the python code (this only affects upgrades and installs) they can try this change which should allow all the profiles to be imported from disk again by the ipa-server-upgrade.

diff --git a/ipaserver/install/cainstance.py b/ipaserver/install/cainstance.py
index 38693c954..d37348d13 100644
--- a/ipaserver/install/cainstance.py
+++ b/ipaserver/install/cainstance.py
@@ -2129,12 +2129,15 @@ def _get_ldap_profile_states():
     :return: mapping of lowercase profile id to state enum member
     """
     conn = api.Backend.ldap2
-    entries = conn.get_entries(
-        base_dn=PROFILES_DN,
-        scope=conn.SCOPE_SUBTREE,
-        filter="(objectClass=certProfile)",
-        attrs_list=["cn", "certProfileConfig"]
-    )
+    try:
+        entries = conn.get_entries(
+            base_dn=PROFILES_DN,
+            scope=conn.SCOPE_SUBTREE,
+            filter="(objectClass=certProfile)",
+            attrs_list=["cn", "certProfileConfig"]
+        )
+    except errors.EmptyResult:
+        entries = tuple()
     results = {}
     for entry in entries:
         single = entry.single_value

To apply the patch, copy and paste the above as-is, spacing is important and paste it into a file on the server, say /tmp/profile.patch

Install the patch package: dnf -y install patch

Apply the patch:
# cd /usr/lib/python3.6/site-packages/
# patch -p1 < /tmp/profile.patch 

The only output should be "patching file ipaserver/install/cainstance.py". Anything else represents a failure.

Run the upgrade again:
# ipa-server-upgrade

The upgrade should succeed.

If they want to restore the previous code (so rpm -V doesn't complain): dnf reinstall python3-ipaserver

This should be a one-off operation for them as all profiles missing post-install is not expected.