+++ This bug was initially created as a clone of Bug #1894781 +++
Description of problem:
A PKI KRA subsystem exist but the agent LDAP entry does not
uid=ipakra,ou=people,o=kra,o=ipaca
fix_kra_people_entry does test the PKI KRA subsystem existence, but does not test if KRA_AGENT_DN exists, because the default expectation is to always have a KRA agent for a fully functioning PKI KRA subsystem.
this is a corner case to have a KRA and not a KRA agent entry in the LDAP backend, but it seems to happen in the real world (cause unknown).
the problem is the upgrade script fails with an exception, preventing an ipactl restart that is always called into the upgrade script.
a manual restart of the services till worls as workaround.
adding a validation test in
./freeipa/ipaserver/install/plugins/fix_kra_people_entry.py
fix_kra_people_entry
for this LDAP lookup
entry = self.api.Backend.ldap2.get_entry(krainstance.KRA_AGENT_DN)
# check description attribute
description_values = entry.get('description', [])
could prevent an exception and provide a smoother error handling of the failing IPA update process, and subsequent failing ipactl restart.
( is there a way to not have the "Automatically running upgrade: in ipactl start or restart ? may be with --skip-version-check )
Version-Release number of selected component (if applicable):
RHEL IdM update from RHEL-7.8 to RHEL-7.9
How reproducible:
not tested, but likely on demand
, have to manually remove the
Steps to Reproduce:
1. RHEL-7.8 IdM replica: manually remove the KRA agent LDAP entry with a ldapmodify
2. update the system to RHEL-7.9
3. IPA restart fail
Actual results:
/var/log/dirsrv/slapd-xx/access
[03/Nov/2020:10:27:14.264587951 -0700] conn=3 op=9 SRCH base="uid=ipakra,ou=people,o=kra,o=ipaca" scope=0 filter="(objectClass=*)" attrs=ALL
[03/Nov/2020:10:27:14.264683425 -0700] conn=3 op=9 RESULT err=32 tag=101 nentries=0 etime=0.000190580
ipaupgrade.log
2020-11-03T17:27:14Z DEBUG AD Trusts are not enabled on this server
2020-11-03T17:27:14Z DEBUG Executing upgrade plugin: fix_kra_people_entry
2020-11-03T17:27:14Z DEBUG raw: fix_kra_people_entry
2020-11-03T17:27:14Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2020-11-03T17:27:14Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index'
2020-11-03T17:27:14Z DEBUG Destroyed connection context.ldap2_140376115621328
2020-11-03T17:27:14Z ERROR Upgrade failed with no such entry
2020-11-03T17:27:14Z DEBUG Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py", line 274, in __upgrade
self.modified = (ld.update(self.files) or self.modified)
File "/usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py", line 966, in update
self._run_updates(all_updates)
File "/usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py", line 928, in _run_updates
self._run_update_plugin(update['plugin'])
File "/usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py", line 903, in _run_update_plugin
restart_ds, updates = self.api.Updater[plugin_name]()
File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 1474, in __call__
return self.execute(**options)
File "/usr/lib/python2.7/site-packages/ipaserver/install/plugins/fix_kra_people_entry.py", line 35, in execute
entry = self.api.Backend.ldap2.get_entry(krainstance.KRA_AGENT_DN)
File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1565, in get_entry
size_limit=size_limit, get_effective_rights=get_effective_rights,
File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1377, in get_entries
**kwargs)
File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1515, in find_entries
break
File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1026, in error_handler
raise errors.NotFound(reason=arg_desc or 'no such entry')
NotFound: no such entry
2020-11-03T17:27:14Z DEBUG Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 567, in start_creation
run_step(full_msg, method)
File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 557, in run_step
method()
File "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py", line 282, in __upgrade
raise RuntimeError(e)
RuntimeError: no such entry
2020-11-03T17:27:14Z DEBUG [error] RuntimeError: no such entry
2020-11-03T17:27:14Z DEBUG [cleanup]: stopping directory server
2020-11-03T17:27:14Z DEBUG Destroyed connection context.ldap2_140376143874960
2020-11-03T17:27:14Z DEBUG Starting external process
2020-11-03T17:27:14Z DEBUG args=/bin/systemctl stop dirsrv
2020-11-03T17:27:16Z DEBUG Process finished, return code=0
2020-11-03T17:27:16Z DEBUG stdout=
2020-11-03T17:27:16Z DEBUG stderr=
2020-11-03T17:27:16Z DEBUG Stop of dirsrv complete
2020-11-03T17:27:16Z DEBUG duration: 2 seconds
2020-11-03T17:27:16Z DEBUG [cleanup]: restoring configuration
...snip...
2020-11-03T17:27:16Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2020-11-03T17:27:16Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute
return_value = self.run()
File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 56, in run
raise admintool.ScriptError(str(e))
2020-11-03T17:27:16Z DEBUG The ipa-server-upgrade command failed, exception: ScriptError: ('IPA upgrade failed.', 1)
2020-11-03T17:27:16Z ERROR ('IPA upgrade failed.', 1)
2020-11-03T17:27:16Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
(END)
and then:
root $ ipactl restart
IPA version error: data needs to be upgraded (expected version '4.6.8-5.el7', current version '4.6.6-11.el7')
Automatically running upgrade, for details see /var/log/ipaupgrade.log
Be patient, this may take a few minutes.
Automatic upgrade failed: Upgrade failed with no such entry
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
('IPA upgrade failed.', 1)
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
See the upgrade log for more details and/or run /usr/sbin/ipa-server-upgrade again
Stopping ipa-dnskeysyncd Service
Stopping ipa-otpd Service
Stopping pki-tomcatd Service
Stopping ipa-custodia Service
Stopping httpd Service
Stopping named Service
Stopping kadmin Service
Stopping krb5kdc Service
Stopping Directory Service
Aborting ipactl
Expected results:
yes
Additional info:
current code:
./freeipa/ipaserver/install/plugins/fix_kra_people_entry.py
...
def execute(self, **options):
kra = krainstance.KRAInstance(self.api.env.realm)
if not kra.is_installed():
return False, []
cert = x509.load_certificate_from_file(paths.RA_AGENT_PEM)
entry = self.api.Backend.ldap2.get_entry(krainstance.KRA_AGENT_DN)
# check description attribute
description_values = entry.get('description', [])
this fails if the entry variable cannot be initialized with the LDAP lookup of the KRA agent user entry uid=ipakra,ou=people,o=kra,o=ipaca when it does not exist ( LDAP err=32)
--- Additional comment from Florence Blanc-Renaud on 2020-11-06 10:41:12 UTC ---
See also https://bugzilla.redhat.com/show_bug.cgi?id=1895197
This issue is slightly different as it could happen if the customer deletes the uid=ipakra,ou=people,o=kra,o=ipaca entry by mistake. When the entry is missing, the KRA functionality is broken (ipa vault-* commands don't work anymore because the IPA framework cannot authenticate to the KRA service).
So IMO it is ok for ipa-server-upgrade to exit on error in order to signal that something abnormal was detected.
Moreover, it may get tricky to try to generate a new entry based on the content of the local file /var/lib/ipa/ra-agent.pem because the local file may be different on replicas if renewal happened on one of them but was not done on the others. A manual operation would be preferred as the user would have to check which ra-agent.pem is the most recent and select this one as the source to create the LDAP entry.
--- Additional comment from Florence Blanc-Renaud on 2020-11-12 20:11:13 UTC ---
During the team triage, we decided to implement a ipa-healthcheck check to ensure that the entry uid=ipakra,ou=people,o=kra,o=ipaca exists and its content is consistent with the RA cert stored in /var/lib/ipa/ra-agent.pem.
Hence moving to RHEL 8, ipa-healthcheck component.
--- Additional comment from Steven Mercurio on 2021-01-31 09:24:33 UTC ---
Is there a guide or help DOC to try and fix this issue on a 7.8 to 7.9 upgrade?
Maybe a back end process at every install, upgrade, backup, etc that backs up all system required entries like KRA and a script that restores or regenerates them.
Question I have though is Why allow this entry to be deleted in the first place or have in the shut script a check for backup of these critical entries and a auto-restore at start?
--- Additional comment from Rob Crittenden on 2021-01-31 17:06:45 UTC ---
To re-create the entry, assuming as Flo pointed out that the certificates are in sync, convert the RA Agent PEM file into DER:
# openssl x509 -in /var/lib/ipa/ra-agent.pem -out /tmp/ra-agent.der -outform der
Obtain the current description from the RA agent:
# ldapsearch -LLL -o ldif-wrap=no -x -D 'cn=directory manager' -W -b uid=ipara,ou=people,o=ipaca description
Create the entry in LDAP
# ldapmodify -x -D 'cn=Directory Manager' -W
LDAP Password:
dn: uid=ipakra,ou=people,o=kra,o=ipaca
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: cmsuser
uid: ipakra
sn: IPA KRA User
cn: IPA KRA User
usertype: undefined
userCertificate:< file:///tmp/ra-agent.der
description: [ the value of description from above ]
<hit enter>
^D
You can remove the temporary file afterwards (/tmp/ra-agent.der).
IPA provides a management CLI and UI to manage entries (safe) but leaves flexibility if users want to manipulate entries directly in LDAP (risky). The root cause of why the entry was removed is unknown, whether user error or a coding error in either IPA or CS.
Note that while this represents as an error during upgrade it also means the KRA is effectively not working in the pre-upgraded state. The upgrade process merely makes this painfully visible.
--- Additional comment from Steven Mercurio on 2021-02-14 02:46:02 UTC ---
I get this error:
ldapmodify: modify operation type is missing at line 2, entry "uid=ipakra,ou=people,o=kra,o=ipaca"
[root@idm01 ~]# ldapsearch -LLL -o ldif-wrap=no -x -D 'cn=directory manager' -W -b uid=ipara,ou=people,o=ipaca description
Enter LDAP Password:
dn: uid=ipara,ou=people,o=ipaca
description: 2;26;CN=Certificate Authority,O=HOMELINUX.COM;CN=IPA RA,O=HOMELINUX.COM
[root@idm01 ~]# ldapmodify -x -D 'cn=Directory Manager' -W
Enter LDAP Password:
dn: uid=ipakra,ou=people,o=kra,o=ipaca
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: cmsuser
uid: ipakra
sn: IPA KRA User
cn: IPA KRA User
usertype: undefined
userCertificate:< file:///tmp/ra-agent.der
description: 2;26;CN=Certificate Authority,O=HOMELINUX.COM;CN=IPA RA,O=HOMELINUX.COM
ldapmodify: modify operation type is missing at line 2, entry "uid=ipakra,ou=people,o=kra,o=ipaca"
[root@idm01 ~]#
--- Additional comment from Steven Mercurio on 2021-02-14 02:48:42 UTC ---
As this seems to be fixable (assuming I get past the error I have) is there a way to just prompt the user for Y/N to fix this or re-create the entry for them? Another option would be a script to rebuild this entry and just add in the error that the user should run thwe script.
--- Additional comment from Steven Mercurio on 2021-02-14 02:54:30 UTC ---
Got it to work using ldapadd:
[root@idm01 ~]# ldapadd -x -D 'cn=Directory Manager' -W
Enter LDAP Password:
dn: uid=ipakra,ou=people,o=kra,o=ipaca
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: cmsuser
uid: ipakra
sn: IPA KRA User
cn: IPA KRA User
usertype: undefined
userCertificate:< file:///tmp/ra-agent.der
description: 2;26;CN=Certificate Authority,O=HOMELINUX.COM;CN=IPA RA,O=HOMELINUX.COM
adding new entry "uid=ipakra,ou=people,o=kra,o=ipaca"
ldap_add: No such object (32)
matched DN: o=ipaca
[root@idm01 ~]#
So is this something the upgrade script can do/fix or let someone know to run another provided script to fix then run the upgrade again?
--- Additional comment from Steven Mercurio on 2021-02-14 03:00:24 UTC ---
Actually didn't work but thought it did. Not surehow to fix this.
--- Additional comment from Steven Mercurio on 2021-02-14 03:04:23 UTC ---
One thouhgt.... What if when I installed IDM pre 4.5 I never installed KRA so essentially it never existed?
[root@idm02 ~]# ipa --version
VERSION: 4.5.4, API_VERSION: 2.228
[root@idm02 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.5 (Maipo)
[root@idm02 ~]#
--- Additional comment from Rob Crittenden on 2021-02-14 04:25:06 UTC ---
After the dn add:
changetype: modify
This seems very much like a corner case. I'm not sure we'll provide automation to address it. A KCS is probably sufficient.
--- Additional comment from Antonio Torres on 2021-02-23 15:34:33 UTC ---
PR in ipa-healthcheck: https://github.com/freeipa/freeipa-healthcheck/pull/188
--- Additional comment from Rob Crittenden on 2021-02-25 20:49:48 UTC ---
merged upstream:
3f6ed4393dfa9ddf982e326065a3ea160bef90b6
a6504bd7d32fe3553b9f6f807f3d84a1b87bb77c