Bug 2227832

Summary: Interrupt request processing in ipadb_fill_info3() if connection to 389ds is lost [rhel-7]
Product: Red Hat Enterprise Linux 7 Reporter: Julien Rische <jrische>
Component: ipaAssignee: Julien Rische <jrische>
Status: CLOSED NOTABUG QA Contact: Michal Polovka <mpolovka>
Severity: high Docs Contact:
Priority: high    
Version: 7.9CC: abokovoy, bgollahe, cheimes, frenaud, gkaihoro, ipa-qe, jrische, jverreng, mjurasek, mpolovka, pkulkarn, rakkumar, rcritten, tmihinto, tscherf
Target Milestone: rcKeywords: Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 2214638 Environment:
Last Closed: 2023-08-02 12:58:35 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2214638    
Bug Blocks: 2227831    

Description Julien Rische 2023-07-31 15:47:27 UTC
+++ This bug was initially created as a clone of Bug #2214638 +++

Description of problem:
krb5kdc is crashing intermittently on the IPA server even after updating the packages to latest version.

Version-Release number of selected component (if applicable):
ipa-server-4.9.11-5.module+el8.8.0+18146+a1d8660b.x86_64
krb5-server-1.18.2-22.el8_7.x86_64


How reproducible:
Intermittently

--- Additional comment from Rob Crittenden on 2023-06-13 17:53:31 UTC ---

The crash is happening in the ipa kdb code. Changing components.

The ld connection is NULL, hence the failure.

Did this fail in a particular context? e.g. under load, a specific Kerberos request?

--- Additional comment from Rob Crittenden on 2023-06-13 19:43:10 UTC ---

Alexander thinks this may be upstream issue https://pagure.io/freeipa/issue/9251 . We lost traction with an upstream user who had a reproducing system.

--- Additional comment from Têko Mihinto on 2023-07-03 17:33:35 UTC ---

Hi Rob,

(In reply to Rob Crittenden from comment #1)
> The crash is happening in the ipa kdb code. Changing components.
> 
> The ld connection is NULL, hence the failure.
> 
> Did this fail in a particular context? e.g. under load, a specific Kerberos
> request?

Customer mentioned that this typically happens when the LDAP server is under load
( addition / deletion of new hosts triggering multiple and replicated changes ).

Regards,
Têko.

--- Additional comment from Julien Rische on 2023-07-26 10:03:25 UTC ---

This crash is due to the is_master_host() function[1], which is called in ipadb_fill_info3()[2] as part of the process to populate the PAC.

is_master_host() searches for a master host entry using ipadb_simple_search()[3], and interprets an error as the absence of master entry, and the absence of error as a the presence of a master entry. Hence, a connection error would be misinterpreted.

The crash occurs because in case ipadb_simple_search() faces a connection issue during the query, it will try to reset the connection to run the query again. But if it fails to reset the connection, it returns with an error code right away, leaving the LDAP context uninitialized. But, as mentioned above, is_master_host() won't handle this as an actual error. So the process in ipadb_fill_info3() will continue until the ldap_get_entry_controls() call, where the process crashes because of the uninitialized LDAP context.

[1] https://github.com/freeipa/freeipa/blob/release-4-9-12/daemons/ipa-kdb/ipa_kdb_mspac.c#L404
[2] https://github.com/freeipa/freeipa/blob/release-4-9-12/daemons/ipa-kdb/ipa_kdb_mspac.c#L695-L697
[3] https://github.com/freeipa/freeipa/blob/release-4-9-12/daemons/ipa-kdb/ipa_kdb_common.c#L169

--- Additional comment from Julien Rische on 2023-07-31 11:43:36 UTC ---

Upstream ticket:
https://pagure.io/freeipa/issue/9422

--- Additional comment from Julien Rische on 2023-07-31 11:48:31 UTC ---

Upstream pull request:
https://github.com/freeipa/freeipa/pull/6929

Comment 2 Florence Blanc-Renaud 2023-08-01 11:36:54 UTC
Fixed upstream
master:
https://pagure.io/freeipa/c/c84c59c66f1b22ebc671960cae90088a024d2d62

Comment 4 Julien Rische 2023-08-02 12:58:35 UTC
Function is_master_host() is not handling errors properly in IPA 4.6.8[1], just as it is in 4.9 an 4.10. However in 4.6.8's case, the ipadb_fill_info3() function fails immediately in case is_master_host() returns false[2][3]. This should keep the NULL LDAP context issue from happening.

For this reason, we won't fix is_master_host() on RHEL7.

[1] https://github.com/freeipa/freeipa/blob/release-4-6-8/daemons/ipa-kdb/ipa_kdb_mspac.c#L361-L382
[2] https://github.com/freeipa/freeipa/blob/release-4-6-8/daemons/ipa-kdb/ipa_kdb_mspac.c#L438-L440
[3] https://github.com/freeipa/freeipa/blob/release-4-6-8/daemons/ipa-kdb/ipa_kdb_mspac.c#L487-L489