Bug 1876514
Summary: | High CPU utilization by the sssd_kcm process | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 8 | Reporter: | Thorsten Scherf <tscherf> | ||||
Component: | sssd | Assignee: | Pavel Březina <pbrezina> | ||||
Status: | CLOSED ERRATA | QA Contact: | Anuj Borah <aborah> | ||||
Severity: | high | Docs Contact: | |||||
Priority: | high | ||||||
Version: | 8.2 | CC: | andrew_desiervo, atikhono, chorn, dlavu, grajaiya, haidarvm, jhrozek, lslebodn, mcasabur, mzidek, pbrezina, rharwood, sgoveas, sigbjorn.lie, swachira, tscherf | ||||
Target Milestone: | rc | Keywords: | Triaged | ||||
Target Release: | 8.0 | ||||||
Hardware: | Unspecified | ||||||
OS: | Unspecified | ||||||
Whiteboard: | review, sync-to-jira | ||||||
Fixed In Version: | sssd-2.4.0-4.el8 | Doc Type: | If docs needed, set a value | ||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | Environment: | ||||||
Last Closed: | 2021-05-18 15:03:57 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Bug Depends On: | |||||||
Bug Blocks: | 1894575 | ||||||
Attachments: |
|
Description
Thorsten Scherf
2020-09-07 11:24:34 UTC
Note: presumably this might be similar to BZ 1645624 but this yet to be verified. Might also be duplicate of bz 1867899 This is probably duplicate of Fedora BZ https://bugzilla.redhat.com/show_bug.cgi?id=1645624 which is currently under heavy investigation. *** Bug 1879978 has been marked as a duplicate of this bug. *** (In reply to Pavel Březina from comment #5) > This is probably duplicate of Fedora BZ > https://bugzilla.redhat.com/show_bug.cgi?id=1645624 which is currently under > heavy investigation. I'm not sure those are the same. See bz 1879978 - it talks about big number of tickets (50-60+). I hope we don't have anything O(N^2) in kcm code... No. I just added https://bugzilla.redhat.com/show_bug.cgi?id=1645624#c60 for explanation. It is indeed bound to a large number of tickets but it may not be necessarily visible at the first sight as not all information stored in ccache is shown by default and you need -C switch (klist -A -C). Thorsten, is it possible to get output of 'klist -A -C' from the affected user and machine? Thank you. This is definitely a duplicate of Fedora https://bugzilla.redhat.com/show_bug.cgi?id=1645624. Lets keep this BZ to track this in RHEL. Upstream ticket: https://github.com/SSSD/sssd/issues/5349 Upstream PR: https://github.com/SSSD/sssd/pull/5375 *** Bug 1867899 has been marked as a duplicate of this bug. *** *** Bug 1867899 has been marked as a duplicate of this bug. *** Pushed PR: https://github.com/SSSD/sssd/pull/5375 * `master` * 325de5a5bb97ba026be6d22492bea8ab2605f1b5 - secrets: remove base64 enctype * 39277cdadd317b0ab86cdd37de0616bc3eecbe6a - secrets: move attrs names to macros * 9c1b51d057390fb5b26151f814a480911cda4cc9 - secrets: default to "plaintext" if "enctype" attr is missing * bf127d4f3f42e5b2afe25e512211439bc12a9904 - secrets: fix may_payload_size exceeded debug message * c3b314db57c34f64aaca7d74e76a9a955288bb51 - kcm: store credentials list in hash table to avoid cache lookups * a370553c90c2ed6df3b94c169c4960a6f978031f - sss_ptr_hash: fix double free for circular dependencies * 241ee30da12f564803793ee2b14c1522aabd9235 - kcm: add per-connection data to be shared between requests * 194447d35c11eb914f54719491dc5cfaab01b9a1 - kcm: use binary format to store ccache instead of json * f17740d831e16449495fff4ec57cc4800aaac83d - kcm: add spaces around operators in kcmsrv_ccache_key.c * 15069a647ed6c7f1ead42baa1d421d953c9bc557 - kcm: avoid suppression of cppcheck warning * e63a15038ac9c186626e4fdf681a6492031d1e40 - kcm: move sec key parser to separate file so it can be shared * 9b1631defdcaa3ea7e87889eb136e7fa935ab4ce - kcm: add json suffix to existing searialization functions * b6cc661b9f4162e590137430e945aa321fc13121 - iobuf: add more iobuf functions * ed08ba0023e63024bf1c52ae3f6596b9d804d0a5 - secrets: accept binary data instead of string * 908c15af9a9f8f0556a588e368e4a0b2e24ace1b - secrets: allow to specify secret's data format * 74fdaa64b27e88a6e0f153f8cb59989c572d4294 - kcm: avoid multiple debug messages if sss_sec_put fails * b8f28d9aa9d862cf504691c9c3f92941a63fb0a4 - kcm: disable encryption * 8edcea8c377e85d037e83065c1904fa4b92c4a39 - kcm: avoid name confusion in GET_CRED_UUID_LIST handlers * 47a316c850107f12d406f27abb216e26383dfab7 - kcm: fix typos in debug messages Steps to reproduce: 1. Login as Kerberized SSSD user 2. Use gssapi authentication to obtain large number of service tickets 3. The previous step will take waaay longer time without patch You can create service tickets with 'ipa service-add' and then fetch their keytab. To perform gssapi authentication, you can use a test app as described here: https://github.com/SSSD/sssd/pull/5375#issue-509041627 Or maybe a more simple way 1. Login as Kerberized SSSD user 2. Use kvno to obtain large number of service tickets (200+) 3. Run klist to list the credentials 4. The previous step should take waaay longer time without patch Additional patch. * `master` * 18b98836ef8e337992f0ecb239a32b9c3cedb750 - kcm: decode base64 encoded secret on upgrade path Created attachment 1741516 [details]
test results
Ok, klist was just a suggestion to make the verification easier but it is obviously not enough to measure the performance difference. So gssapi should be used instead. The most reliable way would be to use the steps from upstream PR [1], which obtains each service ticket using gssapi and the performance difference is huge there. The test application can be rewritten to python if you'd like to automate it in multihost tests. Another possible solution (I did not test it) might be to obtain the tickets with kvno and then try to "ssh localhost" which should trigger gssapi authentication attempt (unless disabled) and probably will create a measurable difference. [1] https://github.com/SSSD/sssd/pull/5375#issue-509041627 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (sssd bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:1666 |