Bug 1876514

Summary: High CPU utilization by the sssd_kcm process
Product: Red Hat Enterprise Linux 8 Reporter: Thorsten Scherf <tscherf>
Component: sssdAssignee: Pavel Březina <pbrezina>
Status: CLOSED ERRATA QA Contact: Anuj Borah <aborah>
Severity: high Docs Contact:
Priority: high    
Version: 8.2CC: andrew_desiervo, atikhono, chorn, dlavu, grajaiya, haidarvm, jhrozek, lslebodn, mcasabur, mzidek, pbrezina, rharwood, sgoveas, sigbjorn.lie, swachira, tscherf
Target Milestone: rcKeywords: Triaged
Target Release: 8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: review, sync-to-jira
Fixed In Version: sssd-2.4.0-4.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-18 15:03:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1894575    
Attachments:
Description Flags
test results none

Description Thorsten Scherf 2020-09-07 11:24:34 UTC
Description of problem:
This BZ is about tracking sssd-kcm performance related tasks.

Version-Release number of selected component (if applicable):
sssd-kcm-2.2.3-20.el8.x86_64

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Alexey Tikhonov 2020-09-07 11:32:19 UTC
Note: presumably this might be similar to BZ 1645624 but this yet to be verified.

Comment 3 Alexey Tikhonov 2020-09-07 17:17:58 UTC
Might also be duplicate of bz 1867899

Comment 5 Pavel Březina 2020-09-17 13:07:03 UTC
This is probably duplicate of Fedora BZ https://bugzilla.redhat.com/show_bug.cgi?id=1645624 which is currently under heavy investigation.

Comment 6 Alexey Tikhonov 2020-09-17 14:42:32 UTC
*** Bug 1879978 has been marked as a duplicate of this bug. ***

Comment 7 Alexey Tikhonov 2020-09-17 14:44:45 UTC
(In reply to Pavel Březina from comment #5)
> This is probably duplicate of Fedora BZ
> https://bugzilla.redhat.com/show_bug.cgi?id=1645624 which is currently under
> heavy investigation.

I'm not sure those are the same. See bz 1879978 - it talks about big number of tickets (50-60+). I hope we don't have anything O(N^2) in kcm code...

Comment 8 Pavel Březina 2020-09-23 10:45:56 UTC
No. I just added https://bugzilla.redhat.com/show_bug.cgi?id=1645624#c60 for explanation. It is indeed bound to a large number of tickets but it may not be necessarily visible at the first sight as not all information stored in ccache is shown by default and you need -C switch (klist -A -C).

Thorsten, is it possible to get output of 'klist -A -C' from the affected user and machine?

Comment 13 Pavel Březina 2020-10-05 12:56:09 UTC
Thank you. This is definitely a duplicate of Fedora https://bugzilla.redhat.com/show_bug.cgi?id=1645624.

Lets keep this BZ to track this in RHEL.

Comment 14 Pavel Březina 2020-10-05 12:58:56 UTC
Upstream ticket:
https://github.com/SSSD/sssd/issues/5349

Comment 16 Pavel Březina 2020-10-23 15:28:32 UTC
Upstream PR:
https://github.com/SSSD/sssd/pull/5375

Comment 17 Alexey Tikhonov 2020-10-29 19:51:26 UTC
*** Bug 1867899 has been marked as a duplicate of this bug. ***

Comment 19 Alexey Tikhonov 2020-11-02 15:11:17 UTC
*** Bug 1867899 has been marked as a duplicate of this bug. ***

Comment 20 Pavel Březina 2020-12-04 10:44:50 UTC
Pushed PR: https://github.com/SSSD/sssd/pull/5375

* `master`
    * 325de5a5bb97ba026be6d22492bea8ab2605f1b5 - secrets: remove base64 enctype
    * 39277cdadd317b0ab86cdd37de0616bc3eecbe6a - secrets: move attrs names to macros
    * 9c1b51d057390fb5b26151f814a480911cda4cc9 - secrets: default to "plaintext" if "enctype" attr is missing
    * bf127d4f3f42e5b2afe25e512211439bc12a9904 - secrets: fix may_payload_size exceeded debug message
    * c3b314db57c34f64aaca7d74e76a9a955288bb51 - kcm: store credentials list in hash table to avoid cache lookups
    * a370553c90c2ed6df3b94c169c4960a6f978031f - sss_ptr_hash: fix double free for circular dependencies
    * 241ee30da12f564803793ee2b14c1522aabd9235 - kcm: add per-connection data to be shared between requests
    * 194447d35c11eb914f54719491dc5cfaab01b9a1 - kcm: use binary format to store ccache instead of json
    * f17740d831e16449495fff4ec57cc4800aaac83d - kcm: add spaces around operators in kcmsrv_ccache_key.c
    * 15069a647ed6c7f1ead42baa1d421d953c9bc557 - kcm: avoid suppression of cppcheck warning
    * e63a15038ac9c186626e4fdf681a6492031d1e40 - kcm: move sec key parser to separate file so it can be shared
    * 9b1631defdcaa3ea7e87889eb136e7fa935ab4ce - kcm: add json suffix to existing searialization functions
    * b6cc661b9f4162e590137430e945aa321fc13121 - iobuf: add more iobuf functions
    * ed08ba0023e63024bf1c52ae3f6596b9d804d0a5 - secrets: accept binary data instead of string
    * 908c15af9a9f8f0556a588e368e4a0b2e24ace1b - secrets: allow to specify secret's data format
    * 74fdaa64b27e88a6e0f153f8cb59989c572d4294 - kcm: avoid multiple debug messages if sss_sec_put fails
    * b8f28d9aa9d862cf504691c9c3f92941a63fb0a4 - kcm: disable encryption
    * 8edcea8c377e85d037e83065c1904fa4b92c4a39 - kcm: avoid name confusion in GET_CRED_UUID_LIST handlers
    * 47a316c850107f12d406f27abb216e26383dfab7 - kcm: fix typos in debug messages

Comment 22 Pavel Březina 2020-12-10 11:57:29 UTC
Steps to reproduce:
1. Login as Kerberized SSSD user
2. Use gssapi authentication to obtain large number of service tickets
3. The previous step will take waaay longer time without patch

You can create service tickets with 'ipa service-add' and then fetch their keytab. To perform gssapi authentication, you can use a test app as described here: https://github.com/SSSD/sssd/pull/5375#issue-509041627

Or maybe a more simple way

1. Login as Kerberized SSSD user
2. Use kvno to obtain large number of service tickets (200+)
3. Run klist to list the credentials
4. The previous step should take waaay longer time without patch

Comment 23 Pavel Březina 2020-12-11 09:31:56 UTC
Additional patch.

* `master`
    * 18b98836ef8e337992f0ecb239a32b9c3cedb750 - kcm: decode base64 encoded secret on upgrade path

Comment 26 Anuj Borah 2020-12-23 10:34:55 UTC
Created attachment 1741516 [details]
test results

Comment 32 Pavel Březina 2021-01-06 12:14:19 UTC
Ok, klist was just a suggestion to make the verification easier but it is obviously not enough to measure the performance difference. So gssapi should be used instead.

The most reliable way would be to use the steps from upstream PR [1], which obtains each service ticket using gssapi and the performance difference is huge there. The test application can be rewritten to python if you'd like to automate it in multihost tests.

Another possible solution (I did not test it) might be to obtain the tickets with kvno and then try to "ssh localhost" which should trigger gssapi authentication attempt (unless disabled) and probably will create a measurable difference.

[1] https://github.com/SSSD/sssd/pull/5375#issue-509041627

Comment 39 errata-xmlrpc 2021-05-18 15:03:57 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (sssd bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1666