RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1876514 - High CPU utilization by the sssd_kcm process
Summary: High CPU utilization by the sssd_kcm process
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: sssd
Version: 8.2
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 8.0
Assignee: Pavel Březina
QA Contact: Anuj Borah
URL:
Whiteboard: review, sync-to-jira
: 1867899 1879978 (view as bug list)
Depends On:
Blocks: 1894575
TreeView+ depends on / blocked
 
Reported: 2020-09-07 11:24 UTC by Thorsten Scherf
Modified: 2023-12-15 19:12 UTC (History)
16 users (show)

Fixed In Version: sssd-2.4.0-4.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-05-18 15:03:57 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
test results (239.54 KB, text/plain)
2020-12-23 10:34 UTC, Anuj Borah 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Github SSSD sssd issues 5349 0 None closed kcm: poor performance with large number of credentials 2021-02-19 11:55:51 UTC

Internal Links: 2035496

Description Thorsten Scherf 2020-09-07 11:24:34 UTC 
Description of problem:
This BZ is about tracking sssd-kcm performance related tasks.

Version-Release number of selected component (if applicable):
sssd-kcm-2.2.3-20.el8.x86_64

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 2 Alexey Tikhonov 2020-09-07 11:32:19 UTC 
Note: presumably this might be similar to BZ 1645624 but this yet to be verified.
Comment 3 Alexey Tikhonov 2020-09-07 17:17:58 UTC 
Might also be duplicate of bz 1867899
Comment 5 Pavel Březina 2020-09-17 13:07:03 UTC 
This is probably duplicate of Fedora BZ https://bugzilla.redhat.com/show_bug.cgi?id=1645624 which is currently under heavy investigation.
Comment 6 Alexey Tikhonov 2020-09-17 14:42:32 UTC 
*** Bug 1879978 has been marked as a duplicate of this bug. ***
Comment 7 Alexey Tikhonov 2020-09-17 14:44:45 UTC 
(In reply to Pavel Březina from comment #5)
> This is probably duplicate of Fedora BZ
> https://bugzilla.redhat.com/show_bug.cgi?id=1645624 which is currently under
> heavy investigation.

I'm not sure those are the same. See bz 1879978 - it talks about big number of tickets (50-60+). I hope we don't have anything O(N^2) in kcm code...
Comment 8 Pavel Březina 2020-09-23 10:45:56 UTC 
No. I just added https://bugzilla.redhat.com/show_bug.cgi?id=1645624#c60 for explanation. It is indeed bound to a large number of tickets but it may not be necessarily visible at the first sight as not all information stored in ccache is shown by default and you need -C switch (klist -A -C).

Thorsten, is it possible to get output of 'klist -A -C' from the affected user and machine?
Comment 13 Pavel Březina 2020-10-05 12:56:09 UTC 
Thank you. This is definitely a duplicate of Fedora https://bugzilla.redhat.com/show_bug.cgi?id=1645624.

Lets keep this BZ to track this in RHEL.
Comment 14 Pavel Březina 2020-10-05 12:58:56 UTC 
Upstream ticket:
https://github.com/SSSD/sssd/issues/5349
Comment 16 Pavel Březina 2020-10-23 15:28:32 UTC 
Upstream PR:
https://github.com/SSSD/sssd/pull/5375
Comment 17 Alexey Tikhonov 2020-10-29 19:51:26 UTC 
*** Bug 1867899 has been marked as a duplicate of this bug. ***
Comment 19 Alexey Tikhonov 2020-11-02 15:11:17 UTC 
*** Bug 1867899 has been marked as a duplicate of this bug. ***
Comment 20 Pavel Březina 2020-12-04 10:44:50 UTC 
Pushed PR: https://github.com/SSSD/sssd/pull/5375

* `master`
    * 325de5a5bb97ba026be6d22492bea8ab2605f1b5 - secrets: remove base64 enctype
    * 39277cdadd317b0ab86cdd37de0616bc3eecbe6a - secrets: move attrs names to macros
    * 9c1b51d057390fb5b26151f814a480911cda4cc9 - secrets: default to "plaintext" if "enctype" attr is missing
    * bf127d4f3f42e5b2afe25e512211439bc12a9904 - secrets: fix may_payload_size exceeded debug message
    * c3b314db57c34f64aaca7d74e76a9a955288bb51 - kcm: store credentials list in hash table to avoid cache lookups
    * a370553c90c2ed6df3b94c169c4960a6f978031f - sss_ptr_hash: fix double free for circular dependencies
    * 241ee30da12f564803793ee2b14c1522aabd9235 - kcm: add per-connection data to be shared between requests
    * 194447d35c11eb914f54719491dc5cfaab01b9a1 - kcm: use binary format to store ccache instead of json
    * f17740d831e16449495fff4ec57cc4800aaac83d - kcm: add spaces around operators in kcmsrv_ccache_key.c
    * 15069a647ed6c7f1ead42baa1d421d953c9bc557 - kcm: avoid suppression of cppcheck warning
    * e63a15038ac9c186626e4fdf681a6492031d1e40 - kcm: move sec key parser to separate file so it can be shared
    * 9b1631defdcaa3ea7e87889eb136e7fa935ab4ce - kcm: add json suffix to existing searialization functions
    * b6cc661b9f4162e590137430e945aa321fc13121 - iobuf: add more iobuf functions
    * ed08ba0023e63024bf1c52ae3f6596b9d804d0a5 - secrets: accept binary data instead of string
    * 908c15af9a9f8f0556a588e368e4a0b2e24ace1b - secrets: allow to specify secret's data format
    * 74fdaa64b27e88a6e0f153f8cb59989c572d4294 - kcm: avoid multiple debug messages if sss_sec_put fails
    * b8f28d9aa9d862cf504691c9c3f92941a63fb0a4 - kcm: disable encryption
    * 8edcea8c377e85d037e83065c1904fa4b92c4a39 - kcm: avoid name confusion in GET_CRED_UUID_LIST handlers
    * 47a316c850107f12d406f27abb216e26383dfab7 - kcm: fix typos in debug messages
Comment 22 Pavel Březina 2020-12-10 11:57:29 UTC 
Steps to reproduce:
1. Login as Kerberized SSSD user
2. Use gssapi authentication to obtain large number of service tickets
3. The previous step will take waaay longer time without patch

You can create service tickets with 'ipa service-add' and then fetch their keytab. To perform gssapi authentication, you can use a test app as described here: https://github.com/SSSD/sssd/pull/5375#issue-509041627

Or maybe a more simple way

1. Login as Kerberized SSSD user
2. Use kvno to obtain large number of service tickets (200+)
3. Run klist to list the credentials
4. The previous step should take waaay longer time without patch
Comment 23 Pavel Březina 2020-12-11 09:31:56 UTC 
Additional patch.

* `master`
    * 18b98836ef8e337992f0ecb239a32b9c3cedb750 - kcm: decode base64 encoded secret on upgrade path
Comment 26 Anuj Borah 2020-12-23 10:34:55 UTC 
Created attachment 1741516 [details]
test results
Comment 32 Pavel Březina 2021-01-06 12:14:19 UTC 
Ok, klist was just a suggestion to make the verification easier but it is obviously not enough to measure the performance difference. So gssapi should be used instead.

The most reliable way would be to use the steps from upstream PR [1], which obtains each service ticket using gssapi and the performance difference is huge there. The test application can be rewritten to python if you'd like to automate it in multihost tests.

Another possible solution (I did not test it) might be to obtain the tickets with kvno and then try to "ssh localhost" which should trigger gssapi authentication attempt (unless disabled) and probably will create a measurable difference.

[1] https://github.com/SSSD/sssd/pull/5375#issue-509041627
Comment 39 errata-xmlrpc 2021-05-18 15:03:57 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (sssd bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1666
Note You need to log in before you can comment on or make changes to this bug.