Bug 2123525 - perf - slow memberof fixup task for large static groups, high CPU use
Summary: perf - slow memberof fixup task for large static groups, high CPU use
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Directory Server
Classification: Red Hat
Component: 389-ds-base
Version: 11.5
Hardware: All
OS: Linux
high
urgent
Target Milestone: DS11.7
: dirsrv-11.7
Assignee: thierry bordaz
QA Contact: LDAP QA Team
Zuzana Zoubkova
URL:
Whiteboard: sync-to-jira
: 2141177 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-09-01 22:29 UTC by Marc Sauton
Modified: 2023-05-23 09:28 UTC (History)
13 users (show)

Fixed In Version: redhat-ds-11-8080020221130182235.022a399e
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-05-23 09:27:55 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github 389ds 389-ds-base issues 5440 0 None open memberof is slow on update/fixup if there are several 'groupattr' 2022-09-13 08:11:37 UTC
Red Hat Issue Tracker IDMDS-2521 0 None None None 2022-09-08 15:19:08 UTC
Red Hat Issue Tracker IDMDS-2769 0 None None None 2023-02-09 09:41:59 UTC
Red Hat Product Errata RHBA-2023:3267 0 None None None 2023-05-23 09:28:34 UTC

Comment 3 thierry bordaz 2022-09-02 10:33:32 UTC
The slow performance is related to the fixup thread having a too small NDN cache (evicting entries). The reason is that the NDN cache size is set at server level shared by workers threads. So if the NDN cache size is 2Gb, each worker can use a maximum of 2Gb/32(workers)=64Mb that is too small. (dsconf localhost config replace nsslapd-ndn-cache-max-size=2147483647)

An option would be to remove the limit of 2Gb for the global NDN cache. It is easy to do.

Another option is to make the NDN cache global. The reason it was made per thread was to reduce contention. However I think we can eliminate this contention even with a global cache. The main benefit of a global NDN is that normalization done by a given thread can be reused by the others. I think it is longer term improvement.

A possible workaround is to reduce the number of workers, to get larger NDN cache for the fixup task.

Comment 7 Serisha Nagothu 2022-09-07 18:41:18 UTC
Hi, 

Red Hat Directory Server performance is very critical to the customer. The customer is experiencing performance issues while migrating the data from Oracle LDAP to RHDS. We recommended several tuning parameters, which showed some improvement, but it still takes 3 to 4 days to complete the process. Still, the process is running single-threaded; even though the customer initiated eight threads, they are getting processed sequentially instead of multi-threaded. 

The renewal of the Oracle LDAP subscription is coming up in 10 days. The customer wants to remove Oracle LDAP altogether and migrate to RHDS. If the performance is not improved, they are forced to stay with Oracle LDAP because this delay impacts this manufacturing line. We are requesting help because this will affect the customer and Red Hat's business with the customer.

Thanks
Serisha

Comment 9 thierry bordaz 2022-09-08 14:51:36 UTC
https://github.com/389ds/389-ds-base/issues/5440

Comment 40 Jamie Chapman 2023-02-16 21:05:50 UTC
Verified with the following parameters

Red Hat Enterprise Linux release 8.6 (Ootpa)
RHDS 11.7 
389-ds-base-1.4.3.32-1.module+el8dsrv+17400+a7f2694e

Steps
Follow the steps detailed in Marc/Thierry reproducer notes attached in comment 1

Result
389-ds-base-1.4.3.28-8.module+el8.6.0+16880+945f9b53.x86_64 - memberof fixup takes ~60 min
389-ds-base-1.4.3.32-1.module+el8dsrv+17400+a7f2694e.x86_64 - memberof fixup takes ~4 min

Setting to verified

Comment 41 mreynolds 2023-05-03 15:28:10 UTC
*** Bug 2141177 has been marked as a duplicate of this bug. ***

Comment 43 errata-xmlrpc 2023-05-23 09:27:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (redhat-ds:11 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:3267


Note You need to log in before you can comment on or make changes to this bug.