Bug 1400653 - Improve memberof performance, caching for each group their ancestors DN
Summary: Improve memberof performance, caching for each group their ancestors DN
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: 389-ds-base
Version: 7.4
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Noriko Hosoi
QA Contact: Viktor Ashirov
Aneta Šteflová Petrová
URL:
Whiteboard:
Depends On:
Blocks: 1362239 1399979
TreeView+ depends on / blocked
 
Reported: 2016-12-01 17:40 UTC by thierry bordaz
Modified: 2020-09-13 21:53 UTC (History)
5 users (show)

Fixed In Version: 389-ds-base-1.3.6.1-3.el7
Doc Type: Enhancement
Doc Text:
See the Doc Text in https://bugzilla.redhat.com/show_bug.cgi?id=1395940
Clone Of:
Environment:
Last Closed: 2017-08-01 21:12:24 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github 389ds 389-ds-base issues 2090 0 None closed Improve memberof with a cache of group parents 2020-09-25 09:42:27 UTC
Red Hat Product Errata RHBA-2017:2086 0 normal SHIPPED_LIVE 389-ds-base bug fix and enhancement update 2017-08-01 18:37:38 UTC

Description thierry bordaz 2016-12-01 17:40:56 UTC
Description of problem:

When updating a membership attribute of a group, impacted members will be updated (update of 'memberof'). It triggers internal searches of groups to compute their ancestors. But the same groups a searched multiple times.
The problem is described in http://www.port389.org/docs/389ds/design/memberof-scalability.html

Adding a cache that holds the ancestors of each group, would reduce the number of internal searches

Version-Release number of selected component (if applicable):

All versions

How reproducible:

 
Steps to Reproduce:
1. download https://github.com/freeipa/freeipa-tools/blob/master/create-test-data.py,

2. edit it with
users=1000,
groups=100,
groups_per_user=100,
nested_groups_max_level=2,
nested_groups_per_user=5,
hosts=1000,
hostgroups=100,
hostgroups_per_host=100,
nested_hostgroups_max_level=2,
nested_hostgroups_per_host=5,
direct_sudorules=5,  # users, hosts
indirect_sudorules=50,  # groups, hostgroups
sudorules_per_user=5,
sudorules_per_group=10,
sudorules_per_host=5,
sudorules_per_hostgroup=10,
direct_hbac=5,  # users, hosts
indirect_hbac=50,  # groups, hostgroups
hbac_per_user=5,
hbac_per_group=10,
hbac_per_host=5,
hbac_per_hostgroup=10

3. turn on internal op logging (to count the internal searches)

4. create a ldif file and import the entry (ldapadd)

Actual results:




Expected results:


Additional info:

Comment 1 thierry bordaz 2016-12-01 17:42:01 UTC
upstream tickets

https://fedorahosted.org/389/ticket/49031

Comment 2 thierry bordaz 2017-01-11 15:26:41 UTC
Patch pushed upstream https://fedorahosted.org/389/ticket/49031#comment:12

Comment 5 Sankar Ramalingam 2017-05-19 05:21:03 UTC
Made few changes to ticket48861_test.py script and verified from Pytest.
https://pagure.io/389-ds-base/issue/49031#comment-127558
https://pagure.io/389-ds-base/issue/raw/files/7db1d4f6ce18f73b95f2299ee45247cb8cf46b44def08a5efd274205b931e9c2-ticket48861_test.py

[0 root@qeos-125 tickets]# py.test  -v ticket48861_test.py 
=================== test session starts ======================
platform linux2 -- Python 2.7.5, pytest-3.0.7, py-1.4.33, pluggy-0.4.0 -- /usr/bin/python
cachedir: .cache
metadata: {'Python': '2.7.5', 'Platform': 'Linux-3.10.0-663.el7.x86_64-x86_64-with-redhat-7.4-Maipo', 'Packages': {'py': '1.4.33', 'pytest': '3.0.7', 'pluggy': '0.4.0'}, 'Plugins': {'beakerlib': '0.7.1', 'html': '1.14.2', 'cov': '2.5.1', 'metadata': '1.5.0'}}
DS build: 1.3.6.1
389-ds-base: 1.3.6.1-13.el7
nss: 3.28.4-8.el7
nspr: 4.13.1-1.0.el7_3
openldap: 2.4.44-4.el7
svrcore: 4.1.3-2.el7

rootdir: /mnt/testarea/test/ds/dirsrvtests/tests/tickets, inifile:
plugins: metadata-1.5.0, html-1.14.2, cov-2.5.1, beakerlib-0.7.1
collected 2 items 

ticket48861_test.py::test_ticket48861 PASSED
ticket48861_test.py::test_ticket48861_final PASSED
=================== 2 passed in 11.59 seconds ===================

Comment 6 thierry bordaz 2017-05-19 09:05:36 UTC
Hi Sankar,

thanks for the updating tests and results.
Just for confirmation, this BZ is mainly a performance oriented bug. When you switch it to 'verified' does that mean you completed performance testings ?

regards

Comment 7 Viktor Ashirov 2017-05-19 12:03:01 UTC
Thierry, you're right, I'm switching bz back to ON_QA.

Sankar, please change status to VERIFIED, once you provide performance test results.

Thanks.

Comment 9 errata-xmlrpc 2017-08-01 21:12:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2086


Note You need to log in before you can comment on or make changes to this bug.