Bug 1400653

Summary:	Improve memberof performance, caching for each group their ancestors DN
Product:	Red Hat Enterprise Linux 7	Reporter:	thierry bordaz <tbordaz>
Component:	389-ds-base	Assignee:	Noriko Hosoi <nhosoi>
Status:	CLOSED ERRATA	QA Contact:	Viktor Ashirov <vashirov>
Severity:	high	Docs Contact:	Aneta Šteflová Petrová <apetrova>
Priority:	high
Version:	7.4	CC:	mreynolds, nkinder, pasik, rmeggins, sramling
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	389-ds-base-1.3.6.1-3.el7	Doc Type:	Enhancement
Doc Text:	See the Doc Text in https://bugzilla.redhat.com/show_bug.cgi?id=1395940	Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-08-01 21:12:24 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1362239, 1399979

Description thierry bordaz 2016-12-01 17:40:56 UTC

Description of problem:

When updating a membership attribute of a group, impacted members will be updated (update of 'memberof'). It triggers internal searches of groups to compute their ancestors. But the same groups a searched multiple times.
The problem is described in http://www.port389.org/docs/389ds/design/memberof-scalability.html

Adding a cache that holds the ancestors of each group, would reduce the number of internal searches

Version-Release number of selected component (if applicable):

All versions

How reproducible:

 
Steps to Reproduce:
1. download https://github.com/freeipa/freeipa-tools/blob/master/create-test-data.py,

2. edit it with
users=1000,
groups=100,
groups_per_user=100,
nested_groups_max_level=2,
nested_groups_per_user=5,
hosts=1000,
hostgroups=100,
hostgroups_per_host=100,
nested_hostgroups_max_level=2,
nested_hostgroups_per_host=5,
direct_sudorules=5,  # users, hosts
indirect_sudorules=50,  # groups, hostgroups
sudorules_per_user=5,
sudorules_per_group=10,
sudorules_per_host=5,
sudorules_per_hostgroup=10,
direct_hbac=5,  # users, hosts
indirect_hbac=50,  # groups, hostgroups
hbac_per_user=5,
hbac_per_group=10,
hbac_per_host=5,
hbac_per_hostgroup=10

3. turn on internal op logging (to count the internal searches)

4. create a ldif file and import the entry (ldapadd)

Actual results:




Expected results:


Additional info:

Comment 1 thierry bordaz 2016-12-01 17:42:01 UTC

upstream tickets

https://fedorahosted.org/389/ticket/49031

Comment 2 thierry bordaz 2017-01-11 15:26:41 UTC

Patch pushed upstream https://fedorahosted.org/389/ticket/49031#comment:12

Comment 5 Sankar Ramalingam 2017-05-19 05:21:03 UTC

Made few changes to ticket48861_test.py script and verified from Pytest.
https://pagure.io/389-ds-base/issue/49031#comment-127558
https://pagure.io/389-ds-base/issue/raw/files/7db1d4f6ce18f73b95f2299ee45247cb8cf46b44def08a5efd274205b931e9c2-ticket48861_test.py

[0 root@qeos-125 tickets]# py.test  -v ticket48861_test.py 
=================== test session starts ======================
platform linux2 -- Python 2.7.5, pytest-3.0.7, py-1.4.33, pluggy-0.4.0 -- /usr/bin/python
cachedir: .cache
metadata: {'Python': '2.7.5', 'Platform': 'Linux-3.10.0-663.el7.x86_64-x86_64-with-redhat-7.4-Maipo', 'Packages': {'py': '1.4.33', 'pytest': '3.0.7', 'pluggy': '0.4.0'}, 'Plugins': {'beakerlib': '0.7.1', 'html': '1.14.2', 'cov': '2.5.1', 'metadata': '1.5.0'}}
DS build: 1.3.6.1
389-ds-base: 1.3.6.1-13.el7
nss: 3.28.4-8.el7
nspr: 4.13.1-1.0.el7_3
openldap: 2.4.44-4.el7
svrcore: 4.1.3-2.el7

rootdir: /mnt/testarea/test/ds/dirsrvtests/tests/tickets, inifile:
plugins: metadata-1.5.0, html-1.14.2, cov-2.5.1, beakerlib-0.7.1
collected 2 items 

ticket48861_test.py::test_ticket48861 PASSED
ticket48861_test.py::test_ticket48861_final PASSED
=================== 2 passed in 11.59 seconds ===================

Comment 6 thierry bordaz 2017-05-19 09:05:36 UTC

Hi Sankar,

thanks for the updating tests and results.
Just for confirmation, this BZ is mainly a performance oriented bug. When you switch it to 'verified' does that mean you completed performance testings ?

regards

Comment 7 Viktor Ashirov 2017-05-19 12:03:01 UTC

Thierry, you're right, I'm switching bz back to ON_QA.

Sankar, please change status to VERIFIED, once you provide performance test results.

Thanks.

Comment 9 errata-xmlrpc 2017-08-01 21:12:24 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2086