Bug 1167976

Summary: [RFE] memberOf - add option to skip nested group lookups during delete operations
Product: Red Hat Enterprise Linux 6 Reporter: mreynolds
Component: 389-ds-baseAssignee: mreynolds
Status: CLOSED ERRATA QA Contact: Viktor Ashirov <vashirov>
Severity: medium Docs Contact: Tomas Capek <tcapek>
Priority: high    
Version: 6.7CC: arubin, gparente, jgalipea, mreynolds, nhosoi, nkinder, rmeggins
Target Milestone: rcKeywords: FutureFeature, Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 389-ds-base-1.2.11.15-51.el6 Doc Type: Release Note
Doc Text:
Performance improvements for Directory Server delete operations Previously, the recursive nested group look-ups performed during a group delete operation could take a long time to complete if there were very large static groups. The new *memberOfSkipNested* configuration attribute has been added to allow skipping the nested group check, thus improving performance of delete operations significantly.
Story Points: ---
Clone Of:
: 1178954 (view as bug list) Environment:
Last Closed: 2015-07-22 06:36:12 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1178954    

Description mreynolds 2014-11-25 18:58:26 UTC
This bug is created as a clone of upstream ticket:
https://fedorahosted.org/389/ticket/47963

The recursive nested group lookups performed during a group delete operation can take a very long time to complete if there are very large static groups(groups with with over 10K members).  If there are no nested groups, then it would be nice to have an option to skip the nested group check, which would significantly improve delete performance.

Comment 1 mreynolds 2014-11-26 22:35:06 UTC
Fixed upstream

Comment 2 Sankar Ramalingam 2014-12-09 13:49:19 UTC
I am afraid we could accommodate this for RHEL6.7 cycle.

Comment 3 RHEL Product and Program Management 2014-12-09 13:56:03 UTC
Quality Engineering Management has reviewed and declined this request.
You may appeal this decision by reopening this request.

Comment 5 Sankar Ramalingam 2014-12-17 14:32:55 UTC
I am putting a sample test case to validate this feature. I will go ahead and extend/add test cases if this is the right approach.

Sample test case:
--------
Add static and nested groups with more than 10K members.
a). Deletion of static groups with memberOfSkipNested: to ON.
b). Deletion of static groups with memberOfSkipNested: to OFF.

Exp_Res: Deletion of static groups with memberOfSkipNested: to ON should be significantly faster than with memberOfSkipNested: to OFF.
--------

I have few questions for this feature implementation:

1). Are we going to add this feature for RHEL7.1 as part of memberOf suffixes configurable - Bug #1044170? or a new bug will be added?
2). Are we back porting changes for memberof suffixes to be configurable to RHEL6.x from RHEL7.1? The patch for ticket - https://fedorahosted.org/389/ticket/47963, has references for entryScope attribute. So, I wanted to clarify things.
3). Would this option "memberOfSkipNested: ON", skip deleting memberof attributes when deleting nested groups? Should it be one of the test case?
4). Should the performance be the same, if I don't have any nested groups and trying to delete with options ON and OFF?

Comment 6 mreynolds 2014-12-17 14:54:43 UTC
(In reply to Sankar Ramalingam from comment #5)
> I am putting a sample test case to validate this feature. I will go ahead
> and extend/add test cases if this is the right approach.
> 
> Sample test case:
> --------
> Add static and nested groups with more than 10K members.
> a). Deletion of static groups with memberOfSkipNested: to ON.
> b). Deletion of static groups with memberOfSkipNested: to OFF.
> 
> Exp_Res: Deletion of static groups with memberOfSkipNested: to ON should be
> significantly faster than with memberOfSkipNested: to OFF.

Well I used a specific data set from the customer to reproduce the issue.  Please contact German Parente for more details on this.

> --------
> 
> I have few questions for this feature implementation:
> 
> 1). Are we going to add this feature for RHEL7.1 as part of memberOf
> suffixes configurable - Bug #1044170? or a new bug will be added?

We have a bug for against 7.0, but looks like it was only acked for 7.2 (this might be more a question for management):

https://bugzilla.redhat.com/show_bug.cgi?id=1174457

> 2). Are we back porting changes for memberof suffixes to be configurable to
> RHEL6.x from RHEL7.1? The patch for ticket -
> https://fedorahosted.org/389/ticket/47963, has references for entryScope
> attribute. So, I wanted to clarify things.

EntryScope has nothing to do with this fix.  I'm not sure what references you are referring to.  In the patch file from master branch, it is near some entryScope code/variables, but it has nothing to do with it.

> 3). Would this option "memberOfSkipNested: ON", skip deleting memberof
> attributes when deleting nested groups? Should it be one of the test case?

If there are groups of groups, the users in the nexted groups will not be updated if it's set to "on".  So the only the direct members of the top group will be updated. 

Yes it should be tested.

> 4). Should the performance be the same, if I don't have any nested groups
> and trying to delete with options ON and OFF?

Yes, it should be very close in performance.

Please let me know if you have any more questions.

Thanks,
Mark

Comment 7 Noriko Hosoi 2015-01-06 19:31:42 UTC
*** Bug 1178954 has been marked as a duplicate of this bug. ***

Comment 8 German Parente 2015-02-03 15:43:51 UTC
Re-opening this bug because some tests have not worked for me.

NOTE: I am using a hotfix on top of 6.6 with the backport of the fix.

Here are my tests:


1) Group deletion.

- before delete:

dn: uid=user38,ou=People,o=redhat
objectClass: inetuser
objectClass: top
uid: user38user38
uid: user38
userPassword:: dXNlcjM4
memberOf: cn=directory administrators,o=redhat
memberOf: cn=accounting managers,ou=groups,o=redhat
memberOf: cn=hr managers,ou=groups,o=redhat
memberOf: cn=pd managers,ou=groups,o=redhat

(user is member of four groups)

- delete group entry:
  "cn=accounting managers,ou=groups,o=redhat"

- after delete:

dn: uid=user38,ou=People,o=redhat
objectClass: inetuser
objectClass: top
uid: user38user38
uid: user38
userPassword:: dXNlcjM4

all memberships have been deleted.

2) add membership:

- add membership to group "cn=pd managers,ou=groups,o=redhat"

ldapmodify -p 4389 -h localhost -D "cn=directory manager" -w secret12 << EOF
dn: cn=pd managers,ou=groups,o=redhat
changetype: modify
add: uniquemember
uniquemember: uid=user48,ou=People,o=redhat

- check memberof attribute:

dn: uid=user48,ou=People,o=redhat
objectClass: inetuser
objectClass: top
uid: user48user48
uid: user48
userPassword:: dXNlcjQ4

not there.


NOTE: once memberofskipnested is set to off, both former testcases are giving right results.

1) 
after delete:

After delete
dn: uid=user38,ou=People,o=redhat
objectClass: inetuser
objectClass: top
uid: user38user38
uid: user38
userPassword:: dXNlcjM4
memberOf: cn=directory administrators,o=redhat
memberOf: cn=hr managers,ou=groups,o=redhat
memberOf: cn=pd managers,ou=groups,o=redhat

2) 
after modify
dn: uid=user48,ou=People,o=redhat
objectClass: inetuser
objectClass: top
uid: user48user48
uid: user48
userPassword:: dXNlcjQ4
memberOf: cn=pd managers,ou=groups,o=redhat

Comment 9 mreynolds 2015-02-03 16:46:00 UTC
I can reproduce the problem.  Investigating...

Comment 10 mreynolds 2015-02-04 00:48:20 UTC
Fixed upstream

Comment 11 German Parente 2015-02-04 10:29:28 UTC
Mark, I have rebuilt and reinstalled my hotfix using your new patch.

Re-played my automatic test and it worked perfect ! 

Thanks a lot for such a quick fix. I appreciate it.

German.

Comment 13 Noriko Hosoi 2015-02-20 01:18:47 UTC
test case: dirsrvtests/tickets/ticket47963_test.py

Comment 14 Viktor Ashirov 2015-03-15 22:56:34 UTC
Build tested:
389-ds-base-1.2.11.15-52.el6.x86_64
389-ds-base-libs-1.2.11.15-52.el6.x86_64

============================= test session starts ==============================
platform linux2 -- Python 2.6.6 -- py-1.4.26 -- pytest-2.6.4 -- /usr/bin/python
collected 2 items 

ds/dirsrvtests/tickets/ticket47963_test.py::test_ticket47963 PASSED
ds/dirsrvtests/tickets/ticket47963_test.py::test_ticket47963_final PASSED

========================== 2 passed in 63.32 seconds ===========================

Marking as VERIFIED.

Comment 15 errata-xmlrpc 2015-07-22 06:36:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-1326.html