Bug 1174457 - [RFE] memberOf - add option to skip nested group lookups during delete operations
Summary: [RFE] memberOf - add option to skip nested group lookups during delete operat...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: 389-ds-base
Version: 7.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Noriko Hosoi
QA Contact: Viktor Ashirov
URL:
Whiteboard:
Depends On:
Blocks: 1181710
TreeView+ depends on / blocked
 
Reported: 2014-12-15 21:21 UTC by mreynolds
Modified: 2015-11-19 11:42 UTC (History)
5 users (show)

Fixed In Version: 389-ds-base-1.3.4.0-1.el7
Doc Type: Enhancement
Doc Text:
Added the memberOfSkipNested attribute, which accepts the "on" or "off" values, to improve delete performance when no nested groups are present. When set to "on", nested group lookups are not performed during a group delete operation.
Clone Of:
Environment:
Last Closed: 2015-11-19 11:42:22 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:2351 normal SHIPPED_LIVE 389-ds-base bug fix and enhancement update 2015-11-19 10:28:44 UTC

Description mreynolds 2014-12-15 21:21:37 UTC
This bug is created as a clone of upstream ticket:
https://fedorahosted.org/389/ticket/47963

The recursive nested group lookups performed during a group delete operation can take a very long time to complete if there are very large static groups(groups with with over 10K members).  If there are no nested groups, then it would be nice to have an option to skip the nested group check, which would significantly improve delete performance.

Comment 1 Noriko Hosoi 2015-02-03 19:43:25 UTC
Corresponding 6.7 bug was reopened...
https://bugzilla.redhat.com/show_bug.cgi?id=1167976

Comment 2 mreynolds 2015-02-04 00:47:57 UTC
Fixed upstream

Comment 3 Noriko Hosoi 2015-02-20 01:19:13 UTC
test case: dirsrvtests/tickets/ticket47963_test.py

Also, see this bug for more details.
https://bugzilla.redhat.com/show_bug.cgi?id=1167976

Comment 5 Sankar Ramalingam 2015-08-03 09:32:30 UTC
Deleting a group seems not working when setting memberOfSkipNested to off. The following test cases shows that with memberOfSkipNested:oof, the ldapdelete for groups fails with Operations error.

Where as, deleting a group with 10,000 user members take about more than 5 hrs. It looks to me that the feature is not working as expected. Hence, marking the bug as Assigned.


[root@vm-idm-015 ~]# cat /export/memofskipnest_on.ldif 
dn: cn=MemberOf Plugin,cn=plugins,cn=config
replace: memberOfSkipNested
memberOfSkipNested: on

[root@vm-idm-015 ~]# cat /export/memofskipnest_off.ldif 
dn: cn=MemberOf Plugin,cn=plugins,cn=config
replace: memberOfSkipNested
memberOfSkipNested: off

[root@vm-idm-015 ~]# cat /export/memof_on.ldif 
dn: cn=MemberOf Plugin,cn=plugins,cn=config
replace: nsslapd-pluginEnabled
nsslapd-pluginEnabled: on

[root@vm-idm-015 ~]# cat /export/memof_off.ldif 
dn: cn=MemberOf Plugin,cn=plugins,cn=config
replace: nsslapd-pluginEnabled
nsslapd-pluginEnabled: off

[root@vm-idm-015 slapd-testmember]# echo "Deleting groups with memberOfSkipNested: Off"; ldapmodify -x -p 10389 -h localhost -D "cn=Directory Manager" -w Secret123 -vf /export/memof_on.ldif ; ldapmodify -x -p 10389 -h localhost -D "cn=Directory Manager" -w Secret123 -vf /export/memofskipnest_off.ldif ; /usr/lib64/dirsrv/slapd-testmember/restart-slapd ; sleep 10 ; ldapsearch -LLL -x -p 10389 -h localhost -D "cn=Directory Manager" -w Secret123 -b "cn=memberOf plugin,cn=plugins,cn=config" | egrep 'pluginEnabled|memberofskipnested' ; time ldapdelete -x -p 10389 -h localhost -D "cn=Directory Manager" -w Secret123 "cn=group3,ou=Groups,dc=nestgrpsuff,dc=com" 

Deleting groups with memberOfSkipNested: Off
ldap_initialize( ldap://localhost:10389 )
replace nsslapd-pluginEnabled:
	on
modifying entry "cn=MemberOf Plugin,cn=plugins,cn=config"
modify complete

ldap_initialize( ldap://localhost:10389 )
replace memberOfSkipNested:
	off
modifying entry "cn=MemberOf Plugin,cn=plugins,cn=config"
modify complete

nsslapd-pluginEnabled: on
memberofskipnested: off
ldap_delete: Operations error (1)

real	212m59.436s
user	0m1.599s
sys	0m2.420s


[root@vm-idm-015 ~]# ldapsearch -x -p 10389 -h localhost -D "cn=Directory Manager" -w Secret123 -b "cn=group3,ou=Groups,dc=nestgrpsuff,dc=com" |grep -i "dn: "
dn: cn=group3,ou=Groups,dc=nestgrpsuff,dc=com


echo "Deleting Groups with memberOfSkipNested: On" ; ldapmodify -x -p 10389 -h localhost -D "cn=Directory Manager" -w Secret123 -vf /export/memof_on.ldif ; ldapmodify -x -p 10389 -h localhost -D "cn=Directory Manager" -w Secret123 -vf /export/memofskipnest_on.ldif ; /usr/lib64/dirsrv/slapd-testmember/restart-slapd ; sleep 10 ; ldapsearch -LLL -x -p 10389 -h localhost -D "cn=Directory Manager" -w Secret123 -b "cn=memberOf plugin,cn=plugins,cn=config" | egrep 'pluginEnabled|memberofskipnested' ; time ldapdelete -x -p 10389 -h localhost -D "cn=Directory Manager" -w Secret123 "cn=group2,ou=Groups,dc=nestgrpsuff,dc=com"

Deleting Groups with memberOfSkipNested: On
ldap_initialize( ldap://localhost:10389 )
replace nsslapd-pluginEnabled:
	on
modifying entry "cn=MemberOf Plugin,cn=plugins,cn=config"
modify complete

ldap_initialize( ldap://localhost:10389 )
replace memberOfSkipNested:
	on
modifying entry "cn=MemberOf Plugin,cn=plugins,cn=config"
modify complete

nsslapd-pluginEnabled: on
memberofskipnested: on
^[[6~
real	317m6.715s
user	0m2.232s
sys	0m4.938s


[root@vm-idm-015 ~]# ldapsearch -LLL -x -p 10389 -h localhost -D "cn=Directory Manager" -w Secret123 -b "cn=group2,ou=Groups,dc=nestgrpsuff,dc=com"
No such object (32)
Matched DN: ou=Groups,dc=nestgrpsuff,dc=com

Comment 6 Sankar Ramalingam 2015-08-03 09:37:18 UTC
Build tested:
[root@vm-idm-015 ~]# rpm -qa |grep -i 389-ds
389-ds-base-1.3.4.0-8.el7.x86_64
389-ds-base-libs-1.3.4.0-8.el7.x86_64
389-ds-base-debuginfo-1.3.4.0-8.el7.x86_64

Test setup details:

1. 100 Groups with 10,000 Users. All members are the same across this 100 groups. No nested groups.
2. 100,000 Users in 10 groups with each 10,000 users. Groups withing groups. Nested groups setup.

Comment 7 mreynolds 2015-08-03 12:13:58 UTC
(In reply to Sankar Ramalingam from comment #5)
> Deleting a group seems not working when setting memberOfSkipNested to off.
> The following test cases shows that with memberOfSkipNested:oof, the
> ldapdelete for groups fails with Operations error.

Actually by setting the the attribute to OFF is effectively not using the feature.   Looks like you found a completely separate is issue that has nothing to do with "memberOfSkipNested".  Are there any messages in the errors log from this failed modify?

> 
> Where as, deleting a group with 10,000 user members take about more than 5
> hrs. It looks to me that the feature is not working as expected. 

So how is it not working?  Again, this feature is just supposed to improve the performance compared to when it's set to OFF.  Just because it's taking 5 hours doesn't mean its not working.  In fact, when its "off" the operation completely fails in your test, and when its "on" it succeeds.  That seems to be a positive result, not a negative one.

Comment 8 Sankar Ramalingam 2015-08-03 15:39:48 UTC
(In reply to mreynolds from comment #7)
> (In reply to Sankar Ramalingam from comment #5)
> > Deleting a group seems not working when setting memberOfSkipNested to off.
> > The following test cases shows that with memberOfSkipNested:oof, the
> > ldapdelete for groups fails with Operations error.
> 
> Actually by setting the the attribute to OFF is effectively not using the
> feature.   Looks like you found a completely separate is issue that has
> nothing to do with "memberOfSkipNested".  Are there any messages in the
> errors log from this failed modify?

Error logs:

[31/Jul/2015:18:07:41 +051800] - WARNING: nestgrpsuff10389: entry cache size 10485760B is less than db size 118161408B; We recommend to increase the entry cache size nsslapd-cachememsize.
[31/Jul/2015:18:07:42 +051800] - slapd started.  Listening on All Interfaces port 10389 for LDAP requests
[31/Jul/2015:21:40:48 +051800] - libdb: BDB2055 Lock table is out of available lock entries
[31/Jul/2015:21:40:48 +051800] - idl_new.c BAD 60, err=12 Cannot allocate memory
[31/Jul/2015:21:40:48 +051800] - database index operation failed BAD 1130, err=12 Cannot allocate memory
[31/Jul/2015:21:40:49 +051800] - database index operation failed BAD 1140, err=12 Cannot allocate memory
[31/Jul/2015:21:40:49 +051800] - database index operation failed BAD 1230, err=12 Cannot allocate memory
[31/Jul/2015:21:40:49 +051800] - database index operation failed BAD 1042, err=12 Cannot allocate memory
[31/Jul/2015:21:40:49 +051800] - database index operation failed BAD 1040, err=12 Cannot allocate memory
[31/Jul/2015:21:40:49 +051800] - index_add_mods failed, err=12 Cannot allocate memory
[31/Jul/2015:21:40:49 +051800] memberof-plugin - memberof_postop_del: error deleting attr list - dn (cn=group3,ou=Groups,dc=nestgrpsuff,dc=com). Error (1)

Access logs:

[31/Jul/2015:18:07:52 +051800] conn=2 fd=65 slot=65 connection from ::1 to ::1
[31/Jul/2015:18:07:52 +051800] conn=2 op=0 BIND dn="cn=Directory Manager" method=128 version=3
[31/Jul/2015:18:07:52 +051800] conn=2 op=0 RESULT err=0 tag=97 nentries=0 etime=0 dn="cn=directory manager"
[31/Jul/2015:18:07:52 +051800] conn=2 op=1 DEL dn="cn=group3,ou=Groups,dc=nestgrpsuff,dc=com"
[31/Jul/2015:21:40:51 +051800] conn=2 op=1 RESULT err=1 tag=107 nentries=0 etime=12779
[31/Jul/2015:21:40:51 +051800] conn=2 op=2 UNBIND

> 
> > 
> > Where as, deleting a group with 10,000 user members take about more than 5
> > hrs. It looks to me that the feature is not working as expected. 
> 
> So how is it not working?  Again, this feature is just supposed to improve
> the performance compared to when it's set to OFF.  Just because it's taking
> 5 hours doesn't mean its not working.  In fact, when its "off" the operation
> completely fails in your test, and when its "on" it succeeds.  That seems to
> be a positive result, not a negative one.

I started comparing the results with 1000 users in the group. I will share the results in some time.

Comment 9 mreynolds 2015-08-03 16:51:38 UTC
(In reply to Sankar Ramalingam from comment #8)
> (In reply to mreynolds from comment #7)
> > (In reply to Sankar Ramalingam from comment #5)
> > > Deleting a group seems not working when setting memberOfSkipNested to off.
> > > The following test cases shows that with memberOfSkipNested:oof, the
> > > ldapdelete for groups fails with Operations error.
> > 
> > Actually by setting the the attribute to OFF is effectively not using the
> > feature.   Looks like you found a completely separate is issue that has
> > nothing to do with "memberOfSkipNested".  Are there any messages in the
> > errors log from this failed modify?
> 
> Error logs:
> 
> [31/Jul/2015:18:07:41 +051800] - WARNING: nestgrpsuff10389: entry cache size
> 10485760B is less than db size 118161408B; We recommend to increase the
> entry cache size nsslapd-cachememsize.
> [31/Jul/2015:18:07:42 +051800] - slapd started.  Listening on All Interfaces
> port 10389 for LDAP requests
> [31/Jul/2015:21:40:48 +051800] - libdb: BDB2055 Lock table is out of
> available lock entries

Yeah out of database locks, this is what caused the error 1(operations errors).

You can simply increase the locks when you try this particular test again:

ldapmodify ...
dn: cn=config,cn=ldbm database,cn=plugins,cn=config
changetype: modify
replace: nsslapd-db-locks
nsslapd-db-locks: 100000


Then restart the server.

Thanks,
Mark

Comment 10 Sankar Ramalingam 2015-08-07 05:16:30 UTC
With memberOf plugin enabled, compared the results for group deletions with memberOfSkipNested on/off.

1). 100 groups with each having 1000 members(same users on all the groups)
To delete one of the group, it takes about 5mins.

With memberOfskipNested: Off
real    5m26.776s
user    0m0.032s
sys    0m0.079s 

With memberOfskipNested: On
real    5m22.715s
user    0m0.040s
sys    0m0.088s 

2). 100 groups with each having 5000 members(same users on all the groups)
To delete one of the group, it takes about 1hr 20mins.

With memberOfskipNested: Off
real    79m39.732s
user    0m0.460s
sys    0m1.139s

With memberOfskipNested: On
real    78m1.695s
user    0m0.524s
sys    0m1.213s 

3). 100 groups with each having 10000 members(same users on all the groups)
To delete one of the group, it takes about 5 hrs.

With memberOfskipNested: Off
real    313m38.297s
user    0m1.827s
sys    0m3.707s 

With memberOfskipNested: On
real    307m12.849s
user    0m2.102s
sys    0m4.388s 

4). 1000 groups with each having 10000 members(same users on all the groups)
To delete one of the group, it takes about 50 hrs.

With memberOfskipNested: Off
real	2924m39.516s
user	0m13.925s
sys	0m31.762s

With memberOfskipNested: On
Its still running. I need another day or two to update the results here...

System details:
------------------------
[root@vm-idm-015 ~]# free -m
              total        used        free      shared  buff/cache   available
Mem:           7823         290        1189           8        6343        7219
Swap:          2559           0        2559

[root@vm-idm-015 ~]# top
top - 05:13:13 up 7 days, 19:30,  3 users,  load average: 1.00, 1.01, 1.04
Tasks: 108 total,   2 running, 106 sleeping,   0 stopped,   0 zombie
%Cpu(s): 23.9 us,  0.4 sy,  0.0 ni, 75.5 id,  0.1 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  8011272 total,  1217272 free,   297560 used,  6496440 buff/cache
KiB Swap:  2621436 total,  2621436 free,        0 used.  7392772 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                        
  825 dsuser    20   0 1259848  84368  49248 S  94.4  1.1 559:33.44 ns-slapd      
---------------------

Based on the observation so far... the difference between memberOfSkipNested: on/off is not substantial. However, I will wait for the last test to be completed. 

Meantime, any suggestions welcome to test/compare the performance of groups deletion in a nested/non-nested group setup.

Should I compare the results with 389-ds-base-1.3.1?

If you consider this comparison is more than enough and performance is well within the range, then I will go ahead and mark the bug as Verified.

Comment 15 errata-xmlrpc 2015-11-19 11:42:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2351.html


Note You need to log in before you can comment on or make changes to this bug.