Bug 1174457
Summary: | [RFE] memberOf - add option to skip nested group lookups during delete operations | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | mreynolds |
Component: | 389-ds-base | Assignee: | Noriko Hosoi <nhosoi> |
Status: | CLOSED ERRATA | QA Contact: | Viktor Ashirov <vashirov> |
Severity: | unspecified | Docs Contact: | |
Priority: | unspecified | ||
Version: | 7.0 | CC: | gparente, mreynolds, nkinder, rmeggins, sramling |
Target Milestone: | rc | Keywords: | FutureFeature |
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | 389-ds-base-1.3.4.0-1.el7 | Doc Type: | Enhancement |
Doc Text: |
Added the memberOfSkipNested attribute, which accepts the "on" or "off" values, to improve delete performance when no nested groups are present. When set to "on", nested group lookups are not performed during a group delete operation.
|
Story Points: | --- |
Clone Of: | Environment: | ||
Last Closed: | 2015-11-19 11:42:22 UTC | Type: | --- |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | |||
Bug Blocks: | 1181710 |
Description
mreynolds
2014-12-15 21:21:37 UTC
Corresponding 6.7 bug was reopened... https://bugzilla.redhat.com/show_bug.cgi?id=1167976 Fixed upstream test case: dirsrvtests/tickets/ticket47963_test.py Also, see this bug for more details. https://bugzilla.redhat.com/show_bug.cgi?id=1167976 Deleting a group seems not working when setting memberOfSkipNested to off. The following test cases shows that with memberOfSkipNested:oof, the ldapdelete for groups fails with Operations error. Where as, deleting a group with 10,000 user members take about more than 5 hrs. It looks to me that the feature is not working as expected. Hence, marking the bug as Assigned. [root@vm-idm-015 ~]# cat /export/memofskipnest_on.ldif dn: cn=MemberOf Plugin,cn=plugins,cn=config replace: memberOfSkipNested memberOfSkipNested: on [root@vm-idm-015 ~]# cat /export/memofskipnest_off.ldif dn: cn=MemberOf Plugin,cn=plugins,cn=config replace: memberOfSkipNested memberOfSkipNested: off [root@vm-idm-015 ~]# cat /export/memof_on.ldif dn: cn=MemberOf Plugin,cn=plugins,cn=config replace: nsslapd-pluginEnabled nsslapd-pluginEnabled: on [root@vm-idm-015 ~]# cat /export/memof_off.ldif dn: cn=MemberOf Plugin,cn=plugins,cn=config replace: nsslapd-pluginEnabled nsslapd-pluginEnabled: off [root@vm-idm-015 slapd-testmember]# echo "Deleting groups with memberOfSkipNested: Off"; ldapmodify -x -p 10389 -h localhost -D "cn=Directory Manager" -w Secret123 -vf /export/memof_on.ldif ; ldapmodify -x -p 10389 -h localhost -D "cn=Directory Manager" -w Secret123 -vf /export/memofskipnest_off.ldif ; /usr/lib64/dirsrv/slapd-testmember/restart-slapd ; sleep 10 ; ldapsearch -LLL -x -p 10389 -h localhost -D "cn=Directory Manager" -w Secret123 -b "cn=memberOf plugin,cn=plugins,cn=config" | egrep 'pluginEnabled|memberofskipnested' ; time ldapdelete -x -p 10389 -h localhost -D "cn=Directory Manager" -w Secret123 "cn=group3,ou=Groups,dc=nestgrpsuff,dc=com" Deleting groups with memberOfSkipNested: Off ldap_initialize( ldap://localhost:10389 ) replace nsslapd-pluginEnabled: on modifying entry "cn=MemberOf Plugin,cn=plugins,cn=config" modify complete ldap_initialize( ldap://localhost:10389 ) replace memberOfSkipNested: off modifying entry "cn=MemberOf Plugin,cn=plugins,cn=config" modify complete nsslapd-pluginEnabled: on memberofskipnested: off ldap_delete: Operations error (1) real 212m59.436s user 0m1.599s sys 0m2.420s [root@vm-idm-015 ~]# ldapsearch -x -p 10389 -h localhost -D "cn=Directory Manager" -w Secret123 -b "cn=group3,ou=Groups,dc=nestgrpsuff,dc=com" |grep -i "dn: " dn: cn=group3,ou=Groups,dc=nestgrpsuff,dc=com echo "Deleting Groups with memberOfSkipNested: On" ; ldapmodify -x -p 10389 -h localhost -D "cn=Directory Manager" -w Secret123 -vf /export/memof_on.ldif ; ldapmodify -x -p 10389 -h localhost -D "cn=Directory Manager" -w Secret123 -vf /export/memofskipnest_on.ldif ; /usr/lib64/dirsrv/slapd-testmember/restart-slapd ; sleep 10 ; ldapsearch -LLL -x -p 10389 -h localhost -D "cn=Directory Manager" -w Secret123 -b "cn=memberOf plugin,cn=plugins,cn=config" | egrep 'pluginEnabled|memberofskipnested' ; time ldapdelete -x -p 10389 -h localhost -D "cn=Directory Manager" -w Secret123 "cn=group2,ou=Groups,dc=nestgrpsuff,dc=com" Deleting Groups with memberOfSkipNested: On ldap_initialize( ldap://localhost:10389 ) replace nsslapd-pluginEnabled: on modifying entry "cn=MemberOf Plugin,cn=plugins,cn=config" modify complete ldap_initialize( ldap://localhost:10389 ) replace memberOfSkipNested: on modifying entry "cn=MemberOf Plugin,cn=plugins,cn=config" modify complete nsslapd-pluginEnabled: on memberofskipnested: on ^[[6~ real 317m6.715s user 0m2.232s sys 0m4.938s [root@vm-idm-015 ~]# ldapsearch -LLL -x -p 10389 -h localhost -D "cn=Directory Manager" -w Secret123 -b "cn=group2,ou=Groups,dc=nestgrpsuff,dc=com" No such object (32) Matched DN: ou=Groups,dc=nestgrpsuff,dc=com Build tested: [root@vm-idm-015 ~]# rpm -qa |grep -i 389-ds 389-ds-base-1.3.4.0-8.el7.x86_64 389-ds-base-libs-1.3.4.0-8.el7.x86_64 389-ds-base-debuginfo-1.3.4.0-8.el7.x86_64 Test setup details: 1. 100 Groups with 10,000 Users. All members are the same across this 100 groups. No nested groups. 2. 100,000 Users in 10 groups with each 10,000 users. Groups withing groups. Nested groups setup. (In reply to Sankar Ramalingam from comment #5) > Deleting a group seems not working when setting memberOfSkipNested to off. > The following test cases shows that with memberOfSkipNested:oof, the > ldapdelete for groups fails with Operations error. Actually by setting the the attribute to OFF is effectively not using the feature. Looks like you found a completely separate is issue that has nothing to do with "memberOfSkipNested". Are there any messages in the errors log from this failed modify? > > Where as, deleting a group with 10,000 user members take about more than 5 > hrs. It looks to me that the feature is not working as expected. So how is it not working? Again, this feature is just supposed to improve the performance compared to when it's set to OFF. Just because it's taking 5 hours doesn't mean its not working. In fact, when its "off" the operation completely fails in your test, and when its "on" it succeeds. That seems to be a positive result, not a negative one. (In reply to mreynolds from comment #7) > (In reply to Sankar Ramalingam from comment #5) > > Deleting a group seems not working when setting memberOfSkipNested to off. > > The following test cases shows that with memberOfSkipNested:oof, the > > ldapdelete for groups fails with Operations error. > > Actually by setting the the attribute to OFF is effectively not using the > feature. Looks like you found a completely separate is issue that has > nothing to do with "memberOfSkipNested". Are there any messages in the > errors log from this failed modify? Error logs: [31/Jul/2015:18:07:41 +051800] - WARNING: nestgrpsuff10389: entry cache size 10485760B is less than db size 118161408B; We recommend to increase the entry cache size nsslapd-cachememsize. [31/Jul/2015:18:07:42 +051800] - slapd started. Listening on All Interfaces port 10389 for LDAP requests [31/Jul/2015:21:40:48 +051800] - libdb: BDB2055 Lock table is out of available lock entries [31/Jul/2015:21:40:48 +051800] - idl_new.c BAD 60, err=12 Cannot allocate memory [31/Jul/2015:21:40:48 +051800] - database index operation failed BAD 1130, err=12 Cannot allocate memory [31/Jul/2015:21:40:49 +051800] - database index operation failed BAD 1140, err=12 Cannot allocate memory [31/Jul/2015:21:40:49 +051800] - database index operation failed BAD 1230, err=12 Cannot allocate memory [31/Jul/2015:21:40:49 +051800] - database index operation failed BAD 1042, err=12 Cannot allocate memory [31/Jul/2015:21:40:49 +051800] - database index operation failed BAD 1040, err=12 Cannot allocate memory [31/Jul/2015:21:40:49 +051800] - index_add_mods failed, err=12 Cannot allocate memory [31/Jul/2015:21:40:49 +051800] memberof-plugin - memberof_postop_del: error deleting attr list - dn (cn=group3,ou=Groups,dc=nestgrpsuff,dc=com). Error (1) Access logs: [31/Jul/2015:18:07:52 +051800] conn=2 fd=65 slot=65 connection from ::1 to ::1 [31/Jul/2015:18:07:52 +051800] conn=2 op=0 BIND dn="cn=Directory Manager" method=128 version=3 [31/Jul/2015:18:07:52 +051800] conn=2 op=0 RESULT err=0 tag=97 nentries=0 etime=0 dn="cn=directory manager" [31/Jul/2015:18:07:52 +051800] conn=2 op=1 DEL dn="cn=group3,ou=Groups,dc=nestgrpsuff,dc=com" [31/Jul/2015:21:40:51 +051800] conn=2 op=1 RESULT err=1 tag=107 nentries=0 etime=12779 [31/Jul/2015:21:40:51 +051800] conn=2 op=2 UNBIND > > > > > Where as, deleting a group with 10,000 user members take about more than 5 > > hrs. It looks to me that the feature is not working as expected. > > So how is it not working? Again, this feature is just supposed to improve > the performance compared to when it's set to OFF. Just because it's taking > 5 hours doesn't mean its not working. In fact, when its "off" the operation > completely fails in your test, and when its "on" it succeeds. That seems to > be a positive result, not a negative one. I started comparing the results with 1000 users in the group. I will share the results in some time. (In reply to Sankar Ramalingam from comment #8) > (In reply to mreynolds from comment #7) > > (In reply to Sankar Ramalingam from comment #5) > > > Deleting a group seems not working when setting memberOfSkipNested to off. > > > The following test cases shows that with memberOfSkipNested:oof, the > > > ldapdelete for groups fails with Operations error. > > > > Actually by setting the the attribute to OFF is effectively not using the > > feature. Looks like you found a completely separate is issue that has > > nothing to do with "memberOfSkipNested". Are there any messages in the > > errors log from this failed modify? > > Error logs: > > [31/Jul/2015:18:07:41 +051800] - WARNING: nestgrpsuff10389: entry cache size > 10485760B is less than db size 118161408B; We recommend to increase the > entry cache size nsslapd-cachememsize. > [31/Jul/2015:18:07:42 +051800] - slapd started. Listening on All Interfaces > port 10389 for LDAP requests > [31/Jul/2015:21:40:48 +051800] - libdb: BDB2055 Lock table is out of > available lock entries Yeah out of database locks, this is what caused the error 1(operations errors). You can simply increase the locks when you try this particular test again: ldapmodify ... dn: cn=config,cn=ldbm database,cn=plugins,cn=config changetype: modify replace: nsslapd-db-locks nsslapd-db-locks: 100000 Then restart the server. Thanks, Mark With memberOf plugin enabled, compared the results for group deletions with memberOfSkipNested on/off. 1). 100 groups with each having 1000 members(same users on all the groups) To delete one of the group, it takes about 5mins. With memberOfskipNested: Off real 5m26.776s user 0m0.032s sys 0m0.079s With memberOfskipNested: On real 5m22.715s user 0m0.040s sys 0m0.088s 2). 100 groups with each having 5000 members(same users on all the groups) To delete one of the group, it takes about 1hr 20mins. With memberOfskipNested: Off real 79m39.732s user 0m0.460s sys 0m1.139s With memberOfskipNested: On real 78m1.695s user 0m0.524s sys 0m1.213s 3). 100 groups with each having 10000 members(same users on all the groups) To delete one of the group, it takes about 5 hrs. With memberOfskipNested: Off real 313m38.297s user 0m1.827s sys 0m3.707s With memberOfskipNested: On real 307m12.849s user 0m2.102s sys 0m4.388s 4). 1000 groups with each having 10000 members(same users on all the groups) To delete one of the group, it takes about 50 hrs. With memberOfskipNested: Off real 2924m39.516s user 0m13.925s sys 0m31.762s With memberOfskipNested: On Its still running. I need another day or two to update the results here... System details: ------------------------ [root@vm-idm-015 ~]# free -m total used free shared buff/cache available Mem: 7823 290 1189 8 6343 7219 Swap: 2559 0 2559 [root@vm-idm-015 ~]# top top - 05:13:13 up 7 days, 19:30, 3 users, load average: 1.00, 1.01, 1.04 Tasks: 108 total, 2 running, 106 sleeping, 0 stopped, 0 zombie %Cpu(s): 23.9 us, 0.4 sy, 0.0 ni, 75.5 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 8011272 total, 1217272 free, 297560 used, 6496440 buff/cache KiB Swap: 2621436 total, 2621436 free, 0 used. 7392772 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 825 dsuser 20 0 1259848 84368 49248 S 94.4 1.1 559:33.44 ns-slapd --------------------- Based on the observation so far... the difference between memberOfSkipNested: on/off is not substantial. However, I will wait for the last test to be completed. Meantime, any suggestions welcome to test/compare the performance of groups deletion in a nested/non-nested group setup. Should I compare the results with 389-ds-base-1.3.1? If you consider this comparison is more than enough and performance is well within the range, then I will go ahead and mark the bug as Verified. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2015-2351.html |