Red Hat Bugzilla – Bug 1236148
Slow replication when deleting large quantities of multi-valued attributes
Last modified: 2016-05-10 15:19:35 EDT
This bug is created as a clone of upstream ticket: https://fedorahosted.org/389/ticket/48195 We write to our directory via Java API. When changing our groups the ModificationItems are lists of distinguished names of users. There is no problem when we add new uniqueMembers to our groups, but deleting them takes much too long on the second (replicated) master. (example deleting 850 members from 86000 takes 159 seconds on dldap02). dldap01 [01/Jun/2015:13:44:47 +0200] conn=141407 op=3 MOD dn="cn=TSNMAIL-VerteilerZugriff,cn=TSNMAIL,cn=Applications,ou=Groups,ou=TSN-AD,dc=tirol+gvOuId=AT:L7:LVN:000001,dc=gv,dc=at" [01/Jun/2015:13:44:48 +0200] conn=141407 op=3 RESULT err=0 tag=103 nentries=0 etime=0.858000 csn=556c45b0000000010000 dldap02 [01/Jun/2015:13:44:48 +0200] conn=141354 op=4 MOD dn="cn=TSNMAIL-VerteilerZugriff,cn=TSNMAIL,cn=Applications,ou=Groups,ou=TSN-AD,dc=tirol+gvOuId=AT:L7:LVN:000001,dc=gv,dc=at" [01/Jun/2015:13:47:28 +0200] conn=141354 op=4 RESULT err=0 tag=103 nentries=0 etime=159.670000 csn=556c45b0000000010000 I found a closed ticket (#346) with some simmilarities, maybe the problem still exists in a multimaster replication environment when deleting attributes. Logging shows millions of calls to plugin_call_syntax_filter_ava uniqueMember=gvGid=AT:L7:TSN:i.weitlaner,ou=People,ou=TSN-AD,dc=tirol+gvOuId=AT:L7:LVN:000001,dc=gv,dc=at ?
$ rpm -qa | grep 389-ds-base 389-ds-base-libs-1.2.11.15-74.el6.x86_64 389-ds-base-1.2.11.15-74.el6.x86_641 Verification scenario by lkrispen - https://fedorahosted.org/389/ticket/48195#comment:21 "I tested this fix with the following scenario: have 2 masters, have a group with 90000 members, delete 200 members (members 70001-70200). without the fix, on the master it takes 1 sec, on the replica 40 sec with the fix it takes 1 sec on both" 1) Setup two masters replication master1 - 389 master2 - 390 2) Enable MemberOf Plugin on both instances: $ ldapmodify -x -p 389 -h localhost -D "cn=Directory Manager" -w Secret123 dn: cn=MemberOf Plugin,cn=plugins,cn=config changetype: modify add: memberofgroupattr memberofgroupattr: uniquemember - replace: nsslapd-pluginEnabled nsslapd-pluginEnabled: on 3) Increase nsslapd-maxbersize for adding big attributes on both instances: $ ldapmodify -x -p 389 -h localhost -D "cn=Directory Manager" -w Secret123 dn: cn=config changetype: modify replace: nsslapd-maxbersize nsslapd-maxbersize: 400000000 4) Add 90000 users and add them as members on the group $ head 02users_usrA_grpA.ldif dn: uid=usrA1,dc=example,dc=com uid: usrA1 objectClass: top objectClass: person objectClass: inetUser sn: usrA1 cn: usrA1 memberOf: cn=grpA,ou=groups,dc=example,dc=com dn: uid=usrA2,dc=example,dc=com --- $ head grpA.ldif dn: cn=grpA,ou=groups,dc=example,dc=com objectclass: top objectclass: groupOfNames cn: grpA member: uid=usrA1,dc=example,dc=com member: uid=usrA2,dc=example,dc=com member: uid=usrA3,dc=example,dc=com member: uid=usrA4,dc=example,dc=com member: uid=usrA5,dc=example,dc=com member: uid=usrA6,dc=example,dc=com --- $ ldapmodify -a -x -p 389 -h localhost -D "cn=Directory Manager" -w Secret123 -f 02users_usrA_grpA.ldif $ ldapmodify -a -x -p 389 -h localhost -D "cn=Directory Manager" -w Secret123 -f grpA.ldif 5) Wait for replica $ ldapsearch -x -p 389 -h localhost -D "cn=Directory Manager" -w Secret123 -b "cn=grpA,ou=groups,dc=example,dc=com" |grep -i member: |wc -l 90000 $ ldapsearch -x -p 390 -h localhost -D "cn=Directory Manager" -w Secret123 -b "cn=grpA,ou=groups,dc=example,dc=com" |grep -i member: |wc -l 90000 6) Delete 200 members from master1 $ head del200members.ldif dn: cn=grpA,ou=groups,dc=example,dc=com changetype: modify delete: member member: uid=usrA70001,dc=example,dc=com member: uid=usrA70002,dc=example,dc=com member: uid=usrA70003,dc=example,dc=com member: uid=usrA70004,dc=example,dc=com member: uid=usrA70005,dc=example,dc=com member: uid=usrA70006,dc=example,dc=com member: uid=usrA70007,dc=example,dc=com --- $ time ldapmodify -x -p 389 -h localhost -D "cn=Directory Manager" -w Secret123 -f del200members.ldif modifying entry "cn=grpA,ou=groups,dc=example,dc=com" real 0m0.325s user 0m0.001s sys 0m0.001s 7) Delete 200 members from master2 $ head del200members2.ldif dn: cn=grpA,ou=groups,dc=example,dc=com changetype: modify delete: member member: uid=usrA60001,dc=example,dc=com member: uid=usrA60002,dc=example,dc=com member: uid=usrA60003,dc=example,dc=com member: uid=usrA60004,dc=example,dc=com member: uid=usrA60005,dc=example,dc=com member: uid=usrA60006,dc=example,dc=com member: uid=usrA60007,dc=example,dc=com --- $ time ldapmodify -x -p 390 -h localhost -D "cn=Directory Manager" -w Secret123 -f del200members2.ldif modifying entry "cn=grpA,ou=groups,dc=example,dc=com" real 0m0.326s user 0m0.001s sys 0m0.001s 8) Check $ ldapsearch -x -p 389 -h localhost -D "cn=Directory Manager" -w Secret123 -b "cn=grpA,ou=groups,dc=example,dc=com" |grep -i member: |wc -l 89800 $ ldapsearch -x -p 390 -h localhost -D "cn=Directory Manager" -w Secret123 -b "cn=grpA,ou=groups,dc=example,dc=com" |grep -i member: |wc -l 89600 Result: deletion times on master and replica are the same. Marking as verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0737.html