Description of problem:
Entry is added to the master server but fails to replicate due to a changelog error caused by a database deadlock.

Version-Release number of selected component (if applicable):
389-ds-base.x86_64            1.2.11.15-14.el6_4   rhel-x86_64-server-6

How reproducible:
Consistently

Steps to Reproduce:
1. Setup a multi-master replication environment (2 masters) and 3 consumers. Replication should be real-time i.e. directories always kept in sync.
2. Enable the DNA plugin on the master servers. Example config is as follows:

master server 1:
dn: cn=Posix IDs,cn=Distributed Numeric Assignment Plugin,cn=plugins,cn=config
changetype: add
objectClass: top
objectClass: extensibleObject
cn: Posix IDs
dnafilter: (|(objectclass=posixAccount)(objectClass=posixGroup))
dnamagicregen: 999
dnamaxvalue: 4294967295
dnanextvalue: 131073
dnascope: dc=dev,dc=id,dc=ubc,dc=ca
dnasharedcfgdn: cn=posix-ids,cn=dna,cn=plugins,cn=configuration,ou=ELDAP,ou=Services,dc=dev,dc=id,dc=ubc,dc=ca
dnathreshold: 1000
dnatype: uidNumber
dnatype: gidNumber

master server 2:
dn: cn=Posix IDs,cn=Distributed Numeric Assignment Plugin,cn=plugins,cn=config
changetype: add
objectClass: top
objectClass: extensibleObject
cn: Posix IDs
dnafilter: (|(objectclass=posixAccount)(objectClass=posixGroup))
dnamagicregen: 999
dnamaxvalue: 0
dnanextvalue: 0
dnascope: dc=dev,dc=id,dc=ubc,dc=ca
dnasharedcfgdn: cn=posix-ids,cn=dna,cn=plugins,cn=configuration,ou=ELDAP,ou=Services,dc=dev,dc=id,dc=ubc,dc=ca
dnathreshold: 1000
dnatype: uidNumber
dnatype: gidNumber

3. Setup a LDAP client (e.g. JMeter LDAP tester) to create and delete entries on one of the master servers. An example entry is:

objectClass	top
objectClass	person
objectClass	organizationalPerson
objectClass	inetOrgPerson
cn	Test Plan
sn	Plan
givenName	Test
userPassword	{SSHA}aBIV4atRWyMZqiWucSiZgYGVEw1bJa7V
objectClass	posixAccount
gidNumber	999
uidNumber	999
uid	${entrydn}
homeDirectory	/home/somedir

${entrydn} can just be a unique uid that is sequentially listed in a text file e.g. jmeter0, jmeter1, jmeter2, etc.

4. Setup the following sequence of actions to run in a loop from the LDAP client:

a. BIND
b. ADD entry.
c. DEL entry.
d. UNBIND.

5. After running for about 5-6 minutes, you should trigger the error. An example of the error log is given below.

Actual results:
1. Entry is added to master server and a return code of 0 is given to the client.
2. The changelog for the entry fails to write, so while the entry is successfully written to the master, it fails to replicate to the other servers in the environment.

Error log:

[17/Jun/2013:15:23:13 -0700] NSMMReplicationPlugin - replica_replace_ruv_tombstone: failed to update replication update vector for replica dc=id,dc=ubc,dc=ca: LDAP error - 51
[17/Jun/2013:15:23:15 -0700] NSMMReplicationPlugin - changelog program - _cl5WriteOperationTxn: retry (49) the transaction (csn=51bf8c50004100010000) failed (rc=-30994 (DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock))
[17/Jun/2013:15:23:15 -0700] NSMMReplicationPlugin - changelog program - _cl5WriteOperationTxn: failed to write entry with csn (51bf8c50004100010000); db error - -30994 DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock
[17/Jun/2013:15:23:15 -0700] NSMMReplicationPlugin - write_changelog_and_ruv: can't add a change for uid=jmeter741,dc=id,dc=ubc,dc=ca (uniqid: 70b03020-d79c11e2-8c03dfeb-4acc1d05, optype: 16) to changelog csn 51bf8c50004100010000

Access log:

[17/Jun/2013:15:23:10 -0700] conn=119116 fd=90 slot=90 connection from 142.103.1.221 to 10.7.0.51
[17/Jun/2013:15:23:10 -0700] conn=119116 op=0 BIND dn="cn=Directory Manager" method=128 version=3
[17/Jun/2013:15:23:10 -0700] conn=119116 op=0 RESULT err=0 tag=97 nentries=0 etime=0 dn="cn=directory manager"
[17/Jun/2013:15:23:10 -0700] conn=119116 op=1 ADD dn="uid=jmeter741,dc=id,dc=ubc,dc=ca"
[17/Jun/2013:15:23:15 -0700] conn=119116 op=1 RESULT err=0 tag=105 nentries=0 etime=5 csn=51bf8c50004100010000
[17/Jun/2013:15:23:15 -0700] conn=119116 op=2 DEL dn="uid=jmeter741,dc=id,dc=ubc,dc=ca"
[17/Jun/2013:15:23:15 -0700] conn=119116 op=2 RESULT err=0 tag=107 nentries=0 etime=0 csn=51bf8c55001100010000
[17/Jun/2013:15:23:15 -0700] conn=119116 op=3 UNBIND
[17/Jun/2013:15:23:15 -0700] conn=119116 op=3 fd=90 closed - U1

Expected results:
1. Changelog for all entries is correctly written without deadlocking.
2. The change is replicated to all servers in the environment.

Additional info:
Please see the following discussion thread for more info:

https://lists.fedoraproject.org/pipermail/389-users/2013-June/016014.html