Bug 1258610

Summary:	total update request must not be lost
Product:	Red Hat Enterprise Linux 7	Reporter:	Noriko Hosoi <nhosoi>
Component:	389-ds-base	Assignee:	Noriko Hosoi <nhosoi>
Status:	CLOSED ERRATA	QA Contact:	Viktor Ashirov <vashirov>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	7.0	CC:	nhosoi, nkinder, pkundal, rmeggins
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	389-ds-base-1.3.5.2-1.el7	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-11-03 20:35:49 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Noriko Hosoi 2015-08-31 18:35:43 UTC

This bug is created as a clone of upstream ticket:
https://fedorahosted.org/389/ticket/48255

If a replication agreement is created with the autoinit flag the total protocol is started,
but if it fails it switches to incremntal and can loop forever in comparing the generationID. This is becaus next state is set to INCREMENTAl and the result of the total run is not checked.
{{{
  rp->next_state = STATE_PERFORMING_INCREMENTAL_UPDATE;
  .....
  rp->prp_total->run(rp->prp_total);
                
  agmt_replica_init_done (agmt);
}}}

there is a commment in total update:

{{{
    rc = acquire_replica (prp, REPL_NSDS50_TOTAL_PROTOCOL_OID, NULL /* ruv */);
    /* We never retry total protocol, even in case a transient error.
       This is because if somebody already updated the replica we don't
       want to do it again */
}}}
 
This is reasonable in some cases, but if the genrationID is still different, we know that no other server did update the replica.
Especially if a new server is added and we want to initialize it, the total update request should not get lost

Comment 1 Mike McCune 2016-03-28 23:12:48 UTC

This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions

Comment 4 Noriko Hosoi 2016-06-15 16:30:42 UTC

Based upon the git log, almost simultaneous replication total update and incremental updates were both failing.

> Bug Description:  if the initial replica acquire fails the
>                   protocol switches to incremental update
>                   but the database has not been initialized and
>                   the incremental update will als fail continously
> 
> Fix Description: for transient failures do a few retries.

0. Set up MMR (MasterA and MasterB)
1. Start the total update/bulk import on MasterA
2. Update entries on MasterA
Note: 2 is supposed to follow 1 immediately.

It'd depend upon the timing, repeat 1 & 2 several times.

If the total updates and the updates made on MasterA are successfully replicated to MasterB, we could say the fix is verified.

Comment 5 Punit Kundal 2016-07-15 12:36:48 UTC

RHEL:
RHEL 7.3 x86_64 Server

DS builds:
[root@ds ~]# rpm -qa | grep 389
389-ds-base-debuginfo-1.3.5.8-1.el7.x86_64
389-ds-base-1.3.5.10-5.el7.x86_64
389-ds-base-snmp-1.3.5.10-5.el7.x86_64
389-ds-base-libs-1.3.5.10-5.el7.x86_64

Steps Performed:

1. Created two standalone instances Master1 and Master2
 
2. Created two new suffixes on each instance as below:

[root@ds new_suffix]# ldapadd -x -D 'cn=Directory Manager' -w secret123 -h localhost -p 389 -f new_suffix.ldif
adding new entry "cn=UserData,cn=ldbm database,cn=plugins,cn=config"
 
adding new entry "cn=MyData,cn=ldbm database,cn=plugins,cn=config"
 
adding new entry "cn="dc=redhat,dc=com",cn=mapping tree,cn=config"
 
adding new entry "cn="dc=xyz,dc=com",cn=mapping tree,cn=config"
 
[root@ds new_suffix]# ldapadd -x -D 'cn=Directory Manager' -w secret123 -h localhost -p 1389 -f new_suffix.ldif
adding new entry "cn=UserData,cn=ldbm database,cn=plugins,cn=config"
 
adding new entry "cn=MyData,cn=ldbm database,cn=plugins,cn=config"
 
adding new entry "cn="dc=redhat,dc=com",cn=mapping tree,cn=config"
 
adding new entry "cn="dc=xyz,dc=com",cn=mapping tree,cn=config"
 
3. Imported 10k entries for dc=redhat,dc=com on master1

[root@ds ~]# /usr/lib64/dirsrv/slapd-master1/ldif2db -n UserData -i /var/lib/dirsrv/slapd-master1/ldif/redhat_data.ldif
importing data ...
[15/Jul/2016:15:03:02.164390053 +051800] import UserData: Import complete.  Processed 10002 entries in 4 seconds. (2500.50 entries/sec)
     
4. Imported 15k entries for dc=xyz,dc=com on master1

[root@ds ~]# /usr/lib64/dirsrv/slapd-master1/ldif2db -n MyData -i /var/lib/dirsrv/slapd-master1/ldif/xyz_data.ldif
importing data ...
[15/Jul/2016:15:04:49.187410762 +051800] import MyData: Import complete.  Processed 15002 entries in 4 seconds. (3750.50 entries/sec)
     
5. Verified that the data for both the suffixes has been imported properly on master1

[root@ds ~]# ldapsearch -xLLL -D 'cn=Directory Manager' -w secret123 -h localhost -p 389 -b 'dc=redhat,dc=com' | wc -l
120014
[root@ds ~]# ldapsearch -xLLL -D 'cn=Directory Manager' -w secret123 -h localhost -p 389 -b 'dc=xyz,dc=com' | wc -l
180014
 
6. Configured a 2x MMR setup for both the suffixes dc=redhat,dc=com and dc=xyz,dc=com by adding required replication configuration entries on both masters
 
7. Verified that no data has been replicated to master2 for either suffix

[root@ds ~]# ldapsearch -xLLL -D 'cn=Directory Manager' -w secret123 -h localhost -p 1389 -b 'dc=redhat,dc=com' dn | wc -l
No such object (32)
0
[root@ds ~]# ldapsearch -xLLL -D 'cn=Directory Manager' -w secret123 -h localhost -p 1389 -b 'dc=xyz,dc=com' dn | wc -l
No such object (32)
0
 
8. Initiated a total update for suffix dc=redhat,dc=com on master1

[root@ds ~]# ldapmodify -x -D 'cn=Directory Manager' -w secret123 -h localhost -p 389
dn: cn=Agreement1,cn=replicaA,cn="dc=redhat,dc=com",cn=mapping tree,cn=config
changetype: modify
replace: nsds5BeginReplicaRefresh
nsds5BeginReplicaRefresh: start
modifying entry "cn=Agreement1,cn=replicaA,cn="dc=redhat,dc=com",cn=mapping tree,cn=config"
 
9. while total update was progressing, modified the last 11 entries under dc=redhat,dc=com suffix and added mail attribute to them

[root@ds ~]# ldapmodify -x -D 'cn=Directory Manager' -w secret123 -h localhost -p 389 -f conf_files/ldif_files/redhat_mod.ldif
modifying entry "uid=tuser9990,ou=people,dc=redhat,dc=com"
 
modifying entry "uid=tuser9991,ou=people,dc=redhat,dc=com"
 
modifying entry "uid=tuser9992,ou=people,dc=redhat,dc=com"
 
modifying entry "uid=tuser9993,ou=people,dc=redhat,dc=com"
 
modifying entry "uid=tuser9994,ou=people,dc=redhat,dc=com"
 
modifying entry "uid=tuser9995,ou=people,dc=redhat,dc=com"
 
modifying entry "uid=tuser9996,ou=people,dc=redhat,dc=com"
 
modifying entry "uid=tuser9997,ou=people,dc=redhat,dc=com"
 
modifying entry "uid=tuser9998,ou=people,dc=redhat,dc=com"
 
modifying entry "uid=tuser9999,ou=people,dc=redhat,dc=com"
 
modifying entry "uid=tuser10000,ou=people,dc=redhat,dc=com"
 
10. Verified that data for dc=redhat,dc=com was consistent across both masters

[root@ds ~]# ldapsearch -xLLL -D 'cn=Directory Manager' -w secret123 -h localhost -p 389 -b 'dc=redhat,dc=com' | wc -l
120025
[root@ds ~]# ldapsearch -xLLL -D 'cn=Directory Manager' -w secret123 -h localhost -p 1389 -b 'dc=redhat,dc=com' | wc -l
120025
 
11. Performed steps 9 and 10 for suffix dc=xyz,dc=com on master1
 
12. Verified that data for dc=xyz,dc=com was consistent across both masters

[root@ds ~]# ldapsearch -xLLL -D 'cn=Directory Manager' -w secret123 -h localhost -p 389 -b 'dc=xyz,dc=com' | wc -l
180025
[root@ds ~]# ldapsearch -xLLL -D 'cn=Directory Manager' -w secret123 -h localhost -p 1389 -b 'dc=xyz,dc=com' | wc -l
180025

Comment 7 errata-xmlrpc 2016-11-03 20:35:49 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2594.html