RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1629055 - ds-replcheck unreliable, showing false positives
Summary: ds-replcheck unreliable, showing false positives
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: 389-ds-base
Version: 7.5
Hardware: Unspecified
OS: Linux
high
unspecified
Target Milestone: rc
: 7.7
Assignee: mreynolds
QA Contact: RHDS QE
Marc Muehlfeld
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-09-14 18:13 UTC by Dave
Modified: 2023-03-24 14:14 UTC (History)
15 users (show)

Fixed In Version: 389-ds-base-1.3.9.1-7.el7
Doc Type: Bug Fix
Doc Text:
.The `ds-replcheck` utility no longer incorrectly reports non-matching tombstone entries on replicas Previously, if an administrator ran the `ds-replcheck` utility on different Directory Server replicas with tombstones present, `ds-replcheck` reported that one of the replicas was missing the tombstone entries. It is expected that tombstone entries do not match on each replica. With this update, `ds-replcheck` no longer searches for tombstone entries. As a result, the utility does not report missing tombstone entries as a problem.
Clone Of:
Environment:
Last Closed: 2019-08-06 12:58:51 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
Modified version to skip tombstones from missing entries report (49.69 KB, text/x-python)
2018-09-18 14:48 UTC, mreynolds 		no flags Details
New ds-replcheck (60.56 KB, text/x-python)
2019-01-04 16:37 UTC, mreynolds 		no flags Details
RHEL 7.7 ds-replcheck (python2) (51.77 KB, text/x-python)
2019-05-13 14:36 UTC, mreynolds 		no flags Details
ds-replcheck output from RHEL 7.6 version showing additional detail, as opposed to the 7.7 version output posted below (8.82 KB, application/x-xz)
2019-05-28 17:59 UTC, Dave 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:2152 0 None None None 2019-08-06 12:59:06 UTC

Description Dave 2018-09-14 18:13:14 UTC 
Description of problem:
this is in an IdM server env, we have 6 servers across 3 data centers
periodic runs of ds-replcheck show issues, when all server data is verified correct by running an ldapsearch on all servers and comparing the data. At this point, ds-replcheck is not a reliable tool the customer can use to verify ldap replication consistency

Version-Release number of selected component (if applicable):
kernel-3.10.0-862.9.1.el7.x86_64
ipa-server-4.5.4-10.el7_5.3.x86_64
389-ds-base-1.3.7.5-24.el7_5.x86_64

How reproducible:
issue shows up on a regular basis, more details in case as to which data is showing inconsistently

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
this is for a Red Hat Public Sector customer
troubleshooting and additional info/details in case https://access.redhat.com/support/cases/#/case/02150572
Comment 2 mreynolds 2018-09-14 19:01:57 UTC 
From what I can tell the issue is that tombstones are showing up in the report as missing entries.  Correct?  This should have been clearly stated in the bug, but pointing to the case notes to get the details is not appropriate IMO.

Anyway, if that is the issue, it is probably is a false positive as already stated.  We were actually going to remove tombstones from the missing entry report, but we weren't sure if it was going to be a problem or not.  Well it's apparently a problem so we will work on adding a new option to ignore them.
Comment 5 Dave 2018-09-17 19:42:26 UTC 
(In reply to mreynolds from comment #2)
> From what I can tell the issue is that tombstones are showing up in the
> report as missing entries.  Correct?  This should have been clearly stated
> in the bug, but pointing to the case notes to get the details is not
> appropriate IMO.
> 
> Anyway, if that is the issue, it is probably is a false positive as already
> stated.  We were actually going to remove tombstones from the missing entry
> report, but we weren't sure if it was going to be a problem or not.  Well
> it's apparently a problem so we will work on adding a new option to ignore
> them.

The customer was asked to run it with the following exclude options:

# ds-replcheck -D "cn=directory manager" -W -m ldap://ssc-prd-ipa-099 -r ldap://cl-rhm-0252 -b dc=masked,dc=domain --ignore memberof,idnssoaserial,entryusn,krblastsuccessfulauth,krblastfailedauth,krbloginfailedcount,nsuniqueid

but ds-replcheck is still showing periodic issues

should excluding nsuniqueid be ignoring tombstone entries? 
however, this did not resolve the issue for the customer
Comment 6 mreynolds 2018-09-17 19:50:55 UTC 
(In reply to Dave from comment #5)
> 
> The customer was asked to run it with the following exclude options:
> 
> # ds-replcheck -D "cn=directory manager" -W -m ldap://ssc-prd-ipa-099 -r
> ldap://cl-rhm-0252 -b dc=masked,dc=domain --ignore
> memberof,idnssoaserial,entryusn,krblastsuccessfulauth,krblastfailedauth,
> krbloginfailedcount,nsuniqueid
> 
> but ds-replcheck is still showing periodic issues
> 
> should excluding nsuniqueid be ignoring tombstone entries? 
> however, this did not resolve the issue for the customer

Yeah that won't work.  The "exclude/ignore attributes" only applies when checking the difference between two entries.  It does not impact the "Missing Entries" report.
Comment 7 mreynolds 2018-09-17 21:56:34 UTC 
Something else I want to mention...

Replication, when under load, is never going to be in-sync at any given moment.  So there are going to be times where the tool reports there are differences (because at that moment there are).  

To help avoid this there is a lag time you can adjust:

https://www.port389.org/docs/389ds/design/repl-diff-tool-design.html#usage

The default is 5 minutes, meaning anything out of sync that's within 5 minutes is ignored, but this might need to be increased depending on the customer's load.  Just an FYI.

Tomorrow I will prepare a test version of the script (that ignores missing tombstones).  If the customer could test this to make sure it works for the them I would appreciate it.
Comment 10 Dave 2018-09-18 14:34:50 UTC 
(In reply to mreynolds from comment #7)
> Tomorrow I will prepare a test version of the script (that ignores missing
> tombstones).  If the customer could test this to make sure it works for the
> them I would appreciate it.

yes yes, we/they could definitely test! :)
Comment 11 mreynolds 2018-09-18 14:48:05 UTC 
Created attachment 1484404 [details]
Modified version to skip tombstones from missing entries report
Comment 12 Dave 2018-09-18 19:45:20 UTC 
(In reply to mreynolds from comment #11)
> Created attachment 1484404 [details]
> Modified version to skip tombstones from missing entries report

testing!!
Comment 13 Dave 2018-09-25 12:56:00 UTC 
output identical for both versions

one note, we only have 7.5, this seems to be for 7.6? as it's set to python3, which isn't avail on 7.5. however, it does not error when pointed to the 7.5 python (2.7.5).

also tried the suggested screwing with lag time (btw, the man page doesn't mention a default value or if it's in seconds or minutes/etc, but the ds10 docs do), and haven't found a value that makes a difference (yet)

Missing Entries
=====================================================

  Entries missing on Replica:
   - nsuniqueid=0675aa01-b8eb11e8-95b0b696-12ca8a4f,dnaHostname=cl-rhm-0251.is.cl.ssa.gov+dnaPortNum=389,cn=posix-ids,cn=dna,cn=ipa,cn=etc,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Sat Sep 15 14:27:21 2018)
   - nsuniqueid=b9460801-b8eb11e8-8735cdda-f688984a,dnaHostname=cl-rhm-0252.is.cl.ssa.gov+dnaPortNum=389,cn=posix-ids,cn=dna,cn=ipa,cn=etc,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Sat Sep 15 14:32:26 2018)
   - nsuniqueid=4a47eb16-ba7411e8-95b0b696-12ca8a4f,dnaHostname=cl-rhm-0251.is.cl.ssa.gov+dnaPortNum=389,cn=posix-ids,cn=dna,cn=ipa,cn=etc,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Mon Sep 17 13:22:36 2018)
   - nsuniqueid=6f3c5e16-ba7411e8-8735cdda-f688984a,dnaHostname=cl-rhm-0252.is.cl.ssa.gov+dnaPortNum=389,cn=posix-ids,cn=dna,cn=ipa,cn=etc,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Mon Sep 17 13:23:38 2018)
Comment 14 Dave 2018-09-25 13:02:18 UTC 
(even set lag to 10 hours - 6000)
Comment 15 mreynolds 2018-09-25 13:14:18 UTC 
I'm assuming this is from the "online" mode.  I do see where it is picking it up from, so I need to do another revision.  But, do they see the same missing entries in offline mode (comparing two ldifs from "db2ldif -r")?
Comment 16 Dave 2018-10-05 13:11:38 UTC 
ok, it's doing it this morning, so I can finally get you some output/comparison

online:

# ds-replcheck -D "cn=directory manager" -W -m ldap://ssc-prd-ipa-100 -r ldap://cl-rhm-0252 -b dc=is,dc=cl,dc=ssa,dc=gov --ignore memberof,idnssoaserial,krblastsuccessfulauth,krblastfailedauth,krbloginfailedcount

Performing online report...
Connecting to servers...
Gathering Master's RUV...
Gathering Replica's RUV...
Start searching and comparing...
Preparing final report...
================================================================================
         Replication Synchronization Report  (Fri Oct  5 09:02:22 2018)
================================================================================


Database RUV's
=====================================================

Master RUV:
  {replica 11 ldap://nsc-prd-ipa-099.is.cl.ssa.gov:389} 5aec66900000000b0000 5bb75f1a0028000b0000
  {replica 13 ldap://nsc-prd-ipa-100.is.cl.ssa.gov:389} 5aec69780000000d0000 5bb75f1a002c000d0000
  {replica 3 ldap://cl-rhm-0251.is.cl.ssa.gov:389} 5aec5130000000030000 5bb75f1a002700030000
  {replica 4 ldap://ssc-prd-ipa-099.is.cl.ssa.gov:389} 5aec512b000100040000 5bb7608a000400040000
  {replica 7 ldap://cl-rhm-0252.is.cl.ssa.gov:389} 5aec540d000000070000 5bb7611c000400070000
  {replica 9 ldap://ssc-prd-ipa-100.is.cl.ssa.gov:389} 5aec63cd000000090000 5bb75f1a000100090000
  {replicageneration} 5aec512b000000040000

Replica RUV:
  {replica 11 ldap://nsc-prd-ipa-099.is.cl.ssa.gov:389} 5aec66900000000b0000 5bb75f1a0028000b0000
  {replica 13 ldap://nsc-prd-ipa-100.is.cl.ssa.gov:389} 5aec69780000000d0000 5bb75f1a002c000d0000
  {replica 3 ldap://cl-rhm-0251.is.cl.ssa.gov:389} 5aec5130000000030000 5bb75f1a002700030000
  {replica 4 ldap://ssc-prd-ipa-099.is.cl.ssa.gov:389} 5aec512b000100040000 5bb7608a000400040000
  {replica 7 ldap://cl-rhm-0252.is.cl.ssa.gov:389} 5aec540d000000070000 5bb7611c000400070000
  {replica 9 ldap://ssc-prd-ipa-100.is.cl.ssa.gov:389} 5aec63cd000000090000 5bb75f1a000100090000
  {replicageneration} 5aec512b000000040000


Entry Counts
=====================================================

Master:  1546
Replica: 1545


Tombstones
=====================================================

Master:  15
Replica: 15


Conflict Entries
=====================================================

Master Conflict Entries:  2
Replica Conflict Entries: 2


Missing Entries
=====================================================

  Entries missing on Replica:
   - nsuniqueid=e4553968-44c011e8-bc92cfbd-26e5231b,cn=Host Enrollment,cn=privileges,cn=pbac,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Apr 20 18:33:10 2018)



offline (weird, seems to find more issues this way):

# ds-replcheck -M /tmp/userRoot-ssc-prd-ipa-099.ldif -R /tmp/userRoot-cl-rhm-0252.ldif -b "dc=is,dc=cl,dc=ssa,dc=gov" --ignore memberof,idnssoaserial,krblastsuccessfulauth,krblastfailedauth,krbloginfailedcount
Performing offline report...
Gathering all the DN's...
Gathering the database RUV's...
Comparing Master to Replica...
Comparing Replica to Master...
Preparing report...
================================================================================
         Replication Synchronization Report  (Fri Oct  5 09:03:37 2018)
================================================================================


Database RUV's
=====================================================

Master RUV:
  {replica 11 ldap://nsc-prd-ipa-099.is.cl.ssa.gov:389} 5aec66900000000b0000 5bb75f1a0028000b0000
  {replica 13 ldap://nsc-prd-ipa-100.is.cl.ssa.gov:389} 5aec69780000000d0000 5bb75f1a002c000d0000
  {replica 3 ldap://cl-rhm-0251.is.cl.ssa.gov:389} 5aec5130000000030000 5bb75f1a002700030000
  {replica 4 ldap://ssc-prd-ipa-099.is.cl.ssa.gov:389} 5aec512b000100040000 5bb75ecb002000040000
  {replica 7 ldap://cl-rhm-0252.is.cl.ssa.gov:389} 5aec540d000000070000 5bb75f1a002b00070000
  {replica 9 ldap://ssc-prd-ipa-100.is.cl.ssa.gov:389} 5aec63cd000000090000 5bb75f1a000100090000
  {replicageneration} 5aec512b000000040000

Replica RUV:
  {replica 11 ldap://nsc-prd-ipa-099.is.cl.ssa.gov:389} 5aec66900000000b0000 5bb75f1a0028000b0000
  {replica 13 ldap://nsc-prd-ipa-100.is.cl.ssa.gov:389} 5aec69780000000d0000 5bb75f1a002c000d0000
  {replica 3 ldap://cl-rhm-0251.is.cl.ssa.gov:389} 5aec5130000000030000 5bb75f1a002700030000
  {replica 4 ldap://ssc-prd-ipa-099.is.cl.ssa.gov:389} 5aec512b000100040000 5bb7608a000400040000
  {replica 7 ldap://cl-rhm-0252.is.cl.ssa.gov:389} 5aec540d000000070000 5bb75f1a002b00070000
  {replica 9 ldap://ssc-prd-ipa-100.is.cl.ssa.gov:389} 5aec63cd000000090000 5bb75f1a000100090000
  {replicageneration} 5aec512b000000040000


Entry Counts
=====================================================

Master:  1376
Replica: 1376


Tombstones
=====================================================

Master:  16
Replica: 16


Conflict Entries
=====================================================

Master Conflict Entries:  2
Replica Conflict Entries: 2


Missing Entries
=====================================================

  Entries missing on Replica:
   - nsuniqueid=e4553968-44c011e8-bc92cfbd-26e5231b,cn=host enrollment,cn=privileges,cn=pbac,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Apr 20 18:33:10 2018)

  Entries missing on Master:
   - nsuniqueid=bea35296-b8eb11e8-9babcfbd-26e5231b,dnahostname=ssc-prd-ipa-099.is.cl.ssa.gov+dnaportnum=389,cn=posix-ids,cn=dna,cn=ipa,cn=etc,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Replica at: Sat Sep 15 14:32:38 2018)



Entry Inconsistencies
=====================================================

cn=it security specialist,cn=roles,cn=accounts,dc=is,dc=cl,dc=ssa,dc=gov
------------------------------------------------------------------------
 - Attribute 'description' is different:
      Master:
        - State Info: description;vdcsn-5bae4166001b00040001;vucsn-5bae4181002100040000: Netgroups Administrators, HBAC Administrator, Sudo Administrator: Responsible for managing security policy such as host-based access controls, sudo rules
        - Date:       Fri Sep 28 10:58:09 2018

        - State Info: description;vucsn-5bae4166001b00040000;vdcsn-5bae4181002100040001;deleted: Netgroups Administrators, HBAC Administrator, Sudo Administrator: Responsible for managing security policy such as host-based access controls, sudo rulesx
        - Date:       Fri Sep 28 10:58:09 2018

      Replica:
        - State Info: description;vucsn-5bae4181002100040000: Netgroups Administrators, HBAC Administrator, Sudo Administrator: Responsible for managing security policy such ashost-based access controls, sudo rules
        - Date:       Fri Sep 28 10:58:09 2018

        - State Info: description;vucsn-5bae4166001b00040000;vdcsn-5bae4181002100040001;deleted: Netgroups Administrators, HBAC Administrator, Sudo Administrator: Responsible for managing security policy such as host-based access controls, sudo rulesx
        - Date:       Fri Sep 28 10:58:09 2018


dnahostname=ssc-prd-ipa-099.is.cl.ssa.gov+dnaportnum=389,cn=posix-ids,cn=dna,cn=ipa,cn=etc,dc=is,dc=cl,dc=ssa,dc=gov
--------------------------------------------------------------------------------------------------------------------
 - Attribute 'nsuniqueid' is different:
      Master:
        - Origin value: bea35296-b8eb11e8-9babcfbd-26e5231b

      Replica:
        - Origin value: 4999bc01-c89e11e8-9babcfbd-26e5231b
Comment 17 Dave 2019-01-04 16:33:18 UTC 
is there a newer ds-replcheck we can test? the last provided/attached did not solve/fix the customer's issue
Comment 18 mreynolds 2019-01-04 16:37:09 UTC 
Created attachment 1518440 [details]
New ds-replcheck

Try this one Dave.  Note -the usage changed quite a bit(it's a lot nicer), so run ds-replcheck --help first and try again...
Comment 20 Akshay Adhikari 2019-05-07 13:54:56 UTC 
The version of ds-replcheck in the build: 389-ds-base-1.3.9.1-5.el7.x86_64 is 1.4 whereas in the attached file it is 2.0. Also, the changes could not be seen in /usr/bin/ds-replcheck as compared with the attachment. Therefore marking it as FAILED_QA.
Comment 22 Dave 2019-05-07 16:04:55 UTC 
(In reply to mreynolds from comment #18)
> Created attachment 1518440 [details]
> New ds-replcheck
> 
> Try this one Dave.  Note -the usage changed quite a bit(it's a lot nicer),
> so run ds-replcheck --help first and try again...

/usr/bin/python3 not available on RHEL 7.x (tested on 7.6)

pointing to /usr/bin/python (python2) does not work either, as there are missing modules apparently specific to python3

using rhscl rh-python36-python (rh-python36-runtime) does not work either as there is no ldap module available

does not work for RHEL 7.x, assuming this will also not work for targeted RHEL 7.7 release

unable to test on RHEL 7.x (7.6 attempted)
Comment 23 mreynolds 2019-05-13 14:33:24 UTC 
Dave,

looks like these changes were added to RHEL 7.7 (python2 version).  What was the last version the customer tested?

Anyway I am attaching what is in latest build for 7.7.  Since RHEL 7.7 is wrapping up really soon it would be great if we could get it verified. 

Thanks,
Mark
Comment 24 mreynolds 2019-05-13 14:36:54 UTC 
Created attachment 1567987 [details]
RHEL 7.7 ds-replcheck (python2)
Comment 26 Dave 2019-05-13 17:42:23 UTC 
(In reply to mreynolds from comment #24)
> Created attachment 1567987 [details]
> RHEL 7.7 ds-replcheck (python2)

This one runs (on 7.6) and looks clean, does not exhibit the previous tombstone issues

Looks good, customer has reviewed the new output
Comment 29 Akshay Adhikari 2019-05-21 09:20:09 UTC 
Build Tested: 389-ds-base-1.3.9.1-7.el7.x86_64

1) Create 2 Master A and B but do NOT create agreements, etc.

2) Add and Delete a user on Master A

ldapadd -p 39001 -h localhost -D "cn=Directory Manager" -w password << EOF
dn: uid=test-user,ou=People,dc=example,dc=com
changetype: add
uid: test-user
objectClass: top
objectClass: account
objectClass: posixaccount
objectClass: inetOrgPerson
objectClass: person
objectClass: inetUser
objectClass: organizationalPerson
uidNumber: 1001
gidNumber: 1001
sn: surname
homeDirectory: /home/test-user
cn: common name
EOF
adding new entry "uid=test-user,ou=People,dc=example,dc=com"

ldapdelete -p 39001 -h localhost -D "cn=Directory Manager" -w password uid=test-user,ou=People,dc=example,dc=com

4) Run ds-replcheck and verify there are NO complaints about missing entries/tombstones

[root@master ~]# ds-replcheck -v -D "cn=directory manager" -w password -m ldap://`hostname`:39001 -r ldap://`hostname`:39002 -b dc=example,dc=com -l 1
Performing online report...
Connecting to servers...
Validating suffix ...
Gathering Master's RUV...
Gathering Replica's RUV...
Start searching and comparing...
Preparing final report...
================================================================================
         Replication Synchronization Report  (Wed May 15 09:26:29 2019)
================================================================================


Database RUV's
=====================================================

Master RUV:
  {replica 1 ldap://web9.testrelm.test:39001} 5cdc0edf000100010000 5cdc1354000000010000
  {replica 2 ldap://web9.testrelm.test:39002} 5cdc0ee8000100020000 5cdc0ee8000100020000
  {replicageneration} 5cdc0edf000000010000

Replica RUV:
  {replica 1 ldap://web9.testrelm.test:39001} 5cdc0edf000100010000 5cdc0ee5000200010000
  {replica 2 ldap://web9.testrelm.test:39002} 5cdc0ee8000100020000 5cdc0ee8000100020000
  {replicageneration} 5cdc0edf000000010000


Entry Counts
=====================================================

Master:  15
Replica: 14


Tombstones
=====================================================

Master:  1
Replica: 0

Marking it as VERIFIED.
Comment 30 Dave 2019-05-28 17:55:16 UTC 
(In reply to mreynolds from comment #24)
> Created attachment 1567987 [details]
> RHEL 7.7 ds-replcheck (python2)

unfortunately we are having some issues with the new version this morning (this is probably a candidate for an alternate BZ)

in the previous version (on 7.6), when there was a problem, it summarized the error condition as "Entry Inconsistencies", which we are no longer seeing or able to verify with under this new version (as opposed to the normal 'Good' message which we are still seeing "No differences between Master and Replica")

additionally, the bigger issue is that in this failed/issue state, there is not longer any mention of the inconsistency

New version output:

Performing online report...
Connecting to servers...
Validating suffix ...
Gathering Master's RUV...
Gathering Replica's RUV...
Start searching and comparing...
Preparing final report...
================================================================================
         Replication Synchronization Report  (Tue May 28 13:11:41 2019)
================================================================================


Database RUV's
=====================================================

Master RUV:
  {replica 11 ldap://nsc-prd-ipa-099.is.cl.ssa.gov:389} 5aec66900000000b0000 5ce6b50000d1000b0000
  {replica 13 ldap://nsc-prd-ipa-100.is.cl.ssa.gov:389} 5aec69780000000d0000 5ce6b50000d7000d0000
  {replica 3 ldap://cl-rhm-0251.is.cl.ssa.gov:389} 5aec5130000000030000 5ced63b3000000030000
  {replica 4 ldap://ssc-prd-ipa-099.is.cl.ssa.gov:389} 5aec512b000100040000 5cea445f002a00040000
  {replica 7 ldap://cl-rhm-0252.is.cl.ssa.gov:389} 5aec540d000000070000 5ce82677001200070000
  {replica 9 ldap://ssc-prd-ipa-100.is.cl.ssa.gov:389} 5aec63cd000000090000 5ce6b852000000090000
  {replicageneration} 5aec512b000000040000

Replica RUV:
  {replica 11 ldap://nsc-prd-ipa-099.is.cl.ssa.gov:389} 5aec66900000000b0000 5ce6b50000d1000b0000
  {replica 13 ldap://nsc-prd-ipa-100.is.cl.ssa.gov:389} 5aec69780000000d0000 5ce6b50000d7000d0000
  {replica 3 ldap://cl-rhm-0251.is.cl.ssa.gov:389} 5aec5130000000030000 5ced6512000000030000
  {replica 4 ldap://ssc-prd-ipa-099.is.cl.ssa.gov:389} 5aec512b000100040000 5ce6b50000cb00040000
  {replica 7 ldap://cl-rhm-0252.is.cl.ssa.gov:389} 5aec540d000000070000 5ce82677001200070000
  {replica 9 ldap://ssc-prd-ipa-100.is.cl.ssa.gov:389} 5aec63cd000000090000 5ce6b852000000090000
  {replicageneration} 5aec512b000000040000


Entry Counts
=====================================================

Master:  4608
Replica: 4603


Tombstones
=====================================================

Master:  86
Replica: 81


Conflict Entries
=====================================================

Master Conflict Entries:  2
Replica Conflict Entries: 2



Old version output:

being attached as a file since it is approx 11k lines
Comment 31 Dave 2019-05-28 17:59:37 UTC 
Created attachment 1574437 [details]
ds-replcheck output from RHEL 7.6 version showing additional detail, as opposed to the 7.7 version output posted below
Comment 32 Dave 2019-05-28 18:12:06 UTC 
(In reply to mreynolds from comment #24)
> Created attachment 1567987 [details]
> RHEL 7.7 ds-replcheck (python2)

running an alternate data check tool, shows there is no data inconsistency, but ds-replcheck IS reporting an issue

we're under the impression that this new ds-replcheck does not resolve the issue for the customer and is not yet usable/reliable
Comment 33 mreynolds 2019-05-28 18:26:14 UTC 
(In reply to Dave from comment #32)
> (In reply to mreynolds from comment #24)
> > Created attachment 1567987 [details]
> > RHEL 7.7 ds-replcheck (python2)
> 
> running an alternate data check tool, shows there is no data inconsistency,
> but ds-replcheck IS reporting an issue
> 
> we're under the impression that this new ds-replcheck does not resolve the
> issue for the customer and is not yet usable/reliable

So one of the changes in the latest version of ds-replcheck is that it now sorts multivalued attribute values.  So before it would say there were inconsistencies if multivalued attributes were in a different orders on different servers, but it was a false positive because the entries were actually in sync.  Looking at the old report:


 - Attribute 'nisnetgrouptriple' is different:
      Master: 
        - Origin value: (nsc-p-iscope-prdcol2.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
        - Origin value: (nsc-p-iscope-prdcol4.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
        - Origin value: (nsc-p-iscope-prdcol3.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
        - Origin value: (nsc-p-iscope-prdcol6.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
        - Origin value: (nsc-p-iscope-prdcol8.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
        - Origin value: (nsc-p-iscope-prdwv.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
        - Origin value: (nsc-p-iscope-prddb.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
        - Origin value: (nsc-p-iscope-prdcol7.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
        - Origin value: (nsc-p-iscope-prdcol5.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
        - Origin value: (nsc-p-iscope-prdmom.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
        - Origin value: (nsc-p-iscope-prdcol1.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)

      Replica: 
        - Origin value: (nsc-p-iscope-prdcol2.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
        - Origin value: (nsc-p-iscope-prdcol3.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
        - Origin value: (nsc-p-iscope-prdcol4.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
        - Origin value: (nsc-p-iscope-prdcol8.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
        - Origin value: (nsc-p-iscope-prdcol6.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
        - Origin value: (nsc-p-iscope-prdwv.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
        - Origin value: (nsc-p-iscope-prddb.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
        - Origin value: (nsc-p-iscope-prdcol7.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
        - Origin value: (nsc-p-iscope-prdcol5.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
        - Origin value: (nsc-p-iscope-prdmom.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
        - Origin value: (nsc-p-iscope-prdcol1.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)


If you look closely these are all the same values, just in a slightly different order.  So this was a false positive.  This was fixed, so it no longer complains about these "inconsistencies".  So it appears the tool is now running correctly.
Comment 34 Dave 2019-05-28 18:29:12 UTC 
(In reply to Dave from comment #30)
> (In reply to mreynolds from comment #24)
> > Created attachment 1567987 [details]
> > RHEL 7.7 ds-replcheck (python2)


> Performing online report...
> Connecting to servers...
> Validating suffix ...
> Gathering Master's RUV...
> Gathering Replica's RUV...
> Start searching and comparing...
> Preparing final report...
> =============================================================================
> ===
>          Replication Synchronization Report  (Tue May 28 13:11:41 2019)
> =============================================================================
> ===
> 
> 
> Database RUV's
> =====================================================
> 
> Master RUV:
>   {replica 11 ldap://nsc-prd-ipa-099.is.cl.ssa.gov:389} 5aec66900000000b0000
> 5ce6b50000d1000b0000
>   {replica 13 ldap://nsc-prd-ipa-100.is.cl.ssa.gov:389} 5aec69780000000d0000
> 5ce6b50000d7000d0000
>   {replica 3 ldap://cl-rhm-0251.is.cl.ssa.gov:389} 5aec5130000000030000
> 5ced63b3000000030000
>   {replica 4 ldap://ssc-prd-ipa-099.is.cl.ssa.gov:389} 5aec512b000100040000
> 5cea445f002a00040000
>   {replica 7 ldap://cl-rhm-0252.is.cl.ssa.gov:389} 5aec540d000000070000
> 5ce82677001200070000
>   {replica 9 ldap://ssc-prd-ipa-100.is.cl.ssa.gov:389} 5aec63cd000000090000
> 5ce6b852000000090000
>   {replicageneration} 5aec512b000000040000
> 
> Replica RUV:
>   {replica 11 ldap://nsc-prd-ipa-099.is.cl.ssa.gov:389} 5aec66900000000b0000
> 5ce6b50000d1000b0000
>   {replica 13 ldap://nsc-prd-ipa-100.is.cl.ssa.gov:389} 5aec69780000000d0000
> 5ce6b50000d7000d0000
>   {replica 3 ldap://cl-rhm-0251.is.cl.ssa.gov:389} 5aec5130000000030000
> 5ced6512000000030000
>   {replica 4 ldap://ssc-prd-ipa-099.is.cl.ssa.gov:389} 5aec512b000100040000
> 5ce6b50000cb00040000
>   {replica 7 ldap://cl-rhm-0252.is.cl.ssa.gov:389} 5aec540d000000070000
> 5ce82677001200070000
>   {replica 9 ldap://ssc-prd-ipa-100.is.cl.ssa.gov:389} 5aec63cd000000090000
> 5ce6b852000000090000
>   {replicageneration} 5aec512b000000040000
> 
> 
> Entry Counts
> =====================================================
> 
> Master:  4608
> Replica: 4603
> 
> 
> Tombstones
> =====================================================
> 
> Master:  86
> Replica: 81
> 
> 
> Conflict Entries
> =====================================================
> 
> Master Conflict Entries:  2
> Replica Conflict Entries: 2
> 
> 
> 
> Old version output:
> 
> being attached as a file since it is approx 11k lines



there is no Result section, as in the previous version.. we would normally see something like this:


Result
=====================================================

Entry Inconsistencies
Comment 35 Dave 2019-05-28 18:31:38 UTC 
(In reply to mreynolds from comment #33)
> (In reply to Dave from comment #32)
> > (In reply to mreynolds from comment #24)
> > > Created attachment 1567987 [details]
> > > RHEL 7.7 ds-replcheck (python2)
> > 
> > running an alternate data check tool, shows there is no data inconsistency,
> > but ds-replcheck IS reporting an issue
> > 
> > we're under the impression that this new ds-replcheck does not resolve the
> > issue for the customer and is not yet usable/reliable
> 
> So one of the changes in the latest version of ds-replcheck is that it now
> sorts multivalued attribute values.  So before it would say there were
> inconsistencies if multivalued attributes were in a different orders on
> different servers, but it was a false positive because the entries were
> actually in sync.  Looking at the old report:
> 
> 
>  - Attribute 'nisnetgrouptriple' is different:
>       Master: 
>         - Origin value:
> (nsc-p-iscope-prdcol2.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
>         - Origin value:
> (nsc-p-iscope-prdcol4.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
>         - Origin value:
> (nsc-p-iscope-prdcol3.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
>         - Origin value:
> (nsc-p-iscope-prdcol6.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
>         - Origin value:
> (nsc-p-iscope-prdcol8.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
>         - Origin value:
> (nsc-p-iscope-prdwv.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
>         - Origin value:
> (nsc-p-iscope-prddb.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
>         - Origin value:
> (nsc-p-iscope-prdcol7.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
>         - Origin value:
> (nsc-p-iscope-prdcol5.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
>         - Origin value:
> (nsc-p-iscope-prdmom.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
>         - Origin value:
> (nsc-p-iscope-prdcol1.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
> 
>       Replica: 
>         - Origin value:
> (nsc-p-iscope-prdcol2.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
>         - Origin value:
> (nsc-p-iscope-prdcol3.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
>         - Origin value:
> (nsc-p-iscope-prdcol4.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
>         - Origin value:
> (nsc-p-iscope-prdcol8.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
>         - Origin value:
> (nsc-p-iscope-prdcol6.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
>         - Origin value:
> (nsc-p-iscope-prdwv.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
>         - Origin value:
> (nsc-p-iscope-prddb.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
>         - Origin value:
> (nsc-p-iscope-prdcol7.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
>         - Origin value:
> (nsc-p-iscope-prdcol5.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
>         - Origin value:
> (nsc-p-iscope-prdmom.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
>         - Origin value:
> (nsc-p-iscope-prdcol1.aci.is.cl.ssa.gov,-,is.cl.ssa.gov)
> 
> 
> If you look closely these are all the same values, just in a slightly
> different order.  So this was a false positive.  This was fixed, so it no
> longer complains about these "inconsistencies".  So it appears the tool is
> now running correctly.

if it was running correctly, we would expect something like:


Result
=====================================================

No differences between Master and Replica
Comment 36 mreynolds 2019-05-28 18:54:50 UTC 
Sorry I'm really confused.  From what I can tell from comment 30 everything is running correctly, and the old output verifies that they were previously seeing false positives.  So the tool appears to be running correctly.  Is the problem now that there is no "Result" summary message?
Comment 37 Dave 2019-05-28 19:08:58 UTC 
(In reply to mreynolds from comment #36)
> Sorry I'm really confused.  From what I can tell from comment 30 everything
> is running correctly, and the old output verifies that they were previously
> seeing false positives.  So the tool appears to be running correctly.  Is
> the problem now that there is no "Result" summary message?

The new version had been running as expected over the last couple weeks of testing, until this afternoon. We had been looking for this Result section, and could verify it was good when we saw:

No differences between Master and Replica

*Until today, when we did not see this line, noticed there was no Result section at all, 

*and these count issues:

Entry Counts
=====================================================
 
Master:  4608
Replica: 4603


Tombstones
=====================================================
 
Master:  86
Replica: 81


*The previous version would show some data when the counts were not equal, and we saw no data to try and dump/compare between servers.

Then with what you posted about it being a sorting issue that is being falsely reported, this new version seems to still be failing on sorting "multivalued attribute values" ??

We saw 3 (*) issues when it failed to report a "good" status
Comment 38 mreynolds 2019-05-28 19:53:06 UTC 
(In reply to Dave from comment #37)
> (In reply to mreynolds from comment #36)
> > Sorry I'm really confused.  From what I can tell from comment 30 everything
> > is running correctly, and the old output verifies that they were previously
> > seeing false positives.  So the tool appears to be running correctly.  Is
> > the problem now that there is no "Result" summary message?
> 
> The new version had been running as expected over the last couple weeks of
> testing, until this afternoon. We had been looking for this Result section,
> and could verify it was good when we saw:
> 
> No differences between Master and Replica
> 
> *Until today, when we did not see this line, noticed there was no Result
> section at all, 
> 
> *and these count issues:
> 
> Entry Counts
> =====================================================
>  
> Master:  4608
> Replica: 4603
> 
> 
> Tombstones
> =====================================================
>  
> Master:  86
> Replica: 81
> 
> 
> *The previous version would show some data when the counts were not equal,
> and we saw no data to try and dump/compare between servers.
> 
> Then with what you posted about it being a sorting issue that is being
> falsely reported, this new version seems to still be failing on sorting
> "multivalued attribute values" ??
> 
> We saw 3 (*) issues when it failed to report a "good" status

These counts are not expected to be equal.  Tombstones can and will vary (they are not expected to ever be in sync), and the entry count is potentially always in flux.  We also ignore missing tombstones for the "missing entry" report.  So if the numbers are off and any missing entries are NOT tombstones it will report on them.  If you don't get a "missing entry" report then the counts don't mean a thing.  Really the counts are just informational (maybe they should be removed if its causing confusion?)

As for the "Result" line it is dictated in thecode as follows:

    if missing_report == "" and len(diff_report) == 0 and m_count == r_count:
        final_report += ('\nResult\n')
        final_report += ('=====================================================\n\n')
        final_report += ('No differences between Master and Replica\n')


In this case m_count(the Master count) is different than r_count(the Replica count) and we don't get our "Result" summary.  There is definitely a flaw with this algorithm since it is okay for m_count and r_count to be different.  The other issue is that we only write this "Result" line under these impossibly pristine conditions.  This needs to be made more robust, better reporting of the results, and always write a Result summary.  So we need a new bug to improve the Result summary.
Comment 39 Dave 2019-05-29 14:23:01 UTC 
(In reply to mreynolds from comment #38)

...

> In this case m_count(the Master count) is different than r_count(the Replica
> count) and we don't get our "Result" summary.  There is definitely a flaw
> with this algorithm since it is okay for m_count and r_count to be
> different.  The other issue is that we only write this "Result" line under
> these impossibly pristine conditions.  This needs to be made more robust,
> better reporting of the results, and always write a Result summary.  So we
> need a new bug to improve the Result summary.

cool, sounds good.. the Result section is really what we're looking for to get the status.
Could you mention the new BZ for this missing Result issue, and/or would you like me to open one?
Comment 40 mreynolds 2019-05-29 14:43:44 UTC 
(In reply to Dave from comment #39)
> (In reply to mreynolds from comment #38)
> 
> ...
> 
> > In this case m_count(the Master count) is different than r_count(the Replica
> > count) and we don't get our "Result" summary.  There is definitely a flaw
> > with this algorithm since it is okay for m_count and r_count to be
> > different.  The other issue is that we only write this "Result" line under
> > these impossibly pristine conditions.  This needs to be made more robust,
> > better reporting of the results, and always write a Result summary.  So we
> > need a new bug to improve the Result summary.
> 
> cool, sounds good.. the Result section is really what we're looking for to
> get the status.
> Could you mention the new BZ for this missing Result issue, and/or would you
> like me to open one?

Done!

https://bugzilla.redhat.com/show_bug.cgi?id=1715091
Comment 44 errata-xmlrpc 2019-08-06 12:58:51 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2152
Comment 45 Dave 2019-09-06 18:18:02 UTC 
seems to still be mis-reporting? when we do data comparisons in a different manner (dumping ldap data to ldif text files), we do not see a data issue

one:

=== COMPARING ssc-prd-ipa-099 & cl-rhm-0252 ===

Performing online report...
Connecting to servers...
Validating suffix ...
Gathering Master's RUV...
Gathering Replica's RUV...
Start searching and comparing...
Preparing final report...
================================================================================
         Replication Synchronization Report  (Fri Aug 16 10:43:26 2019)
================================================================================


Database RUV's
=====================================================

Master RUV:
  {replica 11 ldap://nsc-prd-ipa-099.is.cl.ssa.gov:389} 5aec66900000000b0000 5d56c13a0045000b0000
  {replica 13 ldap://nsc-prd-ipa-100.is.cl.ssa.gov:389} 5aec69780000000d0000 5d56c1990000000d0000
  {replica 3 ldap://cl-rhm-0251.is.cl.ssa.gov:389} 5aec5130000000030000 5d55afd800a000030000
  {replica 4 ldap://ssc-prd-ipa-099.is.cl.ssa.gov:389} 5aec512b000100040000 5d56c13a006800040000
  {replica 7 ldap://cl-rhm-0252.is.cl.ssa.gov:389} 5aec540d000000070000 5d55afd8008c00070000
  {replica 9 ldap://ssc-prd-ipa-100.is.cl.ssa.gov:389} 5aec63cd000000090000 5d55a245006600090000
  {replicageneration} 5aec512b000000040000

Replica RUV:
  {replica 11 ldap://nsc-prd-ipa-099.is.cl.ssa.gov:389} 5aec66900000000b0000 5d56c13a0045000b0000
  {replica 13 ldap://nsc-prd-ipa-100.is.cl.ssa.gov:389} 5aec69780000000d0000 5d56c1990000000d0000
  {replica 3 ldap://cl-rhm-0251.is.cl.ssa.gov:389} 5aec5130000000030000 5d55afd800a000030000
  {replica 4 ldap://ssc-prd-ipa-099.is.cl.ssa.gov:389} 5aec512b000100040000 5d55c764000000040000
  {replica 7 ldap://cl-rhm-0252.is.cl.ssa.gov:389} 5aec540d000000070000 5d56c13a005c00070000
  {replica 9 ldap://ssc-prd-ipa-100.is.cl.ssa.gov:389} 5aec63cd000000090000 5d55a245006600090000
  {replicageneration} 5aec512b000000040000


Entry Counts
=====================================================

Master:  4993
Replica: 4979


Tombstones
=====================================================

Master:  22
Replica: 22


Conflict Entries
=====================================================

Master Conflict Entries:  2
Replica Conflict Entries: 2


Missing Entries
=====================================================

  Entries missing on Replica:
   - cn=test,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 26 18:51:16 2019)
   - cn=fredtest-sbx-app1,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 12 19:05:27 2019)
   - cn=test,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 26 18:51:16 2019)
   - cn=fredtest-sbx-app1,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 12 19:05:27 2019)
   - cn=test,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 26 18:51:16 2019)
   - cn=fredtest-sbx-app1,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 12 19:05:27 2019)
   - cn=test,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 26 18:51:16 2019)
   - cn=fredtest-sbx-app1,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 12 19:05:27 2019)
   - cn=test,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 26 18:51:16 2019)
   - cn=fredtest-sbx-app1,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 12 19:05:27 2019)
   - cn=test,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 26 18:51:16 2019)
   - cn=fredtest-sbx-app1,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 12 19:05:27 2019)
   - cn=test,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 26 18:51:16 2019)
   - cn=fredtest-sbx-app1,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 12 19:05:27 2019)

Result
=====================================================

There are replication differences between Master and Replica



Do we continue on this BZ, or open one anew?
Comment 46 mreynolds 2019-09-06 19:19:27 UTC 
Dave,

So the offline report says there are no problem, but the online does?  Did you verify if the differences reported in the online report are actually NOT different in the ldif files?  Or, is the offline report incorrectly reporting there there are no differences but there are differences in the ldif files?

There is another bug too, the missing entries report has duplicates.  Not sure how that is possible, but it needs to be fixed.

Either way this is a different issue, so we should open a new bug for it once it is determined which report is wrong and why.

Thanks,
Mark
Comment 49 Dave 2019-09-11 15:37:25 UTC 
(In reply to mreynolds from comment #46)

>> when we do data comparisons in a different manner (dumping ldap data to ldif text files), we do not see a data issue
> So the offline report says there are no problem, but the online does?

We normally do not use offline mode, since this is a production env and offline mode requires stopping directory server to run db2ldif, we have a different scripted method (though quite slow, relatively) just using ldapsearch to dump the data

A recent run (online, offline run follows this one)

Performing online report...
Connecting to servers...
Validating suffix ...
Gathering Master's RUV...
Gathering Replica's RUV...
Start searching and comparing...
Preparing final report...
================================================================================
         Replication Synchronization Report  (Tue Sep 10 14:34:11 2019)
================================================================================


Database RUV's
=====================================================

Master RUV:
  {replica 11 ldap://nsc-prd-ipa-099.is.cl.ssa.gov:389} 5aec66900000000b0000 5d77c25d0047000b0000
  {replica 13 ldap://nsc-prd-ipa-100.is.cl.ssa.gov:389} 5aec69780000000d0000 5d77773a0000000d0000
  {replica 3 ldap://cl-rhm-0251.is.cl.ssa.gov:389} 5aec5130000000030000 5d77df10004200030000
  {replica 4 ldap://ssc-prd-ipa-099.is.cl.ssa.gov:389} 5aec512b000100040000 5d77df10005d00040000
  {replica 7 ldap://cl-rhm-0252.is.cl.ssa.gov:389} 5aec540d000000070000 5d77df0f003200070000
  {replica 9 ldap://ssc-prd-ipa-100.is.cl.ssa.gov:389} 5aec63cd000000090000 5d767d1c000300090000
  {replicageneration} 5aec512b000000040000

Replica RUV:
  {replica 11 ldap://nsc-prd-ipa-099.is.cl.ssa.gov:389} 5aec66900000000b0000 5d77c25d0047000b0000
  {replica 13 ldap://nsc-prd-ipa-100.is.cl.ssa.gov:389} 5aec69780000000d0000 5d77773a0000000d0000
  {replica 3 ldap://cl-rhm-0251.is.cl.ssa.gov:389} 5aec5130000000030000 5d77df10004200030000
  {replica 4 ldap://ssc-prd-ipa-099.is.cl.ssa.gov:389} 5aec512b000100040000 5d769e6e001a00040000
  {replica 7 ldap://cl-rhm-0252.is.cl.ssa.gov:389} 5aec540d000000070000 5d77df10005000070000
  {replica 9 ldap://ssc-prd-ipa-100.is.cl.ssa.gov:389} 5aec63cd000000090000 5d767d1c000300090000
  {replicageneration} 5aec512b000000040000


Entry Counts
=====================================================

Master:  5856
Replica: 5832


Tombstones
=====================================================

Master:  8
Replica: 8


Conflict Entries
=====================================================

Master Conflict Entries:  2
Replica Conflict Entries: 2


Missing Entries
=====================================================

  Entries missing on Replica:
   - cn=ansible-prd-towertest,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Aug  9 19:56:02 2019)
   - cn=test,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 26 18:51:16 2019)
   - cn=fredtest-sbx-app1,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 12 19:05:27 2019)
   - cn=ansible-prd-towertest,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Aug  9 19:56:02 2019)
   - cn=test,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 26 18:51:16 2019)
   - cn=fredtest-sbx-app1,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 12 19:05:27 2019)
   - cn=ansible-prd-towertest,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Aug  9 19:56:02 2019)
   - cn=test,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 26 18:51:16 2019)
   - cn=fredtest-sbx-app1,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 12 19:05:27 2019)
   - cn=ansible-prd-towertest,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Aug  9 19:56:02 2019)
   - cn=test,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 26 18:51:16 2019)
   - cn=fredtest-sbx-app1,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 12 19:05:27 2019)
   - cn=ansible-prd-towertest,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Aug  9 19:56:02 2019)
   - cn=test,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 26 18:51:16 2019)
   - cn=fredtest-sbx-app1,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 12 19:05:27 2019)
   - cn=ansible-prd-towertest,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Aug  9 19:56:02 2019)
   - cn=test,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 26 18:51:16 2019)
   - cn=fredtest-sbx-app1,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 12 19:05:27 2019)
   - cn=ansible-prd-towertest,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Aug  9 19:56:02 2019)
   - cn=test,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 26 18:51:16 2019)
   - cn=fredtest-sbx-app1,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 12 19:05:27 2019)
   - cn=ansible-prd-towertest,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Aug  9 19:56:02 2019)
   - cn=test,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 26 18:51:16 2019)
   - cn=fredtest-sbx-app1,cn=ng,cn=compat,dc=is,dc=cl,dc=ssa,dc=gov  (Created on Master at: Fri Jul 12 19:05:27 2019)



offline

Performing offline report...
Validating Master ldif file (/tmp/userRoot-ssc-prd-ipa-099.ldif)...
Validating Replica ldif file (/tmp/userRoot-cl-rhm-0252.ldif)...
Gathering all the DN's...
Gathering the database RUV's...
Comparing Master to Replica...
Comparing Replica to Master...
Preparing report...
================================================================================
         Replication Synchronization Report  (Wed Sep 11 10:20:41 2019)
================================================================================


Database RUV's
=====================================================

Master RUV:
  {replica 11 ldap://nsc-prd-ipa-099.is.cl.ssa.gov:389} 5aec66900000000b0000 5d77c25d0047000b0000
  {replica 13 ldap://nsc-prd-ipa-100.is.cl.ssa.gov:389} 5aec69780000000d0000 5d77773a0000000d0000
  {replica 3 ldap://cl-rhm-0251.is.cl.ssa.gov:389} 5aec5130000000030000 5d77fe3a004700030000
  {replica 4 ldap://ssc-prd-ipa-099.is.cl.ssa.gov:389} 5aec512b000100040000 5d781f13000100040000
  {replica 7 ldap://cl-rhm-0252.is.cl.ssa.gov:389} 5aec540d000000070000 5d77fe8d000300070000
  {replica 9 ldap://ssc-prd-ipa-100.is.cl.ssa.gov:389} 5aec63cd000000090000 5d767d1c000300090000
  {replicageneration} 5aec512b000000040000

Replica RUV:
  {replica 11 ldap://nsc-prd-ipa-099.is.cl.ssa.gov:389} 5aec66900000000b0000 5d77c25d0047000b0000
  {replica 13 ldap://nsc-prd-ipa-100.is.cl.ssa.gov:389} 5aec69780000000d0000 5d77773a0000000d0000
  {replica 3 ldap://cl-rhm-0251.is.cl.ssa.gov:389} 5aec5130000000030000 5d77fe3a004700030000
  {replica 4 ldap://ssc-prd-ipa-099.is.cl.ssa.gov:389} 5aec512b000100040000 5d781f13000100040000
  {replica 7 ldap://cl-rhm-0252.is.cl.ssa.gov:389} 5aec540d000000070000 5d77fe8d000300070000
  {replica 9 ldap://ssc-prd-ipa-100.is.cl.ssa.gov:389} 5aec63cd000000090000 5d767d1c000300090000
  {replicageneration} 5aec512b000000040000


Entry Counts
=====================================================

Master:  3762
Replica: 3762


Tombstones
=====================================================

Master:  117
Replica: 117


Conflict Entries
=====================================================

Master Conflict Entries:  2
Replica Conflict Entries: 2


Entry Inconsistencies
=====================================================

cn=hbacservicegroups,cn=hbac,dc=is,dc=cl,dc=ssa,dc=gov
------------------------------------------------------
 - Attribute 'aci' is different:
      Master:
        - State Info: aci;vucsn-5bd0b82d001600040000: (targetattr = "description || member")(targetfilter = "(&(cn=bdp-prd-*)(objectclass=ipahbacservicegroup))")(version 3.0;acl "permission:bdp-prd - HBAC Service Group mgmt";allow (all) groupdn = "ldap:///cn=bdp-prd - HBAC Service Group mgmt,cn=permissions,cn=pbac,dc=is,dc=cl,dc=ssa,dc=gov";)
        - Date:       Wed Oct 24 14:21:33 2018

        - State Info: aci;vucsn-5ca65f0b02af00040000: (targetattr = "description || member")(targetfilter = "(&(cn=bdp-dev-*)(objectclass=ipahbacservicegroup))")(version 3.0;acl "permission:bdp-dev - HBAC Service Group mgmt";allow (all) groupdn = "ldap:///cn=bdp-dev - HBAC Service Group mgmt,cn=permissions,cn=pbac,dc=is,dc=cl,dc=ssa,dc=gov";)
        - Date:       Thu Apr  4 15:46:19 2019

        - State Info: aci;vucsn-5ca7a61b014300040000;vdcsn-5ca7a5cf00a500040000: (targetattr = "description || member")(targetfilter = "(&(cn=default-prd-*)(objectclass=ipahbacservicegroup))")(version 3.0;acl "permission:default-prd - HBAC Service Groupmgmt";allow (all) groupdn = "ldap:///cn=default-prd - HBAC Service Group mgmt,cn=permissions,cn=pbac,dc=is,dc=cl,dc=ssa,dc=gov";)
        - Date:       Fri Apr  5 15:01:47 2019

        - State Info: aci;vucsn-5cb0cbff00be00070000;vdcsn-5cb0cca1009300070000;deleted: (targetattr = "description || member")(targetfilter = "(&(cn=desei-ipavm-*)(objectclass=ipahbacservicegroup))")(version 3.0;acl "permission:desei-ipavm - HBAC Service Group mgmt";allow (all) groupdn = "ldap:///cn=desei-ipavm - HBAC Service Group mgmt,cn=permissions,cn=pbac,dc=is,dc=cl,dc=ssa,dc=gov";)
        - Date:       Fri Apr 12 13:36:33 2019

      Replica:
        - State Info: aci;vucsn-5bd0b82d001600040000: (targetattr = "description || member")(targetfilter = "(&(cn=bdp-prd-*)(objectclass=ipahbacservicegroup))")(version 3.0;acl "permission:bdp-prd - HBAC Service Group mgmt";allow (all) groupdn = "ldap:///cn=bdp-prd - HBAC Service Group mgmt,cn=permissions,cn=pbac,dc=is,dc=cl,dc=ssa,dc=gov";)
        - Date:       Wed Oct 24 14:21:33 2018

        - State Info: aci;vucsn-5ca65f0b02af00040000: (targetattr = "description || member")(targetfilter = "(&(cn=bdp-dev-*)(objectclass=ipahbacservicegroup))")(version 3.0;acl "permission:bdp-dev - HBAC Service Group mgmt";allow (all) groupdn = "ldap:///cn=bdp-dev - HBAC Service Group mgmt,cn=permissions,cn=pbac,dc=is,dc=cl,dc=ssa,dc=gov";)
        - Date:       Thu Apr  4 15:46:19 2019

        - State Info: aci;vucsn-5ca7a61b014300040000: (targetattr = "description || member")(targetfilter = "(&(cn=default-prd-*)(objectclass=ipahbacservicegroup))")(version 3.0;acl "permission:default-prd - HBAC Service Group mgmt";allow (all) groupdn = "ldap:///cn=default-prd - HBAC Service Group mgmt,cn=permissions,cn=pbac,dc=is,dc=cl,dc=ssa,dc=gov";)
        - Date:       Fri Apr  5 15:01:47 2019

        - State Info: aci;vucsn-5cb0cbff00be00070000;vdcsn-5cb0cca1009300070000;deleted: (targetattr = "description || member")(targetfilter = "(&(cn=desei-ipavm-*)(objectclass=ipahbacservicegroup))")(version 3.0;acl "permission:desei-ipavm - HBAC Service Group mgmt";allow (all) groupdn = "ldap:///cn=desei-ipavm - HBAC Service Group mgmt,cn=permissions,cn=pbac,dc=is,dc=cl,dc=ssa,dc=gov";)
        - Date:       Fri Apr 12 13:36:33 2019


cn=sudorules,cn=sudo,dc=is,dc=cl,dc=ssa,dc=gov
----------------------------------------------
 - Attribute 'aci' is different:
      Master:
        - State Info: aci;vucsn-5bd0ba6c000100040000: (targetattr = "cmdcategory || description || externalhost || externaluser || hostcategory || hostmask || ipaenabledflag ||ipasudoopt || ipasudorunas || ipasudorunasextgroup || ipasudorunasextuser ||ipasudorunasextusergroup || ipasudorunasgroup || ipasudorunasgroupcategory || ipasudorunasusercategory || memberallowcmd || memberdenycmd || memberhost || memberuser || sudonotafter || sudonotbefore || sudoorder || usercategory")(targetfilter = "(&(cn=bdp-prd-*)(objectclass=ipasudorule))")(version 3.0;acl "permission:bdp-prd - Sudo Rule mgmt";allow (all) groupdn = "ldap:///cn=bdp-prd - Sudo Rule mgmt,cn=permissions,cn=pbac,dc=is,dc=cl,dc=ssa,dc=gov";)
        - Date:       Wed Oct 24 14:31:08 2018

        - State Info: aci;vucsn-5ca65f14030100040000: (targetattr = "cmdcategory || description || externalhost || externaluser || hostcategory || hostmask || ipaenabledflag ||ipasudoopt || ipasudorunas || ipasudorunasextgroup || ipasudorunasextuser ||ipasudorunasextusergroup || ipasudorunasgroup || ipasudorunasgroupcategory || ipasudorunasusercategory || memberallowcmd || memberdenycmd || memberhost || memberuser || sudonotafter || sudonotbefore || sudoorder || usercategory")(targetfilter = "(&(cn=bdp-dev-*)(objectclass=ipasudorule))")(version 3.0;acl "permission:bdp-dev - Sudo Rule mgmt";allow (all) groupdn = "ldap:///cn=bdp-dev - Sudo Rule mgmt,cn=permissions,cn=pbac,dc=is,dc=cl,dc=ssa,dc=gov";)
        - Date:       Thu Apr  4 15:46:28 2019

        - State Info: aci;vucsn-5ca7a62901c800040000;vdcsn-5ca7a4a500c500040000: (targetattr = "cmdcategory || description || externalhost || externaluser || hostcategory || hostmask || ipaenabledflag || ipasudoopt || ipasudorunas || ipasudorunasextgroup || ipasudorunasextuser || ipasudorunasextusergroup || ipasudorunasgroup || ipasudorunasgroupcategory || ipasudorunasusercategory || memberallowcmd || memberdenycmd || memberhost || memberuser || sudonotafter || sudonotbefore || sudoorder || usercategory")(targetfilter = "(&(cn=default-prd-*)(objectclass=ipasudorule))")(version 3.0;acl "permission:default-prd - Sudo Rule mgmt";allow (all) groupdn = "ldap:///cn=default-prd - Sudo Rule mgmt,cn=permissions,cn=pbac,dc=is,dc=cl,dc=ssa,dc=gov";)
        - Date:       Fri Apr  5 15:02:01 2019

        - State Info: aci;vucsn-5cb0cc0d015800070000;vdcsn-5cb0cca900ce00070000;deleted: (targetattr = "cmdcategory || description || externalhost || externaluser || hostcategory || hostmask || ipaenabledflag || ipasudoopt || ipasudorunas || ipasudorunasextgroup || ipasudorunasextuser || ipasudorunasextusergroup || ipasudorunasgroup || ipasudorunasgroupcategory || ipasudorunasusercategory || memberallowcmd || memberdenycmd || memberhost || memberuser || sudonotafter || sudonotbefore || sudoorder || usercategory")(targetfilter = "(&(cn=desei-ipavm-*)(objectclass=ipasudorule))")(version 3.0;acl "permission:desei-ipavm - Sudo Rule mgmt";allow (all) groupdn = "ldap:///cn=desei-ipavm - Sudo Rule mgmt,cn=permissions,cn=pbac,dc=is,dc=cl,dc=ssa,dc=gov";)
        - Date:       Fri Apr 12 13:36:41 2019

      Replica:
        - State Info: aci;vucsn-5bd0ba6c000100040000: (targetattr = "cmdcategory || description || externalhost || externaluser || hostcategory || hostmask || ipaenabledflag ||ipasudoopt || ipasudorunas || ipasudorunasextgroup || ipasudorunasextuser ||ipasudorunasextusergroup || ipasudorunasgroup || ipasudorunasgroupcategory || ipasudorunasusercategory || memberallowcmd || memberdenycmd || memberhost || memberuser || sudonotafter || sudonotbefore || sudoorder || usercategory")(targetfilter = "(&(cn=bdp-prd-*)(objectclass=ipasudorule))")(version 3.0;acl "permission:bdp-prd - Sudo Rule mgmt";allow (all) groupdn = "ldap:///cn=bdp-prd - Sudo Rule mgmt,cn=permissions,cn=pbac,dc=is,dc=cl,dc=ssa,dc=gov";)
        - Date:       Wed Oct 24 14:31:08 2018

        - State Info: aci;vucsn-5ca65f14030100040000: (targetattr = "cmdcategory || description || externalhost || externaluser || hostcategory || hostmask || ipaenabledflag ||ipasudoopt || ipasudorunas || ipasudorunasextgroup || ipasudorunasextuser ||ipasudorunasextusergroup || ipasudorunasgroup || ipasudorunasgroupcategory || ipasudorunasusercategory || memberallowcmd || memberdenycmd || memberhost || memberuser || sudonotafter || sudonotbefore || sudoorder || usercategory")(targetfilter = "(&(cn=bdp-dev-*)(objectclass=ipasudorule))")(version 3.0;acl "permission:bdp-dev - Sudo Rule mgmt";allow (all) groupdn = "ldap:///cn=bdp-dev - Sudo Rule mgmt,cn=permissions,cn=pbac,dc=is,dc=cl,dc=ssa,dc=gov";)
        - Date:       Thu Apr  4 15:46:28 2019

        - State Info: aci;vucsn-5ca7a62901c800040000: (targetattr = "cmdcategory || description || externalhost || externaluser || hostcategory || hostmask || ipaenabledflag ||ipasudoopt || ipasudorunas || ipasudorunasextgroup || ipasudorunasextuser ||ipasudorunasextusergroup || ipasudorunasgroup || ipasudorunasgroupcategory || ipasudorunasusercategory || memberallowcmd || memberdenycmd || memberhost || memberuser || sudonotafter || sudonotbefore || sudoorder || usercategory")(targetfilter = "(&(cn=default-prd-*)(objectclass=ipasudorule))")(version 3.0;acl "permission:default-prd - Sudo Rule mgmt";allow (all) groupdn = "ldap:///cn=default-prd - Sudo Rule mgmt,cn=permissions,cn=pbac,dc=is,dc=cl,dc=ssa,dc=gov";)
        - Date:       Fri Apr  5 15:02:01 2019

        - State Info: aci;vucsn-5cb0cc0d015800070000;vdcsn-5cb0cca900ce00070000;deleted: (targetattr = "cmdcategory || description || externalhost || externaluser || hostcategory || hostmask || ipaenabledflag || ipasudoopt || ipasudorunas || ipasudorunasextgroup || ipasudorunasextuser || ipasudorunasextusergroup || ipasudorunasgroup || ipasudorunasgroupcategory || ipasudorunasusercategory || memberallowcmd || memberdenycmd || memberhost || memberuser || sudonotafter || sudonotbefore || sudoorder || usercategory")(targetfilter = "(&(cn=desei-ipavm-*)(objectclass=ipasudorule))")(version 3.0;acl "permission:desei-ipavm - Sudo Rule mgmt";allow (all) groupdn = "ldap:///cn=desei-ipavm - Sudo Rule mgmt,cn=permissions,cn=pbac,dc=is,dc=cl,dc=ssa,dc=gov";)
        - Date:       Fri Apr 12 13:36:41 2019


cn=automember,cn=etc,dc=is,dc=cl,dc=ssa,dc=gov
----------------------------------------------
 - Attribute 'aci' is different:
      Master:
        - State Info: aci;vucsn-5c9a486a000200040000: (targetattr = "automemberexclusiveregex || automemberinclusiveregex || automembertargetgroup || description || objectclass")(targetfilter = "(&(cn=bdp-prd-*)(objectclass=automemberregexrule))")(version 3.0;acl "permission:bdp-prd - Automember mgmt";allow (all) groupdn = "ldap:///cn=bdp-prd - Automember mgmt,cn=permissions,cn=pbac,dc=is,dc=cl,dc=ssa,dc=gov";)
        - Date:       Tue Mar 26 11:42:34 2019

        - State Info: aci;vucsn-5ca65eff024c00040000: (targetattr = "automemberexclusiveregex || automemberinclusiveregex || automembertargetgroup || description || objectclass")(targetfilter = "(&(cn=bdp-dev-*)(objectclass=automemberregexrule))")(version 3.0;acl "permission:bdp-dev - Automember mgmt";allow (all) groupdn = "ldap:///cn=bdp-dev - Automember mgmt,cn=permissions,cn=pbac,dc=is,dc=cl,dc=ssa,dc=gov";)
        - Date:       Thu Apr  4 15:46:07 2019

        - State Info: aci;vucsn-5ca7a60f00ea00040000;vdcsn-5ca7a5c9007200040000: (targetattr = "automemberexclusiveregex || automemberinclusiveregex || automembertargetgroup ||description || objectclass")(targetfilter = "(&(cn=default-prd-*)(objectclass=automemberregexrule))")(version 3.0;acl "permission:default-prd - Automember mgmt";allow (all) groupdn = "ldap:///cn=default-prd - Automember mgmt,cn=permissions,cn=pbac,dc=is,dc=cl,dc=ssa,dc=gov";)
        - Date:       Fri Apr  5 15:01:35 2019

        - State Info: aci;vucsn-5cb0cbf6004300070000;vdcsn-5cb0cc9b005000070000;deleted: (targetattr = "automemberexclusiveregex || automemberinclusiveregex || automembertargetgroup || description || objectclass")(targetfilter = "(&(cn=desei-ipavm-*)(objectclass=automemberregexrule))")(version 3.0;acl "permission:desei-ipavm - Automember mgmt";allow (all) groupdn = "ldap:///cn=desei-ipavm - Automember mgmt,cn=permissions,cn=pbac,dc=is,dc=cl,dc=ssa,dc=gov";)
        - Date:       Fri Apr 12 13:36:27 2019

      Replica:
        - State Info: aci;vucsn-5c9a486a000200040000: (targetattr = "automemberexclusiveregex || automemberinclusiveregex || automembertargetgroup || description || objectclass")(targetfilter = "(&(cn=bdp-prd-*)(objectclass=automemberregexrule))")(version 3.0;acl "permission:bdp-prd - Automember mgmt";allow (all) groupdn = "ldap:///cn=bdp-prd - Automember mgmt,cn=permissions,cn=pbac,dc=is,dc=cl,dc=ssa,dc=gov";)
        - Date:       Tue Mar 26 11:42:34 2019

        - State Info: aci;vucsn-5ca65eff024c00040000: (targetattr = "automemberexclusiveregex || automemberinclusiveregex || automembertargetgroup || description || objectclass")(targetfilter = "(&(cn=bdp-dev-*)(objectclass=automemberregexrule))")(version 3.0;acl "permission:bdp-dev - Automember mgmt";allow (all) groupdn = "ldap:///cn=bdp-dev - Automember mgmt,cn=permissions,cn=pbac,dc=is,dc=cl,dc=ssa,dc=gov";)
        - Date:       Thu Apr  4 15:46:07 2019

        - State Info: aci;vucsn-5ca7a60f00ea00040000: (targetattr = "automemberexclusiveregex || automemberinclusiveregex || automembertargetgroup || description || objectclass")(targetfilter = "(&(cn=default-prd-*)(objectclass=automemberregexrule))")(version 3.0;acl "permission:default-prd - Automember mgmt";allow (all) groupdn= "ldap:///cn=default-prd - Automember mgmt,cn=permissions,cn=pbac,dc=is,dc=cl,dc=ssa,dc=gov";)
        - Date:       Fri Apr  5 15:01:35 2019

        - State Info: aci;vucsn-5cb0cbf6004300070000;vdcsn-5cb0cc9b005000070000;deleted: (targetattr = "automemberexclusiveregex || automemberinclusiveregex || automembertargetgroup || description || objectclass")(targetfilter = "(&(cn=desei-ipavm-*)(objectclass=automemberregexrule))")(version 3.0;acl "permission:desei-ipavm - Automember mgmt";allow (all) groupdn = "ldap:///cn=desei-ipavm - Automember mgmt,cn=permissions,cn=pbac,dc=is,dc=cl,dc=ssa,dc=gov";)
        - Date:       Fri Apr 12 13:36:27 2019
Comment 50 mreynolds 2019-09-11 17:16:22 UTC 
Hmmm part of the issue could be that the ldif was not generated by db2ldif, but from ldapsearch instead.  The entries would look different, but I'm not sure if that is the issue here or not.  I think the offline might not be processing the replication state attributes correctly and finding false inconsistencies.  Any chance I could get the ldifs they used?  

Side note: there should really be another bug opened to address the differences between the online and offline mode, as this bug is closed as the original reported issue was resolved.
Note You need to log in before you can comment on or make changes to this bug.