Bug 1103337

Summary: find a way to remove replication plugin errors messages "changelog iteration code returned a dummy entry with csn %s, skipping ..."
Product: Red Hat Enterprise Linux 6 Reporter: Marc Sauton <msauton>
Component: 389-ds-baseAssignee: Noriko Hosoi <nhosoi>
Status: CLOSED ERRATA QA Contact: Sankar Ramalingam <sramling>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.5CC: amsharma, jgalipea, msauton, nhosoi, nkinder, rmeggins
Target Milestone: pre-dev-freeze   
Target Release: 6.6   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 389-ds-base-1.2.11.15-34.el6 Doc Type: Bug Fix
Doc Text:
Cause: Message "changelog iteration code returned a dummy entry with csn %s, skipping ..." is coded as an error level, which should not be. Consequence: Once the server runs into the state, the benign error is logged in the error log repeatedly. Fix: Changed the log level to replication log level. Result: Unless the log level is set to the replication log level, the message is not logged in the error log.
Story Points: ---
Clone Of:
: 1108405 (view as bug list) Environment:
Last Closed: 2014-10-14 07:55:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1108405    

Description Marc Sauton 2014-05-30 19:04:42 UTC
Description of problem:

Sometimes, when trying to resolve replication issues, we end up with a newer message flooding the errors log from the replication plug-in:
"
changelog iteration code returned a dummy entry with csn 5385ee18000000070000, skipping ...
"
and there is no way to get rid of this messages that keeps filling the errors log file.

this is a report to find a way to try suppress this message when not necessary, in
ldap/servers/plugins/replication/windows_inc_protocol.c
ldap/servers/plugins/replication/repl5_inc_protocol.c


examples:

RHDS 9.1 RHEL 6.5 with
389-ds-base-1.2.11.15-30.el6_5.x86_64
[29/May/2014:12:29:09 -0400] agmt="cn=ds1-xp" (ds1:636) - Can't locate CSN 5385e594000000070000 in the changelog (DB rc=-30988). The consumer may need to be reinitialized.
[29/May/2014:12:29:09 -0400] NSMMReplicationPlugin - agmt="cn=ds1-xp" (ds1:636): changelog iteration code returned a dummy entry with csn 5385ee18000000070000, skipping ...

RHEL 6.4 and IPA 3 with:
389-ds-base-1.2.11.15-20.el6_4.x86_64
ipa-server-3.0.0-26.el6_4.4.x86_64
[05/Feb/2014:09:22:36 -0800] agmt="cn=meTods1.example.com" (ds1:389) - Can't locate CSN 5244d0b8000100040000 in the changelog (DB rc=-30988). The consumer may need to be reinitialized.
[05/Feb/2014:09:22:36 -0800] NSMMReplicationPlugin - agmt="cn=meTods1.example.com" (ds1:389): changelog iteration code returned a dummy entry with csn 52d42b20000000320000, skipping ...

other example:

[18/Oct/2011:13:09:57 +0000] NSMMReplicationPlugin - agmt="cn=srvAtosrvB" (srvB:389): changelog iteration code returned a dummy entry with csn 4e9d7bc2000000080000, skipping ...


Version-Release number of selected component (if applicable):

RHDS 9.1 RHEL 6.5 with
389-ds-base-1.2.11.15-30.el6_5.x86_64

RHEL 6.4 and IPA 3 with:
389-ds-base-1.2.11.15-20.el6_4.x86_64
ipa-server-3.0.0-26.el6_4.4.x86_64


How reproducible:
N/A, happened with several customers while trying to resolve replication issues.


Steps to Reproduce:
1. N/A
2.
3.

Actual results:

errors log filling with:

[29/May/2014:12:29:09 -0400] NSMMReplicationPlugin - agmt="cn=ds1-xp" (ds1:636): changelog iteration code returned a dummy entry with csn 5385ee18000000070000, skipping ...


Expected results:


Additional info:

Comment 1 Marc Sauton 2014-05-30 19:06:11 UTC
added upstream ticket
https://fedorahosted.org/389/ticket/47809
find a way to remove replication plugin errors messages "changelog iteration code returned a dummy entry with csn %s, skipping ..."

Comment 3 Sankar Ramalingam 2014-06-12 12:13:30 UTC
Is this error coming when the replication error log level is enabled?
Can this be reproduced in QE environment?

Comment 4 Noriko Hosoi 2014-06-12 17:05:16 UTC
(In reply to Sankar Ramalingam from comment #3)
> Is this error coming when the replication error log level is enabled?
> Can this be reproduced in QE environment?

I ran this test first:
https://bugzilla.redhat.com/show_bug.cgi?id=1080185#c1

then checked the error log to make sure there is ...
1) no "changelog iteration code returned ..." message is loged, and
2) no duplicate of "Can't locate CSN 539946eb000b00020000 in the changelog" message.

Comment 6 Amita Sharma 2014-08-14 12:33:31 UTC
[root@dhcp201-155 export]# tail -f /var/log/dirsrv/slapd-M1/errors
[14/Aug/2014:08:13:17 -0400] attrcrypt - No symmetric key found for cipher AES in backend repman, attempting to create one...
[14/Aug/2014:08:13:17 -0400] attrcrypt - Key for cipher AES successfully generated and stored
[14/Aug/2014:08:13:17 -0400] attrcrypt - No symmetric key found for cipher 3DES in backend repman, attempting to create one...
[14/Aug/2014:08:13:17 -0400] attrcrypt - Key for cipher 3DES successfully generated and stored
[14/Aug/2014:08:13:17 -0400] - slapd started.  Listening on All Interfaces port 30100 for LDAP requests
[14/Aug/2014:08:13:17 -0400] - Listening on All Interfaces port 30101 for LDAPS requests
[14/Aug/2014:08:15:26 -0400] NSMMReplicationPlugin - agmt="cn=M1_to_M2" (dhcp201-155:30103): Replica has a different generation ID than the local data.
[14/Aug/2014:08:15:27 -0400] NSMMReplicationPlugin - agmt="cn=M1_to_M4" (dhcp201-155:30107): Replica has a different generation ID than the local data.
[14/Aug/2014:08:16:57 -0400] NSMMReplicationPlugin - Beginning total update of replica "agmt="cn=M1_to_M2" (dhcp201-155:30103)".
[14/Aug/2014:08:17:02 -0400] NSMMReplicationPlugin - Finished total update of replica "agmt="cn=M1_to_M2" (dhcp201-155:30103)". Sent 160 entries.

^C
[root@dhcp201-155 export]# rpm -qa | grep 389
389-admin-debuginfo-1.1.34-1.el6.x86_64
389-ds-base-1.2.11.15-39.el6.x86_64
389-ds-base-debuginfo-1.2.11.15-36.el6.x86_64
389-adminutil-1.1.17-1.el6.x86_64
389-adminutil-debuginfo-1.1.17-1.el6.x86_64
389-ds-base-libs-1.2.11.15-39.el6.x86_64
389-console-1.1.7-1.el6.noarch

Hence Verified.

Comment 7 errata-xmlrpc 2014-10-14 07:55:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1385.html