Bug 1034902 - replica init/bulk import errors should be more verbose
Summary: replica init/bulk import errors should be more verbose
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: 389-ds-base
Version: 7.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Rich Megginson
QA Contact: Sankar Ramalingam
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-11-26 16:52 UTC by Nathan Kinder
Modified: 2020-09-13 20:51 UTC (History)
2 users (show)

Fixed In Version: 389-ds-base-1.3.1.6-14.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-06-13 10:02:31 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github 389ds 389-ds-base issues 943 0 None closed replica init/bulk import errors should be more verbose 2020-11-09 12:55:38 UTC

Description Nathan Kinder 2013-11-26 16:52:12 UTC
This bug is created as a clone of upstream ticket:
https://fedorahosted.org/389/ticket/47606

For example - attempting to do repl init with a very large 100MB entry - the messages in the errors and access logs are not informative, and the user has no idea how to proceed e.g. increase nsslapd-maxbersize (but how large?  there is no information in the access or errors log), increase nsslapd-cachememsize, increase nsslapd-import-cachesize (but again, there is no information about how much to increase these).

Comment 3 Noriko Hosoi 2014-02-11 17:40:28 UTC
1. Default nsslapd-maxbersize is 2KB.  Add an entry larger than 2KB to the master and run total update.  The consumer should log this error:
     Incoming BER Element was @@@ bytes, max allowable is ### bytes.
     Change the nsslapd-maxbersize attribute in cn=config to increase.

2. Then, increase nsslapd-maxbersize value to larger than nsslapd-cachememsize.  Add and entry larger than nsslapd-cachememsize and smaller than nsslapd-maxbersize and run total update.  The consumer should log this error:
     import <backend>: REASON: entry too large (@@@ bytes) for the
     effective import buffer size (### bytes). Try increasing nsslapd-
     cachememsize for the backend instance "userRoot".

Comment 4 Amita Sharma 2014-02-13 15:12:29 UTC
Hi Noriko,

I am not able to get it ::
M1 - 30100, M2 - 30102
[root@dhcp201-149 ~]# ldapsearch -D "cn=directory manager" -w Secret123 -p 30102 -h localhost -b "cn=config" | grep -i "nsslapd-maxbersize"
nsslapd-maxbersize: 0

ldapadd -x -h localhost -p 30100 -D "cn=Directory Manager" -w Secret123 -f /export/a.ldif

[root@dhcp201-149 ~]# vim /export/a.ldif 
# Jennifer's Entry
dn: cn=Jennifer, dc=example, dc=com
cn: Jennifer J Jensen
cn: Jennifer Jensen
objectClass: person
objectClass: inetOrgPerson
sn: Jensen
jpegPhoto:< file://export/we.JPG

[root@dhcp201-149 ~]# ls -alh /export/we.jpg 
-rwxrwxr-x. 1 root root 89K Feb 13 19:17 /export/we.jpg

ldapmodify -D "cn=directory manager" -w Secret123 -p 30100 -h localhost -x << EOF
dn: cn=M1_to_M2,cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config
changetype: modify
add: nsDS5BeginReplicaRefresh
nsDS5BeginReplicaRefresh: start

[root@dhcp201-149 export]# tail -f /var/log/dirsrv/slapd-M2/errors
[13/Feb/2014:19:38:33 +051800] - import userRoot: Workers finished; cleaning up...
[13/Feb/2014:19:38:34 +051800] - import userRoot: Workers cleaned up.
[13/Feb/2014:19:38:34 +051800] - import userRoot: Indexing complete.  Post-processing...
[13/Feb/2014:19:38:34 +051800] - import userRoot: Generating numSubordinates complete.
[13/Feb/2014:19:38:34 +051800] - import userRoot: Flushing caches...
[13/Feb/2014:19:38:34 +051800] - import userRoot: Closing files...
[13/Feb/2014:19:38:34 +051800] - import userRoot: Import complete.  Processed 161 entries in 4 seconds. (40.25 entries/sec)
[13/Feb/2014:19:38:34 +051800] NSMMReplicationPlugin - multimaster_be_state_change: replica dc=example,dc=com is coming online; enabling replication
[13/Feb/2014:19:38:35 +051800] NSMMReplicationPlugin - replica_reload_ruv: Warning: new data for replica dc=example,dc=com does not match the data in the changelog.
 Recreating the changelog file. This could affect replication with replica's  consumers in which case the consumers should be reinitialized.

Comment 5 Noriko Hosoi 2014-02-13 17:56:55 UTC
(In reply to Amita Sharma from comment #4)
> Hi Noriko,
> 
> I am not able to get it ::
> M1 - 30100, M2 - 30102
> [root@dhcp201-149 ~]# ldapsearch -D "cn=directory manager" -w Secret123 -p
> 30102 -h localhost -b "cn=config" | grep -i "nsslapd-maxbersize"
> nsslapd-maxbersize: 0
> 
> ldapadd -x -h localhost -p 30100 -D "cn=Directory Manager" -w Secret123 -f
> /export/a.ldif
> 
> [root@dhcp201-149 ~]# vim /export/a.ldif 
> # Jennifer's Entry
> dn: cn=Jennifer, dc=example, dc=com
> cn: Jennifer J Jensen
> cn: Jennifer Jensen
> objectClass: person
> objectClass: inetOrgPerson
> sn: Jensen
> jpegPhoto:< file://export/we.JPG
> 
> [root@dhcp201-149 ~]# ls -alh /export/we.jpg 
> -rwxrwxr-x. 1 root root 89K Feb 13 19:17 /export/we.jpg

So, your entry is about 89KB?

If you set 0 to nsslapd-maxbersize, it picks up the default size.
> nsslapd-maxbersize: 0

The default size is 2MB.
Default Value 	2097152 
https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html/Configuration_Command_and_File_Reference/Core_Server_Configuration_Reference.html#cnconfig-nsslapd_maxbersize_Maximum_Message_Size

Comment 6 Amita Sharma 2014-02-14 05:34:02 UTC
(In reply to Noriko Hosoi from comment #5)
> (In reply to Amita Sharma from comment #4)
> > Hi Noriko,
> > 
> > I am not able to get it ::
> > M1 - 30100, M2 - 30102
> > [root@dhcp201-149 ~]# ldapsearch -D "cn=directory manager" -w Secret123 -p
> > 30102 -h localhost -b "cn=config" | grep -i "nsslapd-maxbersize"
> > nsslapd-maxbersize: 0
> > 
> > ldapadd -x -h localhost -p 30100 -D "cn=Directory Manager" -w Secret123 -f
> > /export/a.ldif
> > 
> > [root@dhcp201-149 ~]# vim /export/a.ldif 
> > # Jennifer's Entry
> > dn: cn=Jennifer, dc=example, dc=com
> > cn: Jennifer J Jensen
> > cn: Jennifer Jensen
> > objectClass: person
> > objectClass: inetOrgPerson
> > sn: Jensen
> > jpegPhoto:< file://export/we.JPG
> > 
> > [root@dhcp201-149 ~]# ls -alh /export/we.jpg 
> > -rwxrwxr-x. 1 root root 89K Feb 13 19:17 /export/we.jpg
> 
> So, your entry is about 89KB?
> 
> If you set 0 to nsslapd-maxbersize, it picks up the default size.
> > nsslapd-maxbersize: 0
> 
> The default size is 2MB.
> Default Value 	2097152 
Oh.. I followed https://bugzilla.redhat.com/show_bug.cgi?id=1034902#c3 - Default nsslapd-maxbersize is 2KB.
I will try with 2MB now.
> https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/
> 9.0/html/Configuration_Command_and_File_Reference/
> Core_Server_Configuration_Reference.html#cnconfig-
> nsslapd_maxbersize_Maximum_Message_Size

Comment 7 Amita Sharma 2014-02-14 10:34:21 UTC
(In reply to Noriko Hosoi from comment #3)
> 1. Default nsslapd-maxbersize is 2KB.  Add an entry larger than 2KB to the
> master and run total update.  The consumer should log this error:
>      Incoming BER Element was @@@ bytes, max allowable is ### bytes.
>      Change the nsslapd-maxbersize attribute in cn=config to increase.

Here when I added entry larger than 2KB, master logs an error rather then consumer ::
[root@dhcp201-149 ~]# ldapadd -x -h localhost -p 30100 -D "cn=Directory Manager" -w Secret123 -f /export/b.ldif 
adding new entry "cn=Jennifer2,dc=example,dc=com"
ldapadd: update failed: cn=Jennifer2,dc=example,dc=com
ldap_add: Can't contact LDAP server (-1)

error logs of M1
========================
[14/Feb/2014:14:37:11 +051800] connection - conn=24 fd=64 Incoming BER Element was 4431539 bytes, max allowable is 2097152 bytes. Change the nsslapd-maxbersize attribute in cn=config to increase.

Comment 8 Rich Megginson 2014-02-14 15:02:56 UTC
(In reply to Amita Sharma from comment #7)
> (In reply to Noriko Hosoi from comment #3)
> > 1. Default nsslapd-maxbersize is 2KB.  Add an entry larger than 2KB to the
> > master and run total update.  The consumer should log this error:
> >      Incoming BER Element was @@@ bytes, max allowable is ### bytes.
> >      Change the nsslapd-maxbersize attribute in cn=config to increase.
> 
> Here when I added entry larger than 2KB, master logs an error rather then
> consumer ::
> [root@dhcp201-149 ~]# ldapadd -x -h localhost -p 30100 -D "cn=Directory
> Manager" -w Secret123 -f /export/b.ldif 
> adding new entry "cn=Jennifer2,dc=example,dc=com"
> ldapadd: update failed: cn=Jennifer2,dc=example,dc=com
> ldap_add: Can't contact LDAP server (-1)
> 
> error logs of M1
> ========================
> [14/Feb/2014:14:37:11 +051800] connection - conn=24 fd=64 Incoming BER
> Element was 4431539 bytes, max allowable is 2097152 bytes. Change the
> nsslapd-maxbersize attribute in cn=config to increase.

You will probably have to import the large entry into the master using ldif2db or something like that, then try a replica init from the master to the consumer.

Comment 9 Amita Sharma 2014-02-17 08:57:27 UTC
Default nsslapd-maxbersize is 2MB.  Add an entry larger than 2KB to the master and run total update.  The consumer should log this error:
     Incoming BER Element was @@@ bytes, max allowable is ### bytes.
     Change the nsslapd-maxbersize attribute in cn=config to increase.
You will probably have to import the large entry into the master using ldif2db or something like that, then try a replica init from the master to the consumer.

1. Master - 389
Consumer - 390
[root@dhcp201-149 ~]# ldapsearch -D "cn=directory manager" -w Secret123 -p 389 -h localhost -b "cn=config" | grep -i "nsslapd-maxbersize"
nsslapd-maxbersize: 0

2. Imported data to master which has an entry ::
[root@dhcp201-149 export]# ls -alh /export/we1.jpg 
-rwxr-xr-x. 1 root root 4.3M Feb 14 14:26 /export/we1.jpg

3. [root@dhcp201-149 export]# ldif2db.pl -n userRoot -Z dhcp201-149 -D "cn=Directory Manager" -w Secret123 -i /export/test.ldif
Successfully added task entry "cn=import_2014_2_17_13_5_27, cn=import, cn=tasks, cn=config"


4. error logs of Master
============================
[17/Feb/2014:13:05:28 +051800] - import userRoot: Import complete.  Processed 107 entries in 1 seconds. (107.00 entries/sec)
[17/Feb/2014:13:05:28 +051800] NSMMReplicationPlugin - multimaster_be_state_change: replica dc=example,dc=com is coming online; enabling replication
[17/Feb/2014:13:06:17 +051800] NSMMReplicationPlugin - Beginning total update of replica "agmt="cn=Master-Consumer" (dhcp201-149:390)".
[17/Feb/2014:13:06:22 +051800] NSMMReplicationPlugin - agmt="cn=Master-Consumer" (dhcp201-149:390): Failed to send extended operation: LDAP error -1 (Can't contact LDAP server)
[17/Feb/2014:13:06:22 +051800] NSMMReplicationPlugin - agmt="cn=Master-Consumer" (dhcp201-149:390): Received error -1 (Can't contact LDAP server):  for total update operation
[17/Feb/2014:13:06:23 +051800] NSMMReplicationPlugin - agmt="cn=Master-Consumer" (dhcp201-149:390): Warning: unable to send endReplication extended operation (Can't contact LDAP server)

5. error logs of consumer
===========================
[17/Feb/2014:13:06:16 +051800] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database
[17/Feb/2014:13:06:17 +051800] connection - conn=12 fd=66 Incoming BER Element was 4432141 bytes, max allowable is 2097152 bytes. Change the nsslapd-maxbersize attribute in cn=config to increase.
[17/Feb/2014:13:06:17 +051800] - ERROR bulk import abandoned
[17/Feb/2014:13:06:17 +051800] - import userRoot: Thread monitoring returned: -23

[17/Feb/2014:13:06:17 +051800] - import userRoot: Aborting all Import threads...
[17/Feb/2014:13:06:22 +051800] - import userRoot: Import threads aborted.
[17/Feb/2014:13:06:22 +051800] - import userRoot: Closing files...
[17/Feb/2014:13:06:22 +051800] - libdb: BDB3028 userRoot/entryrdn.db: unable to flush: No such file or directory
[17/Feb/2014:13:06:22 +051800] - import userRoot: Import failed.
[17/Feb/2014:13:06:22 +051800] - process_bulk_import_op: NULL target sdn

Is this error as expected on consumer?

==================================================================================================================================================================
Then, increase nsslapd-maxbersize value to larger than nsslapd-cachememsize.  Add and entry larger than nsslapd-cachememsize and smaller than nsslapd-maxbersize and run total update.  The consumer should log this error:
     import <backend>: REASON: entry too large (@@@ bytes) for the
     effective import buffer size (### bytes). Try increasing nsslapd-
     cachememsize for the backend instance "userRoot".

[root@dhcp201-149 export]# ldapsearch -D "cn=directory manager" -w Secret123 -p 389 -h localhost -b "cn=config" | grep -i "nsslapd-maxbersize"
nsslapd-maxbersize: 0
[root@dhcp201-149 export]# ldapsearch -D "cn=directory manager" -w Secret123 -p 389 -h localhost -b "cn=config" | grep -i "nsslapd-cachememsize"
nsslapd-cachememsize: 10485760

1. increase nsslapd-maxbersize value to larger than nsslapd-cachememsize
ldapmodify -x -h localhost -p 389 -D "cn=directory manager" -w Secret123 << EOF
dn: cn=config
changetype: modify
replace: nsslapd-maxbersize
nsslapd-maxbersize: 12500000
EOF

2. Add and entry larger than nsslapd-cachememsize and smaller than nsslapd-maxbersize and run total update

[root@dhcp201-149 export]# ls -alh sample.jpg
-rw-r--r--. 1 root root 12M Feb 14 19:29 sample.jpg

[root@dhcp201-149 export]# ldif2db.pl -n userRoot -Z dhcp201-149 -D "cn=Directory Manager" -w Secret123 -i /export/test.ldif
Successfully added task entry "cn=import_2014_2_17_13_53_54, cn=import, cn=tasks, cn=config"

Error on Master ::
[17/Feb/2014:13:53:53 +051800] - Bringing userRoot offline...
[17/Feb/2014:13:53:54 +051800] NSMMReplicationPlugin - multimaster_be_state_change: replica dc=example,dc=com is going offline; disabling replication
[17/Feb/2014:13:53:54 +051800] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database
[17/Feb/2014:13:53:54 +051800] - import userRoot: Beginning import job...
[17/Feb/2014:13:53:54 +051800] - import userRoot: Index buffering enabled with bucket size 19
[17/Feb/2014:13:53:54 +051800] - import userRoot: Processing file "/export/test.ldif"
[17/Feb/2014:13:53:54 +051800] - import userRoot: WARNING: skipping entry "cn=Jennifer J Jensen,dc=example,dc=com" ending line 422 of file "/export/test.ldif"
[17/Feb/2014:13:53:54 +051800] - import userRoot: REASON: entry too large (12495580 bytes) for the buffer size (8388608 bytes)
[17/Feb/2014:13:53:55 +051800] - import userRoot: Finished scanning file "/export/test.ldif" (16 entries)
[17/Feb/2014:13:53:55 +051800] - import userRoot: Workers finished; cleaning up...
[17/Feb/2014:13:53:55 +051800] - import userRoot: Workers cleaned up.
[17/Feb/2014:13:53:55 +051800] - import userRoot: Cleaning up producer thread...
[17/Feb/2014:13:53:55 +051800] - import userRoot: Indexing complete.  Post-processing...
[17/Feb/2014:13:53:55 +051800] - import userRoot: Generating numSubordinates complete.
[17/Feb/2014:13:53:55 +051800] - import userRoot: Flushing caches...
[17/Feb/2014:13:53:55 +051800] - import userRoot: Closing files...
[17/Feb/2014:13:53:55 +051800] - import userRoot: Import complete.  Processed 16 entries (1 were skipped) in 1 seconds. (16.00 entries/sec)
[17/Feb/2014:13:53:55 +051800] NSMMReplicationPlugin - multimaster_be_state_change: replica dc=example,dc=com is coming online; enabling replication
[17/Feb/2014:13:54:08 +051800] NSMMReplicationPlugin - Beginning total update of replica "agmt="cn=Master-Consumer" (dhcp201-149:390)".
[17/Feb/2014:13:54:12 +051800] NSMMReplicationPlugin - Finished total update of replica "agmt="cn=Master-Consumer" (dhcp201-149:390)". Sent 16 entries.

Error on consumer ::
[17/Feb/2014:13:54:08 +051800] NSMMReplicationPlugin - multimaster_be_state_change: replica dc=example,dc=com is going offline; disabling replication
[17/Feb/2014:13:54:08 +051800] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database
[17/Feb/2014:13:54:10 +051800] - import userRoot: Workers finished; cleaning up...
[17/Feb/2014:13:54:11 +051800] - import userRoot: Workers cleaned up.
[17/Feb/2014:13:54:11 +051800] - import userRoot: Indexing complete.  Post-processing...
[17/Feb/2014:13:54:11 +051800] - import userRoot: Generating numSubordinates complete.
[17/Feb/2014:13:54:11 +051800] - import userRoot: Flushing caches...
[17/Feb/2014:13:54:11 +051800] - import userRoot: Closing files...
[17/Feb/2014:13:54:11 +051800] - import userRoot: Import complete.  Processed 16 entries in 3 seconds. (5.33 entries/sec)
[17/Feb/2014:13:54:11 +051800] NSMMReplicationPlugin - multimaster_be_state_change: replica dc=example,dc=com is coming online; enabling replication

3. If I do ldapadd for adding entry larger than nsslapd-cachememsize and smaller than nsslapd-maxbersize and run total update

error on Master
===================
[17/Feb/2014:14:02:01 +051800] NSMMReplicationPlugin - agmt="cn=Master-Consumer" (dhcp201-149:390): Replication bind with SIMPLE auth resumed
[17/Feb/2014:14:02:01 +051800] NSMMReplicationPlugin - agmt="cn=Master-Consumer" (dhcp201-149:390): Failed to send add operation: LDAP error -1 (Can't contact LDAP server)
[17/Feb/2014:14:02:01 +051800] NSMMReplicationPlugin - agmt="cn=Master-Consumer" (dhcp201-149:390): Failed to send update operation to consumer (uniqueid 9d926401-97ad11e3-83f3cd23-c6c0a6aa, CSN 5301c87f000000010000): Can't contact LDAP server. Will retry later.
[17/Feb/2014:14:02:01 +051800] NSMMReplicationPlugin - agmt="cn=Master-Consumer" (dhcp201-149:390): Consumer failed to replay change (uniqueid (null), CSN (null)): Can't contact LDAP server(-1). Will retry later.
[17/Feb/2014:14:02:01 +051800] NSMMReplicationPlugin - agmt="cn=Master-Consumer" (dhcp201-149:390): Warning: unable to send endReplication extended operation (Can't contact LDAP server)

AND THIS ERROR GO ON AND ON AND ON..................

error on consumer
====================
[17/Feb/2014:14:02:06 +051800] connection - conn=52 fd=66 Incoming BER Element was 12493804 bytes, max allowable is 2097152 bytes. Change the nsslapd-maxbersize attribute in cn=config to increase.
[17/Feb/2014:14:02:09 +051800] connection - conn=53 fd=66 Incoming BER Element was 12493804 bytes, max allowable is 2097152 bytes. Change the nsslapd-maxbersize attribute in cn=config to increase.
[17/Feb/2014:14:02:12 +051800] connection - conn=54 fd=66 Incoming BER Element was 12493804 bytes, max allowable is 2097152 bytes. Change the nsslapd-maxbersize attribute in cn=config to increase.
[17/Feb/2014:14:02:15 +051800] connection - conn=55 fd=66 Incoming BER Element was 12493804 bytes, max allowable is 2097152 bytes. Change the nsslapd-maxbersize attribute in cn=config to increase.
[17/Feb/2014:14:02:19 +051800] connection - conn=56 fd=66 Incoming BER Element was 12493804 bytes, max allowable is 2097152 bytes. Change the nsslapd-maxbersize attribute in cn=config to increase.

AND THIS ERROR GO ON AND ON AND ON..................

Is this all as expected?

Comment 10 Noriko Hosoi 2014-02-17 18:42:22 UTC
(In reply to Amita Sharma from comment #9)
> Default nsslapd-maxbersize is 2MB.  Add an entry larger than 2KB to the

Sorry, Amita.  I first put the wrong unit... :(  Yes, "2MB" and "larger than 2MB" is correct...

> master and run total update.  The consumer should log this error:
>      Incoming BER Element was @@@ bytes, max allowable is ### bytes.
>      Change the nsslapd-maxbersize attribute in cn=config to increase.
> You will probably have to import the large entry into the master using
> ldif2db or something like that, then try a replica init from the master to
> the consumer.
>
> 1. Master - 389
> Consumer - 390
> [root@dhcp201-149 ~]# ldapsearch -D "cn=directory manager" -w Secret123 -p
> 389 -h localhost -b "cn=config" | grep -i "nsslapd-maxbersize"
> nsslapd-maxbersize: 0
>
> 2. Imported data to master which has an entry ::
> [root@dhcp201-149 export]# ls -alh /export/we1.jpg
> -rwxr-xr-x. 1 root root 4.3M Feb 14 14:26 /export/we1.jpg
>
> 3. [root@dhcp201-149 export]# ldif2db.pl -n userRoot -Z dhcp201-149 -D
> "cn=Directory Manager" -w Secret123 -i /export/test.ldif
> Successfully added task entry "cn=import_2014_2_17_13_5_27, cn=import,
> cn=tasks, cn=config"
>
>
> 4. error logs of Master
> ============================
> [17/Feb/2014:13:05:28 +051800] - import userRoot: Import complete.
> Processed 107 entries in 1 seconds. (107.00 entries/sec)
> [17/Feb/2014:13:05:28 +051800] NSMMReplicationPlugin -
> multimaster_be_state_change: replica dc=example,dc=com is coming online;
> enabling replication
> [17/Feb/2014:13:06:17 +051800] NSMMReplicationPlugin - Beginning total
> update of replica "agmt="cn=Master-Consumer" (dhcp201-149:390)".
> [17/Feb/2014:13:06:22 +051800] NSMMReplicationPlugin -
> agmt="cn=Master-Consumer" (dhcp201-149:390): Failed to send extended
> operation: LDAP error -1 (Can't contact LDAP server)
> [17/Feb/2014:13:06:22 +051800] NSMMReplicationPlugin -
> agmt="cn=Master-Consumer" (dhcp201-149:390): Received error -1 (Can't
> contact LDAP server):  for total update operation
> [17/Feb/2014:13:06:23 +051800] NSMMReplicationPlugin -
> agmt="cn=Master-Consumer" (dhcp201-149:390): Warning: unable to send
> endReplication extended operation (Can't contact LDAP server)
>
> 5. error logs of consumer
> ===========================
> [17/Feb/2014:13:06:16 +051800] - WARNING: Import is running with
> nsslapd-db-private-import-mem on; No other process is allowed to access the
> database
> [17/Feb/2014:13:06:17 +051800] connection - conn=12 fd=66 Incoming BER
> Element was 4432141 bytes, max allowable is 2097152 bytes. Change the
> nsslapd-maxbersize attribute in cn=config to increase.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This is what we expect.

> [17/Feb/2014:13:06:17 +051800] - ERROR bulk import abandoned
> [17/Feb/2014:13:06:17 +051800] - import userRoot: Thread monitoring
> returned: -23
>
> [17/Feb/2014:13:06:17 +051800] - import userRoot: Aborting all Import
> threads...
> [17/Feb/2014:13:06:22 +051800] - import userRoot: Import threads aborted.
> [17/Feb/2014:13:06:22 +051800] - import userRoot: Closing files...
> [17/Feb/2014:13:06:22 +051800] - libdb: BDB3028 userRoot/entryrdn.db: unable
> to flush: No such file or directory
> [17/Feb/2014:13:06:22 +051800] - import userRoot: Import failed.
> [17/Feb/2014:13:06:22 +051800] - process_bulk_import_op: NULL target sdn
>
> Is this error as expected on consumer?

Yes, this following error message is what we expect to see.  The rest (e.g., libdb error) is a side effect of import failure.  Import completely regenerates the DB.  So, once it fails, there is no much we can do other than getting the wiped out and/or corrupted DB.  We have to redo consumer initialization with the proper config parameters (e.g., nsslapd-maxbersize larger than 4432141).
> [17/Feb/2014:13:06:17 +051800] connection - conn=12 fd=66 Incoming BER
> Element was 4432141 bytes, max allowable is 2097152 bytes. Change the
> nsslapd-maxbersize attribute in cn=config to increase.

> =============================================================================
> =============================================================================
> ========
> Then, increase nsslapd-maxbersize value to larger than nsslapd-cachememsize.
> Add and entry larger than nsslapd-cachememsize and smaller than
> nsslapd-maxbersize and run total update.  The consumer should log this error:
>      import <backend>: REASON: entry too large (@@@ bytes) for the
>      effective import buffer size (### bytes). Try increasing nsslapd-
>      cachememsize for the backend instance "userRoot".
>
> [root@dhcp201-149 export]# ldapsearch -D "cn=directory manager" -w Secret123
> -p 389 -h localhost -b "cn=config" | grep -i "nsslapd-maxbersize"
> nsslapd-maxbersize: 0
> [root@dhcp201-149 export]# ldapsearch -D "cn=directory manager" -w Secret123
> -p 389 -h localhost -b "cn=config" | grep -i "nsslapd-cachememsize"
> nsslapd-cachememsize: 10485760
>
> 1. increase nsslapd-maxbersize value to larger than nsslapd-cachememsize
> ldapmodify -x -h localhost -p 389 -D "cn=directory manager" -w Secret123 <<
> EOF
> dn: cn=config
> changetype: modify
> replace: nsslapd-maxbersize
> nsslapd-maxbersize: 12500000
> EOF
>
> 2. Add and entry larger than nsslapd-cachememsize and smaller than
> nsslapd-maxbersize and run total update
>
> [root@dhcp201-149 export]# ls -alh sample.jpg
> -rw-r--r--. 1 root root 12M Feb 14 19:29 sample.jpg
>
> [root@dhcp201-149 export]# ldif2db.pl -n userRoot -Z dhcp201-149 -D
> "cn=Directory Manager" -w Secret123 -i /export/test.ldif
> Successfully added task entry "cn=import_2014_2_17_13_53_54, cn=import,
> cn=tasks, cn=config"
>
> Error on Master ::
> [17/Feb/2014:13:53:53 +051800] - Bringing userRoot offline...
> [17/Feb/2014:13:53:54 +051800] NSMMReplicationPlugin -
> multimaster_be_state_change: replica dc=example,dc=com is going offline;
> disabling replication
> [17/Feb/2014:13:53:54 +051800] - WARNING: Import is running with
> nsslapd-db-private-import-mem on; No other process is allowed to access the
> database
> [17/Feb/2014:13:53:54 +051800] - import userRoot: Beginning import job...
> [17/Feb/2014:13:53:54 +051800] - import userRoot: Index buffering enabled
> with bucket size 19
> [17/Feb/2014:13:53:54 +051800] - import userRoot: Processing file
> "/export/test.ldif"
> [17/Feb/2014:13:53:54 +051800] - import userRoot: WARNING: skipping entry
> "cn=Jennifer J Jensen,dc=example,dc=com" ending line 422 of file
> "/export/test.ldif"
> [17/Feb/2014:13:53:54 +051800] - import userRoot: REASON: entry too large
> (12495580 bytes) for the buffer size (8388608 bytes)
> [17/Feb/2014:13:53:55 +051800] - import userRoot: Finished scanning file
> "/export/test.ldif" (16 entries)
> [17/Feb/2014:13:53:55 +051800] - import userRoot: Workers finished; cleaning
> up...
> [17/Feb/2014:13:53:55 +051800] - import userRoot: Workers cleaned up.
> [17/Feb/2014:13:53:55 +051800] - import userRoot: Cleaning up producer
> thread...
> [17/Feb/2014:13:53:55 +051800] - import userRoot: Indexing complete.
> Post-processing...
> [17/Feb/2014:13:53:55 +051800] - import userRoot: Generating numSubordinates
> complete.
> [17/Feb/2014:13:53:55 +051800] - import userRoot: Flushing caches...
> [17/Feb/2014:13:53:55 +051800] - import userRoot: Closing files...
> [17/Feb/2014:13:53:55 +051800] - import userRoot: Import complete.
> Processed 16 entries (1 were skipped) in 1 seconds. (16.00 entries/sec)
> [17/Feb/2014:13:53:55 +051800] NSMMReplicationPlugin -
> multimaster_be_state_change: replica dc=example,dc=com is coming online;
> enabling replication
> [17/Feb/2014:13:54:08 +051800] NSMMReplicationPlugin - Beginning total
> update of replica "agmt="cn=Master-Consumer" (dhcp201-149:390)".
> [17/Feb/2014:13:54:12 +051800] NSMMReplicationPlugin - Finished total update
> of replica "agmt="cn=Master-Consumer" (dhcp201-149:390)". Sent 16 entries.
>
> Error on consumer ::
> [17/Feb/2014:13:54:08 +051800] NSMMReplicationPlugin -
> multimaster_be_state_change: replica dc=example,dc=com is going offline;
> disabling replication
> [17/Feb/2014:13:54:08 +051800] - WARNING: Import is running with
> nsslapd-db-private-import-mem on; No other process is allowed to access the
> database
> [17/Feb/2014:13:54:10 +051800] - import userRoot: Workers finished; cleaning
> up...
> [17/Feb/2014:13:54:11 +051800] - import userRoot: Workers cleaned up.
> [17/Feb/2014:13:54:11 +051800] - import userRoot: Indexing complete.
> Post-processing...
> [17/Feb/2014:13:54:11 +051800] - import userRoot: Generating numSubordinates
> complete.
> [17/Feb/2014:13:54:11 +051800] - import userRoot: Flushing caches...
> [17/Feb/2014:13:54:11 +051800] - import userRoot: Closing files...
> [17/Feb/2014:13:54:11 +051800] - import userRoot: Import complete.
> Processed 16 entries in 3 seconds. (5.33 entries/sec)
> [17/Feb/2014:13:54:11 +051800] NSMMReplicationPlugin -
> multimaster_be_state_change: replica dc=example,dc=com is coming online;
> enabling replication
>
> 3. If I do ldapadd for adding entry larger than nsslapd-cachememsize and
> smaller than nsslapd-maxbersize and run total update
>
> error on Master
> ===================
> [17/Feb/2014:14:02:01 +051800] NSMMReplicationPlugin -
> agmt="cn=Master-Consumer" (dhcp201-149:390): Replication bind with SIMPLE
> auth resumed
> [17/Feb/2014:14:02:01 +051800] NSMMReplicationPlugin -
> agmt="cn=Master-Consumer" (dhcp201-149:390): Failed to send add operation:
> LDAP error -1 (Can't contact LDAP server)
> [17/Feb/2014:14:02:01 +051800] NSMMReplicationPlugin -
> agmt="cn=Master-Consumer" (dhcp201-149:390): Failed to send update operation
> to consumer (uniqueid 9d926401-97ad11e3-83f3cd23-c6c0a6aa, CSN
> 5301c87f000000010000): Can't contact LDAP server. Will retry later.
> [17/Feb/2014:14:02:01 +051800] NSMMReplicationPlugin -
> agmt="cn=Master-Consumer" (dhcp201-149:390): Consumer failed to replay
> change (uniqueid (null), CSN (null)): Can't contact LDAP server(-1). Will
> retry later.
> [17/Feb/2014:14:02:01 +051800] NSMMReplicationPlugin -
> agmt="cn=Master-Consumer" (dhcp201-149:390): Warning: unable to send
> endReplication extended operation (Can't contact LDAP server)
>
> AND THIS ERROR GO ON AND ON AND ON..................
>
> error on consumer
> ====================
> [17/Feb/2014:14:02:06 +051800] connection - conn=52 fd=66 Incoming BER
> Element was 12493804 bytes, max allowable is 2097152 bytes. Change the
> nsslapd-maxbersize attribute in cn=config to increase.
> [17/Feb/2014:14:02:09 +051800] connection - conn=53 fd=66 Incoming BER
> Element was 12493804 bytes, max allowable is 2097152 bytes. Change the
> nsslapd-maxbersize attribute in cn=config to increase.
> [17/Feb/2014:14:02:12 +051800] connection - conn=54 fd=66 Incoming BER
> Element was 12493804 bytes, max allowable is 2097152 bytes. Change the
> nsslapd-maxbersize attribute in cn=config to increase.
> [17/Feb/2014:14:02:15 +051800] connection - conn=55 fd=66 Incoming BER
> Element was 12493804 bytes, max allowable is 2097152 bytes. Change the
> nsslapd-maxbersize attribute in cn=config to increase.
> [17/Feb/2014:14:02:19 +051800] connection - conn=56 fd=66 Incoming BER
> Element was 12493804 bytes, max allowable is 2097152 bytes. Change the
> nsslapd-maxbersize attribute in cn=config to increase.
>
> AND THIS ERROR GO ON AND ON AND ON..................
>
> Is this all as expected?

Well, you set 12500000 to nsslapd-maxbersize.  And your test entry is larget than 12MB.  12MB is larger than 12500000:
  12MB = 12 * 1024 * 1024 = 12582912 > 12500000

> 1. increase nsslapd-maxbersize value to larger than nsslapd-cachememsize
> ldapmodify -x -h localhost -p 389 -D "cn=directory manager" -w Secret123 <<
> EOF
> dn: cn=config
> changetype: modify
> replace: nsslapd-maxbersize
> nsslapd-maxbersize: 12500000
> EOF
>
> [root@dhcp201-149 export]# ls -alh sample.jpg
> -rw-r--r--. 1 root root 12M Feb 14 19:29 sample.jpg

Comment 11 Amita Sharma 2014-02-18 10:12:11 UTC
Thanks for replying Noriko.

Then, increase nsslapd-maxbersize value to larger than nsslapd-cachememsize.  Add and entry larger than nsslapd-cachememsize and smaller than nsslapd-maxbersize and run total update.  The consumer should log this error:
     import <backend>: REASON: entry too large (@@@ bytes) for the
     effective import buffer size (### bytes). Try increasing nsslapd-
     cachememsize for the backend instance "userRoot".

[root@dhcp201-149 export]# ldapsearch -D "cn=directory manager" -w Secret123 -p 389 -h localhost -b "cn=config" | grep -i "nsslapd-maxbersize"
nsslapd-maxbersize: 0
[root@dhcp201-149 export]# ldapsearch -D "cn=directory manager" -w Secret123 -p 389 -h localhost -b "cn=config" | grep -i "nsslapd-cachememsize"
nsslapd-cachememsize: 10485760

1. increase nsslapd-maxbersize value to larger than nsslapd-cachememsize
ldapmodify -x -h localhost -p 389 -D "cn=directory manager" -w Secret123 << EOF
dn: cn=config
changetype: modify
replace: nsslapd-maxbersize
nsslapd-maxbersize: 13000000
EOF

[root@dhcp201-149 export]# ldapsearch -D "cn=directory manager" -w Secret123 -p 389 -h localhost -b "cn=config" | grep -i "nsslapd-maxbersize"
nsslapd-maxbersize: 13000000
[root@dhcp201-149 export]# ldapsearch -D "cn=directory manager" -w Secret123 -p 389 -h localhost -b "cn=config" | grep -i "nsslapd-cachememsize"
nsslapd-cachememsize: 10485760
nsslapd-cachememsize: 10485760

2. Add and entry larger than nsslapd-cachememsize and smaller than nsslapd-maxbersize and run total update

[root@dhcp201-149 export]# ls -alh sample.jpg
-rw-r--r--. 1 root root 12M Feb 14 19:29 sample.jpg

[root@dhcp201-149 ~]# cat /export/a.ldif
# Jennifer's Entry
dn: cn=Jennifer1,dc=example,dc=com
cn: Jennifer J Jensen
objectClass: person
objectClass: inetOrgPerson
sn: Jensen
jpegPhoto:< file:/export/sample.jpg

[root@dhcp201-149 ~]# ldapadd -x -h localhost -p 389 -D "cn=Directory Manager" -w Secret123 -f /export/a.ldif
adding new entry "cn=Jennifer1,dc=example,dc=com"

Consumer Error:
======
[18/Feb/2014:15:12:42 +051800] connection - conn=677 fd=66 Incoming BER Element was 12493804 bytes, max allowable is 2097152 bytes. Change the nsslapd-maxbersize attribute in cn=config to increase.
[18/Feb/2014:15:12:46 +051800] connection - conn=678 fd=66 Incoming BER Element was 12493804 bytes, max allowable is 2097152 bytes. Change the nsslapd-maxbersize attribute in cn=config to increase.
[18/Feb/2014:15:12:50 +051800] connection - conn=679 fd=66 Incoming BER Element was 12493804 bytes, max allowable is 2097152 bytes. Change the nsslapd-maxbersize attribute in cn=config to increase.
[18/Feb/2014:15:12:54 +051800] connection - conn=680 fd=66 Incoming BER Element was 12493804 bytes, max allowable is 2097152 bytes. Change the nsslapd-maxbersize attribute in cn=config to increase.

Master Error
=============
[18/Feb/2014:15:12:42 +051800] NSMMReplicationPlugin - agmt="cn=Master-Consumer" (dhcp201-149:390): Failed to send add operation: LDAP error -1 (Can't contact LDAP server)
[18/Feb/2014:15:12:42 +051800] NSMMReplicationPlugin - agmt="cn=Master-Consumer" (dhcp201-149:390): Failed to send update operation to consumer (uniqueid de0ee201-988011e3-83f3cd23-c6c0a6aa, CSN 53032b0e000000010000): Can't contact LDAP server. Will retry later.
[18/Feb/2014:15:12:42 +051800] NSMMReplicationPlugin - agmt="cn=Master-Consumer" (dhcp201-149:390): Consumer failed to replay change (uniqueid (null), CSN (null)): Can't contact LDAP server(-1). Will retry later.
[18/Feb/2014:15:12:42 +051800] NSMMReplicationPlugin - agmt="cn=Master-Consumer" (dhcp201-149:390): Warning: unable to send endReplication extended operation (Can't contact LDAP server)


I have few questions here
==============================
1. Errors are different in case of ldapadd and ldif2db on Master, is it expected?
2. In the above case, I did not run initialize consumer and errors started coming?
3. These errors are coming continuously and did not stop unless I stop the instance, so it won't risk disk space crunch ?
4. I think the expected error did not occur in above case?

Please guide.

Thanks,
Ami

Comment 12 Rich Megginson 2014-02-18 15:19:00 UTC
(In reply to Amita Sharma from comment #11)
> Thanks for replying Noriko.
> 
> Then, increase nsslapd-maxbersize value to larger than nsslapd-cachememsize.
> Add and entry larger than nsslapd-cachememsize and smaller than
> nsslapd-maxbersize and run total update.  The consumer should log this error:
>      import <backend>: REASON: entry too large (@@@ bytes) for the
>      effective import buffer size (### bytes). Try increasing nsslapd-
>      cachememsize for the backend instance "userRoot".
> 
> [root@dhcp201-149 export]# ldapsearch -D "cn=directory manager" -w Secret123
> -p 389 -h localhost -b "cn=config" | grep -i "nsslapd-maxbersize"
> nsslapd-maxbersize: 0
> [root@dhcp201-149 export]# ldapsearch -D "cn=directory manager" -w Secret123
> -p 389 -h localhost -b "cn=config" | grep -i "nsslapd-cachememsize"
> nsslapd-cachememsize: 10485760
> 
> 1. increase nsslapd-maxbersize value to larger than nsslapd-cachememsize
> ldapmodify -x -h localhost -p 389 -D "cn=directory manager" -w Secret123 <<
> EOF
> dn: cn=config
> changetype: modify
> replace: nsslapd-maxbersize
> nsslapd-maxbersize: 13000000
> EOF
> 
> [root@dhcp201-149 export]# ldapsearch -D "cn=directory manager" -w Secret123
> -p 389 -h localhost -b "cn=config" | grep -i "nsslapd-maxbersize"
> nsslapd-maxbersize: 13000000
> [root@dhcp201-149 export]# ldapsearch -D "cn=directory manager" -w Secret123
> -p 389 -h localhost -b "cn=config" | grep -i "nsslapd-cachememsize"
> nsslapd-cachememsize: 10485760
> nsslapd-cachememsize: 10485760
> 
> 2. Add and entry larger than nsslapd-cachememsize and smaller than
> nsslapd-maxbersize and run total update
> 
> [root@dhcp201-149 export]# ls -alh sample.jpg
> -rw-r--r--. 1 root root 12M Feb 14 19:29 sample.jpg
> 
> [root@dhcp201-149 ~]# cat /export/a.ldif
> # Jennifer's Entry
> dn: cn=Jennifer1,dc=example,dc=com
> cn: Jennifer J Jensen
> objectClass: person
> objectClass: inetOrgPerson
> sn: Jensen
> jpegPhoto:< file:/export/sample.jpg
> 
> [root@dhcp201-149 ~]# ldapadd -x -h localhost -p 389 -D "cn=Directory
> Manager" -w Secret123 -f /export/a.ldif
> adding new entry "cn=Jennifer1,dc=example,dc=com"
> 
> Consumer Error:
> ======
> [18/Feb/2014:15:12:42 +051800] connection - conn=677 fd=66 Incoming BER
> Element was 12493804 bytes, max allowable is 2097152 bytes. Change the
> nsslapd-maxbersize attribute in cn=config to increase.
> [18/Feb/2014:15:12:46 +051800] connection - conn=678 fd=66 Incoming BER
> Element was 12493804 bytes, max allowable is 2097152 bytes. Change the
> nsslapd-maxbersize attribute in cn=config to increase.
> [18/Feb/2014:15:12:50 +051800] connection - conn=679 fd=66 Incoming BER
> Element was 12493804 bytes, max allowable is 2097152 bytes. Change the
> nsslapd-maxbersize attribute in cn=config to increase.
> [18/Feb/2014:15:12:54 +051800] connection - conn=680 fd=66 Incoming BER
> Element was 12493804 bytes, max allowable is 2097152 bytes. Change the
> nsslapd-maxbersize attribute in cn=config to increase.
> 
> Master Error
> =============
> [18/Feb/2014:15:12:42 +051800] NSMMReplicationPlugin -
> agmt="cn=Master-Consumer" (dhcp201-149:390): Failed to send add operation:
> LDAP error -1 (Can't contact LDAP server)
> [18/Feb/2014:15:12:42 +051800] NSMMReplicationPlugin -
> agmt="cn=Master-Consumer" (dhcp201-149:390): Failed to send update operation
> to consumer (uniqueid de0ee201-988011e3-83f3cd23-c6c0a6aa, CSN
> 53032b0e000000010000): Can't contact LDAP server. Will retry later.
> [18/Feb/2014:15:12:42 +051800] NSMMReplicationPlugin -
> agmt="cn=Master-Consumer" (dhcp201-149:390): Consumer failed to replay
> change (uniqueid (null), CSN (null)): Can't contact LDAP server(-1). Will
> retry later.
> [18/Feb/2014:15:12:42 +051800] NSMMReplicationPlugin -
> agmt="cn=Master-Consumer" (dhcp201-149:390): Warning: unable to send
> endReplication extended operation (Can't contact LDAP server)
> 
> 
> I have few questions here
> ==============================
> 1. Errors are different in case of ldapadd and ldif2db on Master, is it
> expected?

Yes.

> 2. In the above case, I did not run initialize consumer and errors started
> coming?

You had already initialized the consumer?

> 3. These errors are coming continuously and did not stop unless I stop the
> instance, so it won't risk disk space crunch ?

Which errors?  The errors on the consumer are coming every 4 seconds?

> 4. I think the expected error did not occur in above case?

Not sure what you mean.  I see this:
[18/Feb/2014:15:12:54 +051800] connection - conn=680 fd=66 Incoming BER Element was 12493804 bytes, max allowable is 2097152 bytes. Change the nsslapd-maxbersize attribute in cn=config to increase.

> 
> Please guide.
> 
> Thanks,
> Ami

Comment 13 Noriko Hosoi 2014-02-18 18:14:02 UTC
I have a question, Ami.

Is this setting on master or consumer or both?  I suppose it's done on consumer.  
>
> 1. increase nsslapd-maxbersize value to larger than nsslapd-cachememsize
> ldapmodify -x -h localhost -p 389 -D "cn=directory manager" -w Secret123 << EOF
> dn: cn=config
> changetype: modify
> replace: nsslapd-maxbersize
> nsslapd-maxbersize: 13000000
> EOF
>
> [root@dhcp201-149 export]# ldapsearch -D "cn=directory manager" -w Secret123 -p 389 -h localhost -b "cn=config" | grep -i "nsslapd-maxbersize"
> nsslapd-maxbersize: 13000000
> [root@dhcp201-149 export]# ldapsearch -D "cn=directory manager" -w Secret123 -p 389 -h localhost -b "cn=config" | grep -i "nsslapd-cachememsize"
> nsslapd-cachememsize: 10485760
> nsslapd-cachememsize: 10485760

Could you double check the size of nsslapd-maxbersize on consumer?  If it is set to 13000000, we should not see this error on consumer...
> [18/Feb/2014:15:12:42 +051800] connection - conn=677 fd=66 Incoming BER
> Element was 12493804 bytes, max allowable is 2097152 bytes. Change the
> nsslapd-maxbersize attribute in cn=config to increase.

Comment 14 Amita Sharma 2014-02-20 06:25:48 UTC
(In reply to Noriko Hosoi from comment #13)
> I have a question, Ami.
> 
> Is this setting on master or consumer or both?  I suppose it's done on
> consumer.  
No Noriko, Settings are on Master only. :( , sorry!
As I mentioned in my comment#9 ::
1. Master - 389
Consumer - 390
So all setting I performed is on Master :(
Let me do it on consumer and try again.....
> >
> > 1. increase nsslapd-maxbersize value to larger than nsslapd-cachememsize
> > ldapmodify -x -h localhost -p 389 -D "cn=directory manager" -w Secret123 << EOF
> > dn: cn=config
> > changetype: modify
> > replace: nsslapd-maxbersize
> > nsslapd-maxbersize: 13000000
> > EOF
> >
> > [root@dhcp201-149 export]# ldapsearch -D "cn=directory manager" -w Secret123 -p 389 -h localhost -b "cn=config" | grep -i "nsslapd-maxbersize"
> > nsslapd-maxbersize: 13000000
> > [root@dhcp201-149 export]# ldapsearch -D "cn=directory manager" -w Secret123 -p 389 -h localhost -b "cn=config" | grep -i "nsslapd-cachememsize"
> > nsslapd-cachememsize: 10485760
> > nsslapd-cachememsize: 10485760
> 
> Could you double check the size of nsslapd-maxbersize on consumer?  If it is
> set to 13000000, we should not see this error on consumer...
> > [18/Feb/2014:15:12:42 +051800] connection - conn=677 fd=66 Incoming BER
> > Element was 12493804 bytes, max allowable is 2097152 bytes. Change the
> > nsslapd-maxbersize attribute in cn=config to increase.

Then, increase nsslapd-maxbersize value to larger than nsslapd-cachememsize.  Add and entry larger than nsslapd-cachememsize and smaller than nsslapd-maxbersize and run total update.  The consumer should log this error:
     import <backend>: REASON: entry too large (@@@ bytes) for the
     effective import buffer size (### bytes). Try increasing nsslapd-
     cachememsize for the backend instance "userRoot".

[root@dhcp201-149 export]# ldapsearch -D "cn=directory manager" -w Secret123 -p 390 -h localhost -b "cn=config" | grep -i "nsslapd-maxbersize"
nsslapd-maxbersize: 0
[root@dhcp201-149 export]# ldapsearch -D "cn=directory manager" -w Secret123 -p 390 -h localhost -b "cn=config" | grep -i "nsslapd-cachememsize"
nsslapd-cachememsize: 10485760

1. increase nsslapd-maxbersize value to larger than nsslapd-cachememsize
ldapmodify -x -h localhost -p 390 -D "cn=directory manager" -w Secret123 << EOF
dn: cn=config
changetype: modify
replace: nsslapd-maxbersize
nsslapd-maxbersize: 13000000
EOF

[root@dhcp201-149 export]# ldapsearch -D "cn=directory manager" -w Secret123 -p 390 -h localhost -b "cn=config" | grep -i "nsslapd-maxbersize"
nsslapd-maxbersize: 13000000
[root@dhcp201-149 export]# ldapsearch -D "cn=directory manager" -w Secret123 -p 390 -h localhost -b "cn=config" | grep -i "nsslapd-cachememsize"
nsslapd-cachememsize: 10485760
nsslapd-cachememsize: 10485760

2. Add and entry larger than nsslapd-cachememsize and smaller than nsslapd-maxbersize and run total update

[root@dhcp201-149 export]# ls -alh sample.jpg
-rw-r--r--. 1 root root 12M Feb 14 19:29 sample.jpg

[root@dhcp201-149 ~]# cat /export/a.ldif
# Jennifer's Entry
dn: cn=Jennifer1,dc=example,dc=com
cn: Jennifer J Jensen
objectClass: person
objectClass: inetOrgPerson
sn: Jensen
jpegPhoto:< file:/export/sample.jpg

[root@dhcp201-149 ~]# ldapadd -x -h localhost -p 389 -D "cn=Directory Manager" -w Secret123 -f /export/a.ldif
adding new entry "cn=Jennifer1,dc=example,dc=com"

After reinitializing consumer ::
Errors on Master >>
======================
[root@dhcp201-149 ~]# tail -f /var/log/dirsrv/slapd-dhcp201-149/errors
[20/Feb/2014:11:22:40 +051800] - import userRoot: Indexing complete.  Post-processing...
[20/Feb/2014:11:22:40 +051800] - import userRoot: Generating numSubordinates complete.
[20/Feb/2014:11:22:41 +051800] - import userRoot: Flushing caches...
[20/Feb/2014:11:22:41 +051800] - import userRoot: Closing files...
[20/Feb/2014:11:22:41 +051800] - import userRoot: Import complete.  Processed 16 entries in 2 seconds. (8.00 entries/sec)
[20/Feb/2014:11:22:41 +051800] NSMMReplicationPlugin - multimaster_be_state_change: replica dc=example,dc=com is coming online; enabling replication
[20/Feb/2014:11:22:41 +051800] NSMMReplicationPlugin - replica_reload_ruv: Warning: new data for replica dc=example,dc=com does not match the data in the changelog.
 Recreating the changelog file. This could affect replication with replica's  consumers in which case the consumers should be reinitialized.
[20/Feb/2014:11:23:35 +051800] NSMMReplicationPlugin - Beginning total update of replica "agmt="cn=Master-Consumer" (dhcp201-149:390)".
[20/Feb/2014:11:23:39 +051800] NSMMReplicationPlugin - Finished total update of replica "agmt="cn=Master-Consumer" (dhcp201-149:390)". Sent 16 entries.

[20/Feb/2014:11:30:21 +051800] NSMMReplicationPlugin - Beginning total update of replica "agmt="cn=Master-Consumer" (dhcp201-149:390)".
[20/Feb/2014:11:30:32 +051800] NSMMReplicationPlugin - agmt="cn=Master-Consumer" (dhcp201-149:390): Received error -1 (Can't contact LDAP server):  for total update operation
[20/Feb/2014:11:30:34 +051800] NSMMReplicationPlugin - agmt="cn=Master-Consumer" (dhcp201-149:390): Warning: unable to send endReplication extended operation (Can't contact LDAP server)
[20/Feb/2014:11:30:34 +051800] NSMMReplicationPlugin - agmt="cn=Master-Consumer" (dhcp201-149:390): Replication bind with SIMPLE auth resumed

Errors on Consumer >>
======================
[20/Feb/2014:11:30:20 +051800] NSMMReplicationPlugin - multimaster_be_state_change: replica dc=example,dc=com is going offline; disabling replication
[20/Feb/2014:11:30:20 +051800] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database
[20/Feb/2014:11:30:25 +051800] - import userRoot: REASON: entry too large (12496516 bytes) for the effective import buffer size (8388608 bytes). Try increasing nsslapd-cachememsize for the backend instance "userRoot".
[20/Feb/2014:11:30:25 +051800] - slapi_start_bulk_import: failed; error = -1
[20/Feb/2014:11:30:25 +051800] - ERROR bulk import abandoned
[20/Feb/2014:11:30:25 +051800] - import userRoot: Thread monitoring returned: -23

PERFECT, marking the bug as VERIFIED.

Comment 17 Sankar Ramalingam 2014-04-30 12:42:19 UTC
Test cases added by Noriko in mmrepl/accept test suite. Trac 47606_1, trac47606_2 and trac47606_3.

Comment 18 Ludek Smid 2014-06-13 10:02:31 UTC
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.


Note You need to log in before you can comment on or make changes to this bug.