Bug 663484 - [RFE] Entry usn plugin fails to properly tag entries on initialization - new config option
Summary: [RFE] Entry usn plugin fails to properly tag entries on initialization - new ...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: 389
Classification: Retired
Component: Server - Plugins
Version: 1.2.7
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ---
Assignee: Noriko Hosoi
QA Contact: Viktor Ashirov
URL:
Whiteboard:
Depends On:
Blocks: 639035
TreeView+ depends on / blocked
 
Reported: 2010-12-15 22:28 UTC by Simo Sorce
Modified: 2015-12-07 17:06 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-12-07 17:06:14 UTC


Attachments (Terms of Use)
git patch file (master) (18.47 KB, patch)
2010-12-18 02:37 UTC, Noriko Hosoi
no flags Details | Diff
git patch file (master) (20.42 KB, patch)
2010-12-21 02:36 UTC, Noriko Hosoi
nkinder: review+
Details | Diff

Description Simo Sorce 2010-12-15 22:28:07 UTC
If you take a working replica and re-initialize it entryusn attributes are not properly created on the new entries being replicated in.
In a directory with 194 entries only 24 of them had a entryusn value.
the entryusn value was also wrapped and restarted from a value lower than what was previously available breaking the promise of getting higher values when entries are changed.

Comment 1 Noriko Hosoi 2010-12-16 00:02:54 UTC
Input from Simo:
<simo> yet it would be nice to get the entryusn attribute on all the records even if it is set to 0 for imported ones
<simo> (my preference would be to have it progressing though)

Currently, entryusn is reset to 0 if an entry already hasn't had the entryusn attribute in the import.
 356 void
 357 import_producer(void *param)
 358 {
 ...
 670         /* 
 671          * Check if entryusn plugin is enabled.  
 672          * If yes, add "entryusn: 0" to the entry 
 673          * if it does not have the attr type .
 674          */

I wonder what entryusn values do the 24 entries have.  0? Or some other larger number?

We can change the behaviour such that
1) it ignores the existing entryusn in the ldif to be imported, and
2) it assigns ascendant digits (as Simo suggested).

One concern is Entry USN has a param nsslapd-entryusn-global, which cannot be supported if you import multiple LDIFs to multiple backends.  Is it still worth assigning incremental numbers?

Comment 2 Simo Sorce 2010-12-16 18:56:51 UTC
We use nsslapd-entryusn-global in IPA so if the incremental option is incompatible with it then it is not usable by an IPA Server.(In reply to comment #1)
> Input from Simo:
> <simo> yet it would be nice to get the entryusn attribute on all the records
> even if it is set to 0 for imported ones
> <simo> (my preference would be to have it progressing though)
> 
> Currently, entryusn is reset to 0 if an entry already hasn't had the entryusn
> attribute in the import.
>  356 void
>  357 import_producer(void *param)
>  358 {
>  ...
>  670         /* 
>  671          * Check if entryusn plugin is enabled.  
>  672          * If yes, add "entryusn: 0" to the entry 
>  673          * if it does not have the attr type .
>  674          */
> 
> I wonder what entryusn values do the 24 entries have.  0? Or some other larger
> number?

It had incremental numers starting from 1 to 24, I suspect the additions where caused by the memberof plugin changing the entries to add the memberof value during import.

> We can change the behaviour such that
> 1) it ignores the existing entryusn in the ldif to be imported, and
> 2) it assigns ascendant digits (as Simo suggested).
> 
> One concern is Entry USN has a param nsslapd-entryusn-global, which cannot be
> supported if you import multiple LDIFs to multiple backends.  Is it still worth
> assigning incremental numbers?

We use nsslapd-entryusn-global in IPA so if the incremental option is incompatible with it then it is not usable by an IPA Server.
If it is too difficult to add incremental numbers during the import it is better to set all entries to just 0.

The main problem we face is that sssd uses entryusn in searches used for enumerations in order to get only items that changed since the last search was performed.
So another way could be to set all entries to the previous highest usn (+1) on import. And then let the plugin increment from there on normal operations.

Comment 3 Dmitri Pal 2010-12-16 21:18:43 UTC
Is there a way to correct the entryUSNs after the import but before the replica gets online in IPA case? Since it is an IPA feature can there be a script/plugin that would run through the entryUSNs and reset them to the right incremental numbers. Can such script/plugin be run as a part of the IPA replica initialization and not as a part of the DS server import?

Comment 4 Simo Sorce 2010-12-16 21:39:17 UTC
Well I guess that technically you could "touch" each entry in the tree after the replica has come in, this will cause all entries to get a new entryUSN.
Sounds a bit ugly but potentially doable.

Although this may cause more issues to clients than actually not doing it come to think, as you may end up with a highest USN higher than the one available before the reinitialization which will cause clients not to throw away the current counter they keep. This in turn may cause some unfortunate clients to not see some changes that happened on other masters while the replica was reinitializing, depending on the order in which entries are "touched".
I guess we could try to touch them sorting by modifiedTimestamp, that should normally cause entries get entryusn roughly in the right order.

Another possibility is that we get a way to set the highest USN.
We could then save the highest one before reinitialization and set the new one to the old value before "touching" all entries. Then the "touch" will guarantee that all entries have USNs that are higher than waht was available earlier.
This may cause clients to basically refresh their entire cache, but they wouldn't miss changes.

Comment 5 Noriko Hosoi 2010-12-16 21:51:53 UTC
For now, there is no way for clients to "set" some specific entryusn value.

For instnace, setting this entryusn value ends up with the "next" entryusn.
ldapmodify -D 'cn=directory manager' -w pw << EOF
dn: uid=BDipace9,ou=People,dc=example,dc=com
changetype: modify    
replace: entryusn 
entryusn: 100000
EOF

ldapsearch ... -D 'cn=directory manager' -w pw -b "ou=people,dc=example,dc=com" "(cn=*)" entryusn
...
dn: uid=BDipace9,ou=People,dc=example,dc=com
entryusn: 10

This proposal could be doable with a little change in the import code.  (Of course, we have to introduce a new config option, though.)
> So another way could be to set all entries to the previous highest usn (+1) on
import. And then let the plugin increment from there on normal operations.

Comment 6 Simo Sorce 2010-12-16 21:59:49 UTC
(In reply to comment #5)
> For now, there is no way for clients to "set" some specific entryusn value.

I was arguing for the ability to set the highest usn value as shown by the plugin in rootdse, not a random entryusn value.

 
> This proposal could be doable with a little change in the import code.  (Of
> course, we have to introduce a new config option, though.)
> > So another way could be to set all entries to the previous highest usn (+1) on
> import. And then let the plugin increment from there on normal operations.

This looks also as a reasonable compromise.

Comment 7 Noriko Hosoi 2010-12-18 02:37:07 UTC
Created attachment 469481 [details]
git patch file (master)

Description:
Introducing a config parameter nsslapd-entryusn-import-initval.
If the value is digit, e.g., 0, 10, 100 ..., the imported entries
will have the entryusn value.
If the value is not digit, e.g., "next", the imported entries
will have the last entryusn + 1 from the database existed before
the import was exected or initialized by the master.

In addition, this patch fixes a minor invalid access to a freed
memory in cl5DeleteDBSync.

Comment 8 Noriko Hosoi 2010-12-18 06:59:07 UTC
Steps to verify

case 1.
no nsslapd-entryusn-import-initval in cn=config.
1-1. server is down
  <server_instance_dir>/ldif2db -n <backend> -i /path/to/ldif
1-2. server is up
  <server_instance_dir>/ldif2db.pl -D 'cn=directory manager' -w <password> -n <backend> -i /path/to/ldif
Result: all entry have "entryusn: 0":
  $ ldapsearch ... -b "dc=example,dc=com" "(objectclass=*)" entryusn
  dn: dc=example,dc=com
  entryusn: 0
  dn: ou=People,dc=example,dc=com
  entryusn: 0
  ...

case 2.
"nsslapd-entryusn-import-initval: 0" in cn=config.
2-1. server is down
  <server_instance_dir>/ldif2db -n <backend> -i /path/to/ldif
2-2. server is up
  <server_instance_dir>/ldif2db.pl -D 'cn=directory manager' -w <password> -n <backend> -i /path/to/ldif
Result: all entry have "entryusn: 0":
  $ ldapsearch ... -b "dc=example,dc=com" "(objectclass=*)" entryusn
  dn: dc=example,dc=com
  entryusn: 0
  dn: ou=People,dc=example,dc=com
  entryusn: 0
  ...

case 3.
"nsslapd-entryusn-import-initval: 8589934592" in cn=config (> 32bit uint).
3-1. server is down
  <server_instance_dir>/ldif2db -n <backend> -i /path/to/ldif
3-2. server is up
  <server_instance_dir>/ldif2db.pl -D 'cn=directory manager' -w <password> -n <backend> -i /path/to/ldif
Result: all entry have "entryusn: 8589934592":
  $ ldapsearch ... -b "dc=example,dc=com" "(objectclass=*)" entryusn
  dn: dc=example,dc=com
  entryusn: 8589934592
  dn: ou=People,dc=example,dc=com
  entryusn: 8589934592
  ...

case 4.
"nsslapd-entryusn-import-initval: next" in cn=config.
4-1. server is down
  <server_instance_dir>/ldif2db -n <backend> -i /path/to/ldif
4-2. server is up
  <server_instance_dir>/ldif2db.pl -D 'cn=directory manager' -w <password> -n <backend> -i /path/to/ldif
Result: all entry have "entryusn: 8589934593" (last entryusn value + 1 in the previous db):
  $ ldapsearch ... -b "dc=example,dc=com" "(objectclass=*)" entryusn
  dn: dc=example,dc=com
  entryusn: 8589934593
  dn: ou=People,dc=example,dc=com
  entryusn: 8589934593
  ...

case 5.
Set up MMR.
Instead of ldif2db[.pl], use replica initialization.
Set nsslapd-entryusn-import-initval as in case 1 through 4 on the replica,
execute replica initialization on the master.  Check entryusn on the replica.
Note: when "nsslapd-entryusn-import-initval: next" is set, the entryusn value
could be larger than (previous entryusn + 1) due to the internal update
operations executed by the replication plug-in.

Comment 9 Noriko Hosoi 2010-12-21 02:36:10 UTC
Created attachment 469889 [details]
git patch file (master)

Description:
1) Introducing a config parameter nsslapd-entryusn-import-initval.
If the value is digit, e.g., 0, 10, 100 ..., the imported entries
will have the entryusn value.
If the value is not digit, e.g., "next", the imported entries
will have the last entryusn + 1 from the database existed before
the import was exected or initialized by the master.

2) Internal OID LDBM_ENTRYUSN_OID has been redifined in back-ldbm.h
to avoid a possible conflict.

3) If an attribute to modify is an operational attribute and
that is the only one modify operation (not one of the series
of modify operations), the server returns UNWILLING_TO_PERFORM.

In addition, this patch fixes a minor invalid access to a freed
memory in cl5DeleteDBSync.

Comment 10 Noriko Hosoi 2011-01-03 23:09:24 UTC
Reviewed by Nathan (Thank you!!!)

Pushed to master.
$ git merge 663484
Updating 53c948c..196f1ef
Fast-forward
 ldap/servers/plugins/replication/cl5_api.c    |    9 ++-
 ldap/servers/slapd/back-ldbm/back-ldbm.h      |    4 +-
 ldap/servers/slapd/back-ldbm/import-threads.c |  117 +++++++++++++++++++------
 ldap/servers/slapd/back-ldbm/import.h         |    1 +
 ldap/servers/slapd/back-ldbm/ldbm_usn.c       |    3 +-
 ldap/servers/slapd/back-ldbm/ldif2ldbm.c      |   22 +++++
 ldap/servers/slapd/libglobs.c                 |   35 ++++++++
 ldap/servers/slapd/modify.c                   |    7 +-
 ldap/servers/slapd/proto-slap.h               |    2 +
 ldap/servers/slapd/slap.h                     |    4 +-
 10 files changed, 168 insertions(+), 36 deletions(-)

$ git push
Counting objects: 35, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (18/18), done.
Writing objects: 100% (18/18), 3.92 KiB, done.
Total 18 (delta 16), reused 0 (delta 0)
To ssh://git.fedorahosted.org/git/389/ds.git
   53c948c..196f1ef  master -> master

Comment 13 Amita Sharma 2011-06-07 12:24:44 UTC
dn: cn=USN,cn=plugins,cn=config
objectClass: top
objectClass: nsSlapdPlugin
objectClass: extensibleObject
cn: USN
nsslapd-pluginPath: libusn-plugin
nsslapd-pluginInitfunc: usn_init
nsslapd-pluginType: object
nsslapd-pluginEnabled: on
nsslapd-plugin-depends-on-type: database
and
nsslapd-entryusn-global: on

Now Case 1
After Import, ldapsearch returns
================================
# LTrudell7207, Accounting, example.com
dn: uid=LTrudell7207,ou=Accounting,dc=example,dc=com
entryusn: 7220

# GCogdell7208, Accounting, example.com
dn: uid=GCogdell7208,ou=Accounting,dc=example,dc=com
entryusn: 7221

# BReaves7209, Human Resources, example.com
dn: uid=BReaves7209,ou=Human Resources,dc=example,dc=com
entryusn: 7222

# KTamarell7210, Accounting, example.com
dn: uid=KTamarell7210,ou=Accounting,dc=example,dc=com
entryusn: 7223

# RTanchak7211, Payroll, example.com
dn: uid=RTanchak7211,ou=Payroll,dc=example,dc=com
entryusn: 7224

# JSchoch7212, Product Testing, example.com
dn: uid=JSchoch7212,ou=Product Testing,dc=example,dc=com
entryusn: 7225

case 2
nsslapd-entryusn-global: on
nsslapd-entryusn-import-initval: 0

After Import, ldapsearch returns
================================
[root@rhel61 slapd-rhel61]# ldapsearch -x -h localhost -p 389 -D "cn=directory manager" -w Secret123 -b "ou=people,dc=example,dc=com" "(objectclass=*)" entryusn
# extended LDIF
#
# LDAPv3
# base <ou=people,dc=example,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: entryusn 
#

# people, example.com
dn: ou=people,dc=example,dc=com
entryusn: 7

# TVradmin0, people, example.com
dn: uid=TVradmin0,ou=people,dc=example,dc=com
entryusn: 7346

# VLeBaron1, people, example.com
dn: uid=VLeBaron1,ou=people,dc=example,dc=com
entryusn: 7347

# JOshinski2, people, example.com
dn: uid=JOshinski2,ou=people,dc=example,dc=com
entryusn: 7348

# search result
search: 2
result: 0 Success

# numResponses: 5
# numEntries: 4

case 3 :
nsslapd-entryusn-import-initval: 8589934592
After Import, ldapsearch returns
================================
I removed these entries thn again imported but still same result.
[root@rhel61 slapd-rhel61]# ldapsearch -x -h localhost -p 389 -D "cn=directory manager" -w Secret123 -b "ou=people,dc=example,dc=com" "(objectclass=*)" entryusn
# extended LDIF
#
# LDAPv3
# base <ou=people,dc=example,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: entryusn 
#

# people, example.com
dn: ou=people,dc=example,dc=com
entryusn: 7

# TVradmin0, people, example.com
dn: uid=TVradmin0,ou=people,dc=example,dc=com
entryusn: 7353

# VLeBaron1, people, example.com
dn: uid=VLeBaron1,ou=people,dc=example,dc=com
entryusn: 7354

# JOshinski2, people, example.com
dn: uid=JOshinski2,ou=people,dc=example,dc=com
entryusn: 7355

# search result
search: 2
result: 0 Success



NOTE: I am running the import from the ds-console.. not sure wht is the issue with ldif2db. the same ldif I am able to prcess from ds-console.
I hope that should not make any difference.
[root@rhel61 slapd-rhel61]# ./ldif2db.pl -D 'cn=directory manager' -w Secret123 -n userRoot -i /home/example10.ldif
adding new entry "cn=import_2011_6_7_17_48_49, cn=import, cn=tasks, cn=config"

[root@rhel61 slapd-rhel61]# service dirsrv stop
Shutting down dirsrv: 
    rhel61...                                              [  OK  ]
[root@rhel61 slapd-rhel61]# ./ldif2db -n userRoot -i /home/example10
example100k.ldif  example10.ldif    
[root@rhel61 slapd-rhel61]# ./ldif2db -n userRoot -i /home/example10.ldif 
importing data ...
[07/Jun/2011:17:49:39 +051800] - I'm resizing my cache now...cache was 20000000 and is now 8000000
[07/Jun/2011:17:49:40 +051800] - All database threads now stopped
[07/Jun/2011:17:49:40 +051800] - I'm resizing my cache now...cache was 20000000 and is now 8000000
[07/Jun/2011:17:49:40 +051800] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database
[07/Jun/2011:17:49:40 +051800] - check_and_set_import_cache: pagesize: 4096, pages: 125898, procpages: 48953
[07/Jun/2011:17:49:40 +051800] - WARNING: After allocating import cache 201436KB, the available memory is 302156KB, which is less than the soft limit 1048576KB. You may want to decrease the import cache size and rerun import.
[07/Jun/2011:17:49:40 +051800] - Import allocates 201436KB import cache.
[07/Jun/2011:17:49:40 +051800] - import userRoot: Beginning import job...
[07/Jun/2011:17:49:40 +051800] - import userRoot: Index buffering enabled with bucket size 100
[07/Jun/2011:17:49:40 +051800] - import userRoot: Processing file "/home/example10.ldif"
[07/Jun/2011:17:49:40 +051800] - import userRoot: Finished scanning file "/home/example10.ldif" (0 entries)
[07/Jun/2011:17:49:40 +051800] - import userRoot: Workers finished; cleaning up...
[07/Jun/2011:17:49:41 +051800] - import userRoot: Workers cleaned up.
[07/Jun/2011:17:49:41 +051800] - import userRoot: Cleaning up producer thread...
[07/Jun/2011:17:49:41 +051800] - import userRoot: Indexing complete.  Post-processing...
[07/Jun/2011:17:49:41 +051800] - Nothing to do to build ancestorid index
[07/Jun/2011:17:49:41 +051800] - import userRoot: Flushing caches...
[07/Jun/2011:17:49:41 +051800] - import userRoot: Closing files...
[07/Jun/2011:17:49:41 +051800] - All database threads now stopped
[07/Jun/2011:17:49:41 +051800] - import userRoot: Import complete.  Processed 0 entries in 1 seconds. (0.00 entries/sec)

Sending you the machine info in mail.

Comment 14 Noriko Hosoi 2011-06-07 17:52:24 UTC
Let's make things straight one by one...

1. Your server slapd-rhel61 has 2 backends (except o=netscaperoot):
   Suffix: dc=pnq,dc=redhat,dc=com; backend: userRoot
   Suffix: dc=example,dc=com; backend: exampledb
   When you import, the entries in the ldif file are successfully imported
   if the given backend name matches.  That's said, if you import entries 
   under "dc=example,dc=com", you have to specify "-n exampledb" in the ldif2db
   command line (instead of "userRoot").

2. I'm afraid you cannot use "Import databases" task (on DS Console | Tasks)
   for this case.  If you take a look at the access log, you could see
   this "import" uses ordinary LDAP ADD operation, not the real import.
   [..] conn=1 op=263 ADD dn="uid=YLucas7,ou=People,dc=example,dc=com"
   [..] conn=1 op=263 RESULT err=0 tag=105 nentries=0 etime=0
   [..]
   This bug needs to be verified using import (ldif2db*).
   To do so, you could use ldif2db or ldif2db.pl command line tool, or
   DS Console | Configuration | Data, then expand the suffix and choose its
   backend (e.g., exampledb).  Right click there brings up a menu.  Choose
   initialize database.  Type full path to your ldif file and click OK.
   It calls real import "ldif2db" in the server.  You could verify it with
   the logs in the errors log:
   [..] - Bringing userRoot offline...
   [..] - userRoot: Cleaning up entry cache
   [..] - userRoot: Cleaning up dn cache
   [..] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database
   [..] - import userRoot: Beginning import job...
   [..]

Comment 15 Amita Sharma 2011-06-09 08:00:28 UTC
case 1.
===============
no nsslapd-entryusn-import-initval in cn=config.
1-1. server is down
  <server_instance_dir>/ldif2db -n <backend> -i /path/to/ldif
1-2. server is up
  <server_instance_dir>/ldif2db.pl -D 'cn=directory manager' -w <password> -n
<backend> -i /path/to/ldif
Result: all entry have "entryusn: 0":
  $ ldapsearch ... -b "dc=example,dc=com" "(objectclass=*)" entryusn
  dn: dc=example,dc=com
  entryusn: 0
  dn: ou=People,dc=example,dc=com
  entryusn: 0
  ...

[root@rhel61 slapd-rhel61]# ./ldif2db -n exampledb -i /home/example10.ldif 
importing data ...
[09/Jun/2011:12:51:27 +051800] - All database threads now stopped
[09/Jun/2011:12:51:27 +051800] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database
[09/Jun/2011:12:51:27 +051800] - check_and_set_import_cache: pagesize: 4096, pages: 125898, procpages: 48954
[09/Jun/2011:12:51:27 +051800] - WARNING: After allocating import cache 201436KB, the available memory is 302156KB, which is less than the soft limit 1048576KB. You may want to decrease the import cache size and rerun import.
[09/Jun/2011:12:51:27 +051800] - Import allocates 201436KB import cache.
[09/Jun/2011:12:51:28 +051800] - import exampledb: Beginning import job...
[09/Jun/2011:12:51:28 +051800] - import exampledb: Index buffering enabled with bucket size 100
[09/Jun/2011:12:51:28 +051800] - import exampledb: Processing file "/home/example10.ldif"
[09/Jun/2011:12:51:28 +051800] - import exampledb: Finished scanning file "/home/example10.ldif" (15 entries)
[09/Jun/2011:12:51:28 +051800] - import exampledb: Workers finished; cleaning up...
[09/Jun/2011:12:51:29 +051800] - import exampledb: Workers cleaned up.
[09/Jun/2011:12:51:29 +051800] - import exampledb: Cleaning up producer thread...
[09/Jun/2011:12:51:29 +051800] - import exampledb: Indexing complete.  Post-processing...
[09/Jun/2011:12:51:29 +051800] - import exampledb: Flushing caches...
[09/Jun/2011:12:51:29 +051800] - import exampledb: Closing files...
[09/Jun/2011:12:51:29 +051800] - All database threads now stopped
[09/Jun/2011:12:51:29 +051800] - import exampledb: Import complete.  Processed 15 entries in 1 seconds. (15.00 entries/sec)
[root@rhel61 slapd-rhel61]# service dirsrv start
Starting dirsrv: 
    rhel61...                                              [  OK  ]
[root@rhel61 slapd-rhel61]# ldapsearch -x -h localhost -p 389 -D "cn=directory manager" -w Secret123 -b "dc=example,dc=com" "(objectclass=*)" entryusn
# extended LDIF
#
# LDAPv3
# base <dc=example,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: entryusn 
#

# example.com
dn: dc=example,dc=com
entryusn: 0

# Accounting, example.com
dn: ou=Accounting,dc=example,dc=com
entryusn: 0

# Product Development, example.com
dn: ou=Product Development,dc=example,dc=com
entryusn: 0

# Product Testing, example.com
dn: ou=Product Testing,dc=example,dc=com
entryusn: 0

# Human Resources, example.com
dn: ou=Human Resources,dc=example,dc=com
entryusn: 0

# Payroll, example.com
dn: ou=Payroll,dc=example,dc=com
entryusn: 0


================================================================================
case 2.
=======
"nsslapd-entryusn-import-initval: 0" in cn=config.
2-1. server is down
  <server_instance_dir>/ldif2db -n <backend> -i /path/to/ldif
2-2. server is up
  <server_instance_dir>/ldif2db.pl -D 'cn=directory manager' -w <password> -n
<backend> -i /path/to/ldif
Result: all entry have "entryusn: 0":
  $ ldapsearch ... -b "dc=example,dc=com" "(objectclass=*)" entryusn
  dn: dc=example,dc=com
  entryusn: 0
  dn: ou=People,dc=example,dc=com
  entryusn: 0
  ...

nsslapd-entryusn-global: on
nsslapd-accesslog-maxlogsize: 100
nsslapd-entryusn-import-initval: 0
nsslapd-accesslog-logrotationtime: 1


[root@rhel61 slapd-rhel61]# service dirsrv stop
Shutting down dirsrv: 
    rhel61...                                              [  OK  ]
[root@rhel61 slapd-rhel61]# ./ldif2db -n exampledb -i /home/example10.ldif 
importing data ...
[09/Jun/2011:12:54:33 +051800] - All database threads now stopped
[09/Jun/2011:12:54:33 +051800] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database
[09/Jun/2011:12:54:33 +051800] - check_and_set_import_cache: pagesize: 4096, pages: 125898, procpages: 48954
[09/Jun/2011:12:54:33 +051800] - WARNING: After allocating import cache 201436KB, the available memory is 302156KB, which is less than the soft limit 1048576KB. You may want to decrease the import cache size and rerun import.
[09/Jun/2011:12:54:33 +051800] - Import allocates 201436KB import cache.
[09/Jun/2011:12:54:33 +051800] - import exampledb: Beginning import job...
[09/Jun/2011:12:54:33 +051800] - import exampledb: Index buffering enabled with bucket size 100
[09/Jun/2011:12:54:34 +051800] - import exampledb: Processing file "/home/example10.ldif"
[09/Jun/2011:12:54:34 +051800] - import exampledb: Finished scanning file "/home/example10.ldif" (15 entries)
[09/Jun/2011:12:54:34 +051800] - import exampledb: Workers finished; cleaning up...
[09/Jun/2011:12:54:35 +051800] - import exampledb: Workers cleaned up.
[09/Jun/2011:12:54:35 +051800] - import exampledb: Cleaning up producer thread...
[09/Jun/2011:12:54:35 +051800] - import exampledb: Indexing complete.  Post-processing...
[09/Jun/2011:12:54:35 +051800] - import exampledb: Flushing caches...
[09/Jun/2011:12:54:35 +051800] - import exampledb: Closing files...
[09/Jun/2011:12:54:35 +051800] - All database threads now stopped
[09/Jun/2011:12:54:35 +051800] - import exampledb: Import complete.  Processed 15 entries in 2 seconds. (7.50 entries/sec)
[root@rhel61 slapd-rhel61]# ldapsearch -x -h localhost -p 389 -D "cn=directory manager" -w Secret123 -b "dc=example,dc=com" "(objectclass=*)" entryusn
ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)
[root@rhel61 slapd-rhel61]# service dirsrv start
Starting dirsrv: 
    rhel61...                                              [  OK  ]
[root@rhel61 slapd-rhel61]# ldapsearch -x -h localhost -p 389 -D "cn=directory manager" -w Secret123 -b "dc=example,dc=com" "(objectclass=*)" entryusn
# extended LDIF
#
# LDAPv3
# base <dc=example,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: entryusn 
#

# example.com
dn: dc=example,dc=com
entryusn: 0

# Accounting, example.com
dn: ou=Accounting,dc=example,dc=com
entryusn: 0

# Product Development, example.com
dn: ou=Product Development,dc=example,dc=com
entryusn: 0

# Product Testing, example.com
dn: ou=Product Testing,dc=example,dc=com
entryusn: 0

# Human Resources, example.com
dn: ou=Human Resources,dc=example,dc=com
entryusn: 0

# Payroll, example.com
dn: ou=Payroll,dc=example,dc=com
entryusn: 0


================================================================================
case 3.
=======
"nsslapd-entryusn-import-initval: 8589934592" in cn=config (> 32bit uint).
3-1. server is down
  <server_instance_dir>/ldif2db -n <backend> -i /path/to/ldif
3-2. server is up
  <server_instance_dir>/ldif2db.pl -D 'cn=directory manager' -w <password> -n
<backend> -i /path/to/ldif
Result: all entry have "entryusn: 8589934592":
  $ ldapsearch ... -b "dc=example,dc=com" "(objectclass=*)" entryusn
  dn: dc=example,dc=com
  entryusn: 8589934592
  dn: ou=People,dc=example,dc=com
  entryusn: 8589934592
  ...

[root@rhel61 slapd-rhel61]# ./ldif2db -n exampledb -i /home/example10.ldif 
importing data ...
[09/Jun/2011:12:43:29 +051800] - I'm resizing my cache now...cache was 206270464 and is now 8000000
[09/Jun/2011:12:43:29 +051800] - All database threads now stopped
[09/Jun/2011:12:43:29 +051800] - I'm resizing my cache now...cache was 206270464 and is now 8000000
[09/Jun/2011:12:43:29 +051800] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database
[09/Jun/2011:12:43:29 +051800] - check_and_set_import_cache: pagesize: 4096, pages: 125898, procpages: 48954
[09/Jun/2011:12:43:29 +051800] - WARNING: After allocating import cache 201436KB, the available memory is 302156KB, which is less than the soft limit 1048576KB. You may want to decrease the import cache size and rerun import.
[09/Jun/2011:12:43:29 +051800] - Import allocates 201436KB import cache.
[09/Jun/2011:12:43:29 +051800] - import exampledb: Beginning import job...
[09/Jun/2011:12:43:29 +051800] - import exampledb: Index buffering enabled with bucket size 100
[09/Jun/2011:12:43:30 +051800] - import exampledb: Processing file "/home/example10.ldif"
[09/Jun/2011:12:43:30 +051800] - import exampledb: Finished scanning file "/home/example10.ldif" (15 entries)
[09/Jun/2011:12:43:31 +051800] - import exampledb: Workers finished; cleaning up...
[09/Jun/2011:12:43:31 +051800] - import exampledb: Workers cleaned up.
[09/Jun/2011:12:43:31 +051800] - import exampledb: Cleaning up producer thread...
[09/Jun/2011:12:43:31 +051800] - import exampledb: Indexing complete.  Post-processing...
[09/Jun/2011:12:43:31 +051800] - import exampledb: Flushing caches...
[09/Jun/2011:12:43:31 +051800] - import exampledb: Closing files...
[09/Jun/2011:12:43:31 +051800] - All database threads now stopped
[09/Jun/2011:12:43:31 +051800] - import exampledb: Import complete.  Processed 15 entries in 2 seconds. (7.50 entries/sec)
[root@rhel61 slapd-rhel61]# service dirsrv start
Starting dirsrv: 
    rhel61...                                              [  OK  ]


[root@rhel61 slapd-rhel61]# ldapsearch -x -h localhost -p 389 -D "cn=directory manager" -w Secret123 -b "dc=example,dc=com" "(objectclass=*)" entryusn
# extended LDIF
#
# LDAPv3
# base <dc=example,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: entryusn 
#

# example.com
dn: dc=example,dc=com
entryusn: 8589934592

# Accounting, example.com
dn: ou=Accounting,dc=example,dc=com
entryusn: 8589934592

# Product Development, example.com
dn: ou=Product Development,dc=example,dc=com
entryusn: 8589934592

# Product Testing, example.com
dn: ou=Product Testing,dc=example,dc=com
entryusn: 8589934592

# Human Resources, example.com
dn: ou=Human Resources,dc=example,dc=com
entryusn: 8589934592

# Payroll, example.com
dn: ou=Payroll,dc=example,dc=com
entryusn: 8589934592

# TVradmin0, Accounting, example.com
dn: uid=TVradmin0,ou=Accounting,dc=example,dc=com
entryusn: 8589934592

================================================================================
case 4.
========
"nsslapd-entryusn-import-initval: next" in cn=config.
4-1. server is down
  <server_instance_dir>/ldif2db -n <backend> -i /path/to/ldif
4-2. server is up
  <server_instance_dir>/ldif2db.pl -D 'cn=directory manager' -w <password> -n
<backend> -i /path/to/ldif
Result: all entry have "entryusn: 8589934593" (last entryusn value + 1 in the
previous db):
  $ ldapsearch ... -b "dc=example,dc=com" "(objectclass=*)" entryusn
  dn: dc=example,dc=com
  entryusn: 8589934593
  dn: ou=People,dc=example,dc=com
  entryusn: 8589934593
  ...

Note: This case I have executed after case number 2 where the entryUSN was "ZERO"
[root@rhel61 slapd-rhel61]# ldapsearch -x -h localhost -p 389 -D "cn=directory manager" -w Secret123 -b "dc=example,dc=com" "(objectclass=*)" entryusn
# extended LDIF
#
# LDAPv3
# base <dc=example,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: entryusn 
#

# example.com
dn: dc=example,dc=com
entryusn: 7391

# Accounting, example.com
dn: ou=Accounting,dc=example,dc=com
entryusn: 7391

# Product Development, example.com
dn: ou=Product Development,dc=example,dc=com
entryusn: 7391

# Product Testing, example.com
dn: ou=Product Testing,dc=example,dc=com
entryusn: 7391

# Human Resources, example.com
dn: ou=Human Resources,dc=example,dc=com
entryusn: 7391


================================================================================
case 5.
========
Set up MMR.
Instead of ldif2db[.pl], use replica initialization.
Set nsslapd-entryusn-import-initval as in case 1 through 4 on the replica,
execute replica initialization on the master.  Check entryusn on the replica.
Note: when "nsslapd-entryusn-import-initval: next" is set, the entryusn value
could be larger than (previous entryusn + 1) due to the internal update
operations executed by the replication plug-in.

==========
MMR case 1
==========
 ldapmodify -D "cn=directory manager" -w Secret123 -p 20100 -x -h localhost << EOF
> dn: cn=20100_to_20103,cn=replica,cn=dc\3Dreplsuffix\2Cdc\3Dcom,cn=mapping tree,cn=config
> changetype: modify
> replace: nsds5beginreplicarefresh
> nsds5beginreplicarefresh: start
> EOF
modifying entry "cn=20100_to_20103,cn=replica,cn=dc\3Dreplsuffix\2Cdc\3Dcom,cn=mapping tree,cn=config"


[root@rhel61-ds90-amita ~]# ldapsearch -x -p 20102 -h localhost -D "cn=Directory Manager" -w Secret123 -b "uid=amsharma111,ou=people,dc=replsuffix,dc=com" "(objectclass=*)" entryusn
# extended LDIF
#
# LDAPv3
# base <uid=amsharma111,ou=people,dc=replsuffix,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: entryusn 
#

# amsharma111, People, replsuffix.com
dn: uid=amsharma111,ou=People,dc=replsuffix,dc=com
entryusn: 0

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1

============
MMR case 2
==========
"nsslapd-entryusn-import-initval: 0" in cn=config.

nsslapd-entryusn-global: on
nsslapd-entryusn-import-initval: 0


[root@rhel61-ds90-amita ~]# /usr/lib64/dirsrv/slapd-M1/stop-slapd 
[root@rhel61-ds90-amita ~]# vim /etc/dirsrv/slapd-M1/dse.ldif
[root@rhel61-ds90-amita ~]# /usr/lib64/dirsrv/slapd-M1/start-slapd 
[root@rhel61-ds90-amita ~]# ldapmodify -D "cn=directory manager" -w Secret123 -p 20100 -x -h localhost << EOF
dn: cn=20100_to_20103,cn=replica,cn=dc\3Dreplsuffix\2Cdc\3Dcom,cn=mapping tree,cn=config
changetype: modify
replace: nsds5beginreplicarefresh
nsds5beginreplicarefresh: start
EOF

modifying entry "cn=20100_to_20103,cn=replica,cn=dc\3Dreplsuffix\2Cdc\3Dcom,cn=mapping tree,cn=config"

[root@rhel61-ds90-amita ~]# ldapsearch -x -p 20102 -h localhost -D "cn=Directory Manager" -w Secret123 -b "uid=amsharma111,ou=people,dc=replsuffix,dc=com" "(objectclass=*)" entryusn
# extended LDIF
#
# LDAPv3
# base <uid=amsharma111,ou=people,dc=replsuffix,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: entryusn 
#

# amsharma111, People, replsuffix.com
dn: uid=amsharma111,ou=People,dc=replsuffix,dc=com
entryusn: 0

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1


=========
MMR case 3
==========
"nsslapd-entryusn-import-initval: 8589934592" in cn=config (> 32bit uint).
[root@rhel61-ds90-amita ~]# /usr/lib64/dirsrv/slapd-M1/stop-slapd 
[root@rhel61-ds90-amita ~]# vim /etc/dirsrv/slapd-M1/dse.ldif
[root@rhel61-ds90-amita ~]# /usr/lib64/dirsrv/slapd-M1/start-slapd 
[root@rhel61-ds90-amita ~]# ldapmodify -D "cn=directory manager" -w Secret123 -p 20100 -x -h localhost << EOFdn: cn=20100_to_20103,cn=replica,cn=dc\3Dreplsuffix\2Cdc\3Dcom,cn=mapping tree,cn=config
changetype: modify
replace: nsds5beginreplicarefresh
nsds5beginreplicarefresh: start
EOF

modifying entry "cn=20100_to_20103,cn=replica,cn=dc\3Dreplsuffix\2Cdc\3Dcom,cn=mapping tree,cn=config"

[root@rhel61-ds90-amita ~]# ldapsearch -x -p 20102 -h localhost -D "cn=Directory Manager" -w Secret123 -b "uid=amsharma111,ou=people,dc=replsuffix,dc=com" "(objectclass=*)" entryusn
# extended LDIF
#
# LDAPv3
# base <uid=amsharma111,ou=people,dc=replsuffix,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: entryusn 
#

# search result
search: 2
result: 10 Referral
matchedDN: dc=replsuffix,dc=com
ref: ldap://rhel61-ds90-amita.idm.lab.bos.redhat.com:20100
ref: ldap://rhel61-ds90-amita.idm.lab.bos.redhat.com:20106
ref: ldap://rhel61-ds90-amita.idm.lab.bos.redhat.com:20104

# numResponses: 1
[root@rhel61-ds90-amita ~]# ldapsearch -x -p 20102 -h localhost -D "cn=Directory Manager" -w Secret123 -b "uid=amsharma111,ou=people,dc=replsuffix,dc=com" "(objectclass=*)" entryusn
# extended LDIF
#
# LDAPv3
# base <uid=amsharma111,ou=people,dc=replsuffix,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: entryusn 
#

# amsharma111, People, replsuffix.com
dn: uid=amsharma111,ou=People,dc=replsuffix,dc=com
entryusn: 0

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1
[root@rhel61-ds90-amita ~]# ldapadd -x -h localhost -p 20100 -D "cn=Directory Manager" -w Secret123  << EOFdn: uid=amsharma112,ou=people,dc=replsuffix,dc=com
cn: ams
sn: ams
givenname: ams
objectclass: top
objectclass: person
objectclass: organizationalPerson
objectclass: inetOrgPerson
uid: ams
mail: ams@example.com
userpassword: amsamsams
EOF

adding new entry "uid=amsharma112,ou=people,dc=replsuffix,dc=com"

[root@rhel61-ds90-amita ~]# ldapsearch -x -p 20102 -h localhost -D "cn=Directory Manager" -w Secret123 -b "uid=amsharma112,ou=people,dc=replsuffix,dc=com" "(objectclass=*)" entryusn
# extended LDIF
#
# LDAPv3
# base <uid=amsharma112,ou=people,dc=replsuffix,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: entryusn 
#

# amsharma112, People, replsuffix.com
dn: uid=amsharma112,ou=People,dc=replsuffix,dc=com
entryusn: 3

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1
[root@rhel61-ds90-amita ~]# ldapmodify -D "cn=directory manager" -w Secret123 -p 20100 -x -h localhost << EOFdn: cn=20100_to_20103,cn=replica,cn=dc\3Dreplsuffix\2Cdc\3Dcom,cn=mapping tree,cn=config
changetype: modify
replace: nsds5beginreplicarefresh
nsds5beginreplicarefresh: start
EOF

modifying entry "cn=20100_to_20103,cn=replica,cn=dc\3Dreplsuffix\2Cdc\3Dcom,cn=mapping tree,cn=config"

[root@rhel61-ds90-amita ~]# ldapsearch -x -p 20102 -h localhost -D "cn=Directory Manager" -w Secret123 -b "uid=amsharma112,ou=people,dc=replsuffix,dc=com" "(objectclass=*)" entryusn
# extended LDIF
#
# LDAPv3
# base <uid=amsharma112,ou=people,dc=replsuffix,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: entryusn 
#

# search result
search: 2
result: 10 Referral
matchedDN: dc=replsuffix,dc=com
ref: ldap://rhel61-ds90-amita.idm.lab.bos.redhat.com:20100
ref: ldap://rhel61-ds90-amita.idm.lab.bos.redhat.com:20106
ref: ldap://rhel61-ds90-amita.idm.lab.bos.redhat.com:20104

# numResponses: 1
[root@rhel61-ds90-amita ~]# ldapsearch -x -p 20102 -h localhost -D "cn=Directory Manager" -w Secret123 -b "uid=amsharma112,ou=people,dc=replsuffix,dc=com" "(objectclass=*)" entryusn
# extended LDIF
#
# LDAPv3
# base <uid=amsharma112,ou=people,dc=replsuffix,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: entryusn 
#

# amsharma112, People, replsuffix.com
dn: uid=amsharma112,ou=People,dc=replsuffix,dc=com
entryusn: 0

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1
[root@rhel61-ds90-amita ~]# ldapsearch -x -p 20102 -h localhost -D "cn=Directory Manager" -w Secret123 -b "uid=amsharma112,ou=people,dc=replsuffix,dc=com" "(objectclass=*)" entryusn
# extended LDIF
#
# LDAPv3
# base <uid=amsharma112,ou=people,dc=replsuffix,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: entryusn 
#

# amsharma112, People, replsuffix.com
dn: uid=amsharma112,ou=People,dc=replsuffix,dc=com
entryusn: 0

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1

===========
MMR Case 4
=========
"nsslapd-entryusn-import-initval: next" in cn=config.
[root@rhel61-ds90-amita ~]# vim /etc/dirsrv/slapd-M1/dse.ldif
[root@rhel61-ds90-amita ~]# /usr/lib64/dirsrv/slapd-M1/stop-slapd 
[root@rhel61-ds90-amita ~]# vim /etc/dirsrv/slapd-M1/dse.ldif
[root@rhel61-ds90-amita ~]# /usr/lib64/dirsrv/slapd-M1/start-slapd 
[root@rhel61-ds90-amita ~]# ldapmodify -D "cn=directory manager" -w Secret123 -p 20100 -x -h localhost << EOFdn: cn=20100_to_20103,cn=replica,cn=dc\3Dreplsuffix\2Cdc\3Dcom,cn=mapping tree,cn=config
changetype: modify
replace: nsds5beginreplicarefresh
nsds5beginreplicarefresh: start
EOF

modifying entry "cn=20100_to_20103,cn=replica,cn=dc\3Dreplsuffix\2Cdc\3Dcom,cn=mapping tree,cn=config"

[root@rhel61-ds90-amita ~]# ldapsearch -x -p 20102 -h localhost -D "cn=Directory Manager" -w Secret123 -b "uid=amsharma112,ou=people,dc=replsuffix,dc=com" "(objectclass=*)" entryusn
# extended LDIF
#
# LDAPv3
# base <uid=amsharma112,ou=people,dc=replsuffix,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: entryusn 
#

# search result
search: 2
result: 10 Referral
matchedDN: dc=replsuffix,dc=com
ref: ldap://rhel61-ds90-amita.idm.lab.bos.redhat.com:20100
ref: ldap://rhel61-ds90-amita.idm.lab.bos.redhat.com:20106
ref: ldap://rhel61-ds90-amita.idm.lab.bos.redhat.com:20104

# numResponses: 1
[root@rhel61-ds90-amita ~]# ldapsearch -x -p 20102 -h localhost -D "cn=Directory Manager" -w Secret123 -b "uid=amsharma112,ou=people,dc=replsuffix,dc=com" "(objectclass=*)" entryusn
# extended LDIF
#
# LDAPv3
# base <uid=amsharma112,ou=people,dc=replsuffix,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: entryusn 
#

# amsharma112, People, replsuffix.com
dn: uid=amsharma112,ou=People,dc=replsuffix,dc=com
entryusn: 0

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1
[root@rhel61-ds90-amita ~]# 

For last case two cases of MMR, the results are not as expected, sending you the machine info in mail, Please guide.

Comment 16 Noriko Hosoi 2011-06-09 19:54:49 UTC
> For last case two cases of MMR, the results are not as expected, 
> sending you the machine info in mail, Please guide.

Could you put down the expected results?

On the last 2 tests, ldapsearch was issued against the port 20102.  The port belongs to the server M2: 
# egrep 20102 /etc/dirsrv/slapd-*/dse.ldif | egrep -i nsslapd-port
/etc/dirsrv/slapd-M2/dse.ldif:nsslapd-port: 20102

I grepped nsslapd-entryusn-import-initval in all the config files and found just M1 has it.  That being said, imported/initialized entries on M2 are supposed to have entryusn 0, aren't they?
# egrep nsslapd-entryusn-import-initval /etc/dirsrv/slapd-*/dse.ldif
/etc/dirsrv/slapd-M1/dse.ldif:nsslapd-entryusn-import-initval: next

Comment 17 Amita Sharma 2011-06-10 13:30:56 UTC
oh okie, then my expectation was wrong.
In all four cases in MMR when I am updating the nsslapd-entryusn-import-initval value (no,0,8589934592 and next) at one master M1.
It gives entryusn 0.

Hence marking the bug as verified.


Note You need to log in before you can comment on or make changes to this bug.