If you take a working replica and re-initialize it entryusn attributes are not properly created on the new entries being replicated in. In a directory with 194 entries only 24 of them had a entryusn value. the entryusn value was also wrapped and restarted from a value lower than what was previously available breaking the promise of getting higher values when entries are changed.
Input from Simo: <simo> yet it would be nice to get the entryusn attribute on all the records even if it is set to 0 for imported ones <simo> (my preference would be to have it progressing though) Currently, entryusn is reset to 0 if an entry already hasn't had the entryusn attribute in the import. 356 void 357 import_producer(void *param) 358 { ... 670 /* 671 * Check if entryusn plugin is enabled. 672 * If yes, add "entryusn: 0" to the entry 673 * if it does not have the attr type . 674 */ I wonder what entryusn values do the 24 entries have. 0? Or some other larger number? We can change the behaviour such that 1) it ignores the existing entryusn in the ldif to be imported, and 2) it assigns ascendant digits (as Simo suggested). One concern is Entry USN has a param nsslapd-entryusn-global, which cannot be supported if you import multiple LDIFs to multiple backends. Is it still worth assigning incremental numbers?
We use nsslapd-entryusn-global in IPA so if the incremental option is incompatible with it then it is not usable by an IPA Server.(In reply to comment #1) > Input from Simo: > <simo> yet it would be nice to get the entryusn attribute on all the records > even if it is set to 0 for imported ones > <simo> (my preference would be to have it progressing though) > > Currently, entryusn is reset to 0 if an entry already hasn't had the entryusn > attribute in the import. > 356 void > 357 import_producer(void *param) > 358 { > ... > 670 /* > 671 * Check if entryusn plugin is enabled. > 672 * If yes, add "entryusn: 0" to the entry > 673 * if it does not have the attr type . > 674 */ > > I wonder what entryusn values do the 24 entries have. 0? Or some other larger > number? It had incremental numers starting from 1 to 24, I suspect the additions where caused by the memberof plugin changing the entries to add the memberof value during import. > We can change the behaviour such that > 1) it ignores the existing entryusn in the ldif to be imported, and > 2) it assigns ascendant digits (as Simo suggested). > > One concern is Entry USN has a param nsslapd-entryusn-global, which cannot be > supported if you import multiple LDIFs to multiple backends. Is it still worth > assigning incremental numbers? We use nsslapd-entryusn-global in IPA so if the incremental option is incompatible with it then it is not usable by an IPA Server. If it is too difficult to add incremental numbers during the import it is better to set all entries to just 0. The main problem we face is that sssd uses entryusn in searches used for enumerations in order to get only items that changed since the last search was performed. So another way could be to set all entries to the previous highest usn (+1) on import. And then let the plugin increment from there on normal operations.
Is there a way to correct the entryUSNs after the import but before the replica gets online in IPA case? Since it is an IPA feature can there be a script/plugin that would run through the entryUSNs and reset them to the right incremental numbers. Can such script/plugin be run as a part of the IPA replica initialization and not as a part of the DS server import?
Well I guess that technically you could "touch" each entry in the tree after the replica has come in, this will cause all entries to get a new entryUSN. Sounds a bit ugly but potentially doable. Although this may cause more issues to clients than actually not doing it come to think, as you may end up with a highest USN higher than the one available before the reinitialization which will cause clients not to throw away the current counter they keep. This in turn may cause some unfortunate clients to not see some changes that happened on other masters while the replica was reinitializing, depending on the order in which entries are "touched". I guess we could try to touch them sorting by modifiedTimestamp, that should normally cause entries get entryusn roughly in the right order. Another possibility is that we get a way to set the highest USN. We could then save the highest one before reinitialization and set the new one to the old value before "touching" all entries. Then the "touch" will guarantee that all entries have USNs that are higher than waht was available earlier. This may cause clients to basically refresh their entire cache, but they wouldn't miss changes.
For now, there is no way for clients to "set" some specific entryusn value. For instnace, setting this entryusn value ends up with the "next" entryusn. ldapmodify -D 'cn=directory manager' -w pw << EOF dn: uid=BDipace9,ou=People,dc=example,dc=com changetype: modify replace: entryusn entryusn: 100000 EOF ldapsearch ... -D 'cn=directory manager' -w pw -b "ou=people,dc=example,dc=com" "(cn=*)" entryusn ... dn: uid=BDipace9,ou=People,dc=example,dc=com entryusn: 10 This proposal could be doable with a little change in the import code. (Of course, we have to introduce a new config option, though.) > So another way could be to set all entries to the previous highest usn (+1) on import. And then let the plugin increment from there on normal operations.
(In reply to comment #5) > For now, there is no way for clients to "set" some specific entryusn value. I was arguing for the ability to set the highest usn value as shown by the plugin in rootdse, not a random entryusn value. > This proposal could be doable with a little change in the import code. (Of > course, we have to introduce a new config option, though.) > > So another way could be to set all entries to the previous highest usn (+1) on > import. And then let the plugin increment from there on normal operations. This looks also as a reasonable compromise.
Created attachment 469481 [details] git patch file (master) Description: Introducing a config parameter nsslapd-entryusn-import-initval. If the value is digit, e.g., 0, 10, 100 ..., the imported entries will have the entryusn value. If the value is not digit, e.g., "next", the imported entries will have the last entryusn + 1 from the database existed before the import was exected or initialized by the master. In addition, this patch fixes a minor invalid access to a freed memory in cl5DeleteDBSync.
Steps to verify case 1. no nsslapd-entryusn-import-initval in cn=config. 1-1. server is down <server_instance_dir>/ldif2db -n <backend> -i /path/to/ldif 1-2. server is up <server_instance_dir>/ldif2db.pl -D 'cn=directory manager' -w <password> -n <backend> -i /path/to/ldif Result: all entry have "entryusn: 0": $ ldapsearch ... -b "dc=example,dc=com" "(objectclass=*)" entryusn dn: dc=example,dc=com entryusn: 0 dn: ou=People,dc=example,dc=com entryusn: 0 ... case 2. "nsslapd-entryusn-import-initval: 0" in cn=config. 2-1. server is down <server_instance_dir>/ldif2db -n <backend> -i /path/to/ldif 2-2. server is up <server_instance_dir>/ldif2db.pl -D 'cn=directory manager' -w <password> -n <backend> -i /path/to/ldif Result: all entry have "entryusn: 0": $ ldapsearch ... -b "dc=example,dc=com" "(objectclass=*)" entryusn dn: dc=example,dc=com entryusn: 0 dn: ou=People,dc=example,dc=com entryusn: 0 ... case 3. "nsslapd-entryusn-import-initval: 8589934592" in cn=config (> 32bit uint). 3-1. server is down <server_instance_dir>/ldif2db -n <backend> -i /path/to/ldif 3-2. server is up <server_instance_dir>/ldif2db.pl -D 'cn=directory manager' -w <password> -n <backend> -i /path/to/ldif Result: all entry have "entryusn: 8589934592": $ ldapsearch ... -b "dc=example,dc=com" "(objectclass=*)" entryusn dn: dc=example,dc=com entryusn: 8589934592 dn: ou=People,dc=example,dc=com entryusn: 8589934592 ... case 4. "nsslapd-entryusn-import-initval: next" in cn=config. 4-1. server is down <server_instance_dir>/ldif2db -n <backend> -i /path/to/ldif 4-2. server is up <server_instance_dir>/ldif2db.pl -D 'cn=directory manager' -w <password> -n <backend> -i /path/to/ldif Result: all entry have "entryusn: 8589934593" (last entryusn value + 1 in the previous db): $ ldapsearch ... -b "dc=example,dc=com" "(objectclass=*)" entryusn dn: dc=example,dc=com entryusn: 8589934593 dn: ou=People,dc=example,dc=com entryusn: 8589934593 ... case 5. Set up MMR. Instead of ldif2db[.pl], use replica initialization. Set nsslapd-entryusn-import-initval as in case 1 through 4 on the replica, execute replica initialization on the master. Check entryusn on the replica. Note: when "nsslapd-entryusn-import-initval: next" is set, the entryusn value could be larger than (previous entryusn + 1) due to the internal update operations executed by the replication plug-in.
Created attachment 469889 [details] git patch file (master) Description: 1) Introducing a config parameter nsslapd-entryusn-import-initval. If the value is digit, e.g., 0, 10, 100 ..., the imported entries will have the entryusn value. If the value is not digit, e.g., "next", the imported entries will have the last entryusn + 1 from the database existed before the import was exected or initialized by the master. 2) Internal OID LDBM_ENTRYUSN_OID has been redifined in back-ldbm.h to avoid a possible conflict. 3) If an attribute to modify is an operational attribute and that is the only one modify operation (not one of the series of modify operations), the server returns UNWILLING_TO_PERFORM. In addition, this patch fixes a minor invalid access to a freed memory in cl5DeleteDBSync.
Reviewed by Nathan (Thank you!!!) Pushed to master. $ git merge 663484 Updating 53c948c..196f1ef Fast-forward ldap/servers/plugins/replication/cl5_api.c | 9 ++- ldap/servers/slapd/back-ldbm/back-ldbm.h | 4 +- ldap/servers/slapd/back-ldbm/import-threads.c | 117 +++++++++++++++++++------ ldap/servers/slapd/back-ldbm/import.h | 1 + ldap/servers/slapd/back-ldbm/ldbm_usn.c | 3 +- ldap/servers/slapd/back-ldbm/ldif2ldbm.c | 22 +++++ ldap/servers/slapd/libglobs.c | 35 ++++++++ ldap/servers/slapd/modify.c | 7 +- ldap/servers/slapd/proto-slap.h | 2 + ldap/servers/slapd/slap.h | 4 +- 10 files changed, 168 insertions(+), 36 deletions(-) $ git push Counting objects: 35, done. Delta compression using up to 4 threads. Compressing objects: 100% (18/18), done. Writing objects: 100% (18/18), 3.92 KiB, done. Total 18 (delta 16), reused 0 (delta 0) To ssh://git.fedorahosted.org/git/389/ds.git 53c948c..196f1ef master -> master
Usage is found here. http://directory.fedoraproject.org/wiki/Entry_USN#Import_and_Replica_Initialization
dn: cn=USN,cn=plugins,cn=config objectClass: top objectClass: nsSlapdPlugin objectClass: extensibleObject cn: USN nsslapd-pluginPath: libusn-plugin nsslapd-pluginInitfunc: usn_init nsslapd-pluginType: object nsslapd-pluginEnabled: on nsslapd-plugin-depends-on-type: database and nsslapd-entryusn-global: on Now Case 1 After Import, ldapsearch returns ================================ # LTrudell7207, Accounting, example.com dn: uid=LTrudell7207,ou=Accounting,dc=example,dc=com entryusn: 7220 # GCogdell7208, Accounting, example.com dn: uid=GCogdell7208,ou=Accounting,dc=example,dc=com entryusn: 7221 # BReaves7209, Human Resources, example.com dn: uid=BReaves7209,ou=Human Resources,dc=example,dc=com entryusn: 7222 # KTamarell7210, Accounting, example.com dn: uid=KTamarell7210,ou=Accounting,dc=example,dc=com entryusn: 7223 # RTanchak7211, Payroll, example.com dn: uid=RTanchak7211,ou=Payroll,dc=example,dc=com entryusn: 7224 # JSchoch7212, Product Testing, example.com dn: uid=JSchoch7212,ou=Product Testing,dc=example,dc=com entryusn: 7225 case 2 nsslapd-entryusn-global: on nsslapd-entryusn-import-initval: 0 After Import, ldapsearch returns ================================ [root@rhel61 slapd-rhel61]# ldapsearch -x -h localhost -p 389 -D "cn=directory manager" -w Secret123 -b "ou=people,dc=example,dc=com" "(objectclass=*)" entryusn # extended LDIF # # LDAPv3 # base <ou=people,dc=example,dc=com> with scope subtree # filter: (objectclass=*) # requesting: entryusn # # people, example.com dn: ou=people,dc=example,dc=com entryusn: 7 # TVradmin0, people, example.com dn: uid=TVradmin0,ou=people,dc=example,dc=com entryusn: 7346 # VLeBaron1, people, example.com dn: uid=VLeBaron1,ou=people,dc=example,dc=com entryusn: 7347 # JOshinski2, people, example.com dn: uid=JOshinski2,ou=people,dc=example,dc=com entryusn: 7348 # search result search: 2 result: 0 Success # numResponses: 5 # numEntries: 4 case 3 : nsslapd-entryusn-import-initval: 8589934592 After Import, ldapsearch returns ================================ I removed these entries thn again imported but still same result. [root@rhel61 slapd-rhel61]# ldapsearch -x -h localhost -p 389 -D "cn=directory manager" -w Secret123 -b "ou=people,dc=example,dc=com" "(objectclass=*)" entryusn # extended LDIF # # LDAPv3 # base <ou=people,dc=example,dc=com> with scope subtree # filter: (objectclass=*) # requesting: entryusn # # people, example.com dn: ou=people,dc=example,dc=com entryusn: 7 # TVradmin0, people, example.com dn: uid=TVradmin0,ou=people,dc=example,dc=com entryusn: 7353 # VLeBaron1, people, example.com dn: uid=VLeBaron1,ou=people,dc=example,dc=com entryusn: 7354 # JOshinski2, people, example.com dn: uid=JOshinski2,ou=people,dc=example,dc=com entryusn: 7355 # search result search: 2 result: 0 Success NOTE: I am running the import from the ds-console.. not sure wht is the issue with ldif2db. the same ldif I am able to prcess from ds-console. I hope that should not make any difference. [root@rhel61 slapd-rhel61]# ./ldif2db.pl -D 'cn=directory manager' -w Secret123 -n userRoot -i /home/example10.ldif adding new entry "cn=import_2011_6_7_17_48_49, cn=import, cn=tasks, cn=config" [root@rhel61 slapd-rhel61]# service dirsrv stop Shutting down dirsrv: rhel61... [ OK ] [root@rhel61 slapd-rhel61]# ./ldif2db -n userRoot -i /home/example10 example100k.ldif example10.ldif [root@rhel61 slapd-rhel61]# ./ldif2db -n userRoot -i /home/example10.ldif importing data ... [07/Jun/2011:17:49:39 +051800] - I'm resizing my cache now...cache was 20000000 and is now 8000000 [07/Jun/2011:17:49:40 +051800] - All database threads now stopped [07/Jun/2011:17:49:40 +051800] - I'm resizing my cache now...cache was 20000000 and is now 8000000 [07/Jun/2011:17:49:40 +051800] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [07/Jun/2011:17:49:40 +051800] - check_and_set_import_cache: pagesize: 4096, pages: 125898, procpages: 48953 [07/Jun/2011:17:49:40 +051800] - WARNING: After allocating import cache 201436KB, the available memory is 302156KB, which is less than the soft limit 1048576KB. You may want to decrease the import cache size and rerun import. [07/Jun/2011:17:49:40 +051800] - Import allocates 201436KB import cache. [07/Jun/2011:17:49:40 +051800] - import userRoot: Beginning import job... [07/Jun/2011:17:49:40 +051800] - import userRoot: Index buffering enabled with bucket size 100 [07/Jun/2011:17:49:40 +051800] - import userRoot: Processing file "/home/example10.ldif" [07/Jun/2011:17:49:40 +051800] - import userRoot: Finished scanning file "/home/example10.ldif" (0 entries) [07/Jun/2011:17:49:40 +051800] - import userRoot: Workers finished; cleaning up... [07/Jun/2011:17:49:41 +051800] - import userRoot: Workers cleaned up. [07/Jun/2011:17:49:41 +051800] - import userRoot: Cleaning up producer thread... [07/Jun/2011:17:49:41 +051800] - import userRoot: Indexing complete. Post-processing... [07/Jun/2011:17:49:41 +051800] - Nothing to do to build ancestorid index [07/Jun/2011:17:49:41 +051800] - import userRoot: Flushing caches... [07/Jun/2011:17:49:41 +051800] - import userRoot: Closing files... [07/Jun/2011:17:49:41 +051800] - All database threads now stopped [07/Jun/2011:17:49:41 +051800] - import userRoot: Import complete. Processed 0 entries in 1 seconds. (0.00 entries/sec) Sending you the machine info in mail.
Let's make things straight one by one... 1. Your server slapd-rhel61 has 2 backends (except o=netscaperoot): Suffix: dc=pnq,dc=redhat,dc=com; backend: userRoot Suffix: dc=example,dc=com; backend: exampledb When you import, the entries in the ldif file are successfully imported if the given backend name matches. That's said, if you import entries under "dc=example,dc=com", you have to specify "-n exampledb" in the ldif2db command line (instead of "userRoot"). 2. I'm afraid you cannot use "Import databases" task (on DS Console | Tasks) for this case. If you take a look at the access log, you could see this "import" uses ordinary LDAP ADD operation, not the real import. [..] conn=1 op=263 ADD dn="uid=YLucas7,ou=People,dc=example,dc=com" [..] conn=1 op=263 RESULT err=0 tag=105 nentries=0 etime=0 [..] This bug needs to be verified using import (ldif2db*). To do so, you could use ldif2db or ldif2db.pl command line tool, or DS Console | Configuration | Data, then expand the suffix and choose its backend (e.g., exampledb). Right click there brings up a menu. Choose initialize database. Type full path to your ldif file and click OK. It calls real import "ldif2db" in the server. You could verify it with the logs in the errors log: [..] - Bringing userRoot offline... [..] - userRoot: Cleaning up entry cache [..] - userRoot: Cleaning up dn cache [..] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [..] - import userRoot: Beginning import job... [..]
case 1. =============== no nsslapd-entryusn-import-initval in cn=config. 1-1. server is down <server_instance_dir>/ldif2db -n <backend> -i /path/to/ldif 1-2. server is up <server_instance_dir>/ldif2db.pl -D 'cn=directory manager' -w <password> -n <backend> -i /path/to/ldif Result: all entry have "entryusn: 0": $ ldapsearch ... -b "dc=example,dc=com" "(objectclass=*)" entryusn dn: dc=example,dc=com entryusn: 0 dn: ou=People,dc=example,dc=com entryusn: 0 ... [root@rhel61 slapd-rhel61]# ./ldif2db -n exampledb -i /home/example10.ldif importing data ... [09/Jun/2011:12:51:27 +051800] - All database threads now stopped [09/Jun/2011:12:51:27 +051800] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [09/Jun/2011:12:51:27 +051800] - check_and_set_import_cache: pagesize: 4096, pages: 125898, procpages: 48954 [09/Jun/2011:12:51:27 +051800] - WARNING: After allocating import cache 201436KB, the available memory is 302156KB, which is less than the soft limit 1048576KB. You may want to decrease the import cache size and rerun import. [09/Jun/2011:12:51:27 +051800] - Import allocates 201436KB import cache. [09/Jun/2011:12:51:28 +051800] - import exampledb: Beginning import job... [09/Jun/2011:12:51:28 +051800] - import exampledb: Index buffering enabled with bucket size 100 [09/Jun/2011:12:51:28 +051800] - import exampledb: Processing file "/home/example10.ldif" [09/Jun/2011:12:51:28 +051800] - import exampledb: Finished scanning file "/home/example10.ldif" (15 entries) [09/Jun/2011:12:51:28 +051800] - import exampledb: Workers finished; cleaning up... [09/Jun/2011:12:51:29 +051800] - import exampledb: Workers cleaned up. [09/Jun/2011:12:51:29 +051800] - import exampledb: Cleaning up producer thread... [09/Jun/2011:12:51:29 +051800] - import exampledb: Indexing complete. Post-processing... [09/Jun/2011:12:51:29 +051800] - import exampledb: Flushing caches... [09/Jun/2011:12:51:29 +051800] - import exampledb: Closing files... [09/Jun/2011:12:51:29 +051800] - All database threads now stopped [09/Jun/2011:12:51:29 +051800] - import exampledb: Import complete. Processed 15 entries in 1 seconds. (15.00 entries/sec) [root@rhel61 slapd-rhel61]# service dirsrv start Starting dirsrv: rhel61... [ OK ] [root@rhel61 slapd-rhel61]# ldapsearch -x -h localhost -p 389 -D "cn=directory manager" -w Secret123 -b "dc=example,dc=com" "(objectclass=*)" entryusn # extended LDIF # # LDAPv3 # base <dc=example,dc=com> with scope subtree # filter: (objectclass=*) # requesting: entryusn # # example.com dn: dc=example,dc=com entryusn: 0 # Accounting, example.com dn: ou=Accounting,dc=example,dc=com entryusn: 0 # Product Development, example.com dn: ou=Product Development,dc=example,dc=com entryusn: 0 # Product Testing, example.com dn: ou=Product Testing,dc=example,dc=com entryusn: 0 # Human Resources, example.com dn: ou=Human Resources,dc=example,dc=com entryusn: 0 # Payroll, example.com dn: ou=Payroll,dc=example,dc=com entryusn: 0 ================================================================================ case 2. ======= "nsslapd-entryusn-import-initval: 0" in cn=config. 2-1. server is down <server_instance_dir>/ldif2db -n <backend> -i /path/to/ldif 2-2. server is up <server_instance_dir>/ldif2db.pl -D 'cn=directory manager' -w <password> -n <backend> -i /path/to/ldif Result: all entry have "entryusn: 0": $ ldapsearch ... -b "dc=example,dc=com" "(objectclass=*)" entryusn dn: dc=example,dc=com entryusn: 0 dn: ou=People,dc=example,dc=com entryusn: 0 ... nsslapd-entryusn-global: on nsslapd-accesslog-maxlogsize: 100 nsslapd-entryusn-import-initval: 0 nsslapd-accesslog-logrotationtime: 1 [root@rhel61 slapd-rhel61]# service dirsrv stop Shutting down dirsrv: rhel61... [ OK ] [root@rhel61 slapd-rhel61]# ./ldif2db -n exampledb -i /home/example10.ldif importing data ... [09/Jun/2011:12:54:33 +051800] - All database threads now stopped [09/Jun/2011:12:54:33 +051800] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [09/Jun/2011:12:54:33 +051800] - check_and_set_import_cache: pagesize: 4096, pages: 125898, procpages: 48954 [09/Jun/2011:12:54:33 +051800] - WARNING: After allocating import cache 201436KB, the available memory is 302156KB, which is less than the soft limit 1048576KB. You may want to decrease the import cache size and rerun import. [09/Jun/2011:12:54:33 +051800] - Import allocates 201436KB import cache. [09/Jun/2011:12:54:33 +051800] - import exampledb: Beginning import job... [09/Jun/2011:12:54:33 +051800] - import exampledb: Index buffering enabled with bucket size 100 [09/Jun/2011:12:54:34 +051800] - import exampledb: Processing file "/home/example10.ldif" [09/Jun/2011:12:54:34 +051800] - import exampledb: Finished scanning file "/home/example10.ldif" (15 entries) [09/Jun/2011:12:54:34 +051800] - import exampledb: Workers finished; cleaning up... [09/Jun/2011:12:54:35 +051800] - import exampledb: Workers cleaned up. [09/Jun/2011:12:54:35 +051800] - import exampledb: Cleaning up producer thread... [09/Jun/2011:12:54:35 +051800] - import exampledb: Indexing complete. Post-processing... [09/Jun/2011:12:54:35 +051800] - import exampledb: Flushing caches... [09/Jun/2011:12:54:35 +051800] - import exampledb: Closing files... [09/Jun/2011:12:54:35 +051800] - All database threads now stopped [09/Jun/2011:12:54:35 +051800] - import exampledb: Import complete. Processed 15 entries in 2 seconds. (7.50 entries/sec) [root@rhel61 slapd-rhel61]# ldapsearch -x -h localhost -p 389 -D "cn=directory manager" -w Secret123 -b "dc=example,dc=com" "(objectclass=*)" entryusn ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1) [root@rhel61 slapd-rhel61]# service dirsrv start Starting dirsrv: rhel61... [ OK ] [root@rhel61 slapd-rhel61]# ldapsearch -x -h localhost -p 389 -D "cn=directory manager" -w Secret123 -b "dc=example,dc=com" "(objectclass=*)" entryusn # extended LDIF # # LDAPv3 # base <dc=example,dc=com> with scope subtree # filter: (objectclass=*) # requesting: entryusn # # example.com dn: dc=example,dc=com entryusn: 0 # Accounting, example.com dn: ou=Accounting,dc=example,dc=com entryusn: 0 # Product Development, example.com dn: ou=Product Development,dc=example,dc=com entryusn: 0 # Product Testing, example.com dn: ou=Product Testing,dc=example,dc=com entryusn: 0 # Human Resources, example.com dn: ou=Human Resources,dc=example,dc=com entryusn: 0 # Payroll, example.com dn: ou=Payroll,dc=example,dc=com entryusn: 0 ================================================================================ case 3. ======= "nsslapd-entryusn-import-initval: 8589934592" in cn=config (> 32bit uint). 3-1. server is down <server_instance_dir>/ldif2db -n <backend> -i /path/to/ldif 3-2. server is up <server_instance_dir>/ldif2db.pl -D 'cn=directory manager' -w <password> -n <backend> -i /path/to/ldif Result: all entry have "entryusn: 8589934592": $ ldapsearch ... -b "dc=example,dc=com" "(objectclass=*)" entryusn dn: dc=example,dc=com entryusn: 8589934592 dn: ou=People,dc=example,dc=com entryusn: 8589934592 ... [root@rhel61 slapd-rhel61]# ./ldif2db -n exampledb -i /home/example10.ldif importing data ... [09/Jun/2011:12:43:29 +051800] - I'm resizing my cache now...cache was 206270464 and is now 8000000 [09/Jun/2011:12:43:29 +051800] - All database threads now stopped [09/Jun/2011:12:43:29 +051800] - I'm resizing my cache now...cache was 206270464 and is now 8000000 [09/Jun/2011:12:43:29 +051800] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [09/Jun/2011:12:43:29 +051800] - check_and_set_import_cache: pagesize: 4096, pages: 125898, procpages: 48954 [09/Jun/2011:12:43:29 +051800] - WARNING: After allocating import cache 201436KB, the available memory is 302156KB, which is less than the soft limit 1048576KB. You may want to decrease the import cache size and rerun import. [09/Jun/2011:12:43:29 +051800] - Import allocates 201436KB import cache. [09/Jun/2011:12:43:29 +051800] - import exampledb: Beginning import job... [09/Jun/2011:12:43:29 +051800] - import exampledb: Index buffering enabled with bucket size 100 [09/Jun/2011:12:43:30 +051800] - import exampledb: Processing file "/home/example10.ldif" [09/Jun/2011:12:43:30 +051800] - import exampledb: Finished scanning file "/home/example10.ldif" (15 entries) [09/Jun/2011:12:43:31 +051800] - import exampledb: Workers finished; cleaning up... [09/Jun/2011:12:43:31 +051800] - import exampledb: Workers cleaned up. [09/Jun/2011:12:43:31 +051800] - import exampledb: Cleaning up producer thread... [09/Jun/2011:12:43:31 +051800] - import exampledb: Indexing complete. Post-processing... [09/Jun/2011:12:43:31 +051800] - import exampledb: Flushing caches... [09/Jun/2011:12:43:31 +051800] - import exampledb: Closing files... [09/Jun/2011:12:43:31 +051800] - All database threads now stopped [09/Jun/2011:12:43:31 +051800] - import exampledb: Import complete. Processed 15 entries in 2 seconds. (7.50 entries/sec) [root@rhel61 slapd-rhel61]# service dirsrv start Starting dirsrv: rhel61... [ OK ] [root@rhel61 slapd-rhel61]# ldapsearch -x -h localhost -p 389 -D "cn=directory manager" -w Secret123 -b "dc=example,dc=com" "(objectclass=*)" entryusn # extended LDIF # # LDAPv3 # base <dc=example,dc=com> with scope subtree # filter: (objectclass=*) # requesting: entryusn # # example.com dn: dc=example,dc=com entryusn: 8589934592 # Accounting, example.com dn: ou=Accounting,dc=example,dc=com entryusn: 8589934592 # Product Development, example.com dn: ou=Product Development,dc=example,dc=com entryusn: 8589934592 # Product Testing, example.com dn: ou=Product Testing,dc=example,dc=com entryusn: 8589934592 # Human Resources, example.com dn: ou=Human Resources,dc=example,dc=com entryusn: 8589934592 # Payroll, example.com dn: ou=Payroll,dc=example,dc=com entryusn: 8589934592 # TVradmin0, Accounting, example.com dn: uid=TVradmin0,ou=Accounting,dc=example,dc=com entryusn: 8589934592 ================================================================================ case 4. ======== "nsslapd-entryusn-import-initval: next" in cn=config. 4-1. server is down <server_instance_dir>/ldif2db -n <backend> -i /path/to/ldif 4-2. server is up <server_instance_dir>/ldif2db.pl -D 'cn=directory manager' -w <password> -n <backend> -i /path/to/ldif Result: all entry have "entryusn: 8589934593" (last entryusn value + 1 in the previous db): $ ldapsearch ... -b "dc=example,dc=com" "(objectclass=*)" entryusn dn: dc=example,dc=com entryusn: 8589934593 dn: ou=People,dc=example,dc=com entryusn: 8589934593 ... Note: This case I have executed after case number 2 where the entryUSN was "ZERO" [root@rhel61 slapd-rhel61]# ldapsearch -x -h localhost -p 389 -D "cn=directory manager" -w Secret123 -b "dc=example,dc=com" "(objectclass=*)" entryusn # extended LDIF # # LDAPv3 # base <dc=example,dc=com> with scope subtree # filter: (objectclass=*) # requesting: entryusn # # example.com dn: dc=example,dc=com entryusn: 7391 # Accounting, example.com dn: ou=Accounting,dc=example,dc=com entryusn: 7391 # Product Development, example.com dn: ou=Product Development,dc=example,dc=com entryusn: 7391 # Product Testing, example.com dn: ou=Product Testing,dc=example,dc=com entryusn: 7391 # Human Resources, example.com dn: ou=Human Resources,dc=example,dc=com entryusn: 7391 ================================================================================ case 5. ======== Set up MMR. Instead of ldif2db[.pl], use replica initialization. Set nsslapd-entryusn-import-initval as in case 1 through 4 on the replica, execute replica initialization on the master. Check entryusn on the replica. Note: when "nsslapd-entryusn-import-initval: next" is set, the entryusn value could be larger than (previous entryusn + 1) due to the internal update operations executed by the replication plug-in. ========== MMR case 1 ========== ldapmodify -D "cn=directory manager" -w Secret123 -p 20100 -x -h localhost << EOF > dn: cn=20100_to_20103,cn=replica,cn=dc\3Dreplsuffix\2Cdc\3Dcom,cn=mapping tree,cn=config > changetype: modify > replace: nsds5beginreplicarefresh > nsds5beginreplicarefresh: start > EOF modifying entry "cn=20100_to_20103,cn=replica,cn=dc\3Dreplsuffix\2Cdc\3Dcom,cn=mapping tree,cn=config" [root@rhel61-ds90-amita ~]# ldapsearch -x -p 20102 -h localhost -D "cn=Directory Manager" -w Secret123 -b "uid=amsharma111,ou=people,dc=replsuffix,dc=com" "(objectclass=*)" entryusn # extended LDIF # # LDAPv3 # base <uid=amsharma111,ou=people,dc=replsuffix,dc=com> with scope subtree # filter: (objectclass=*) # requesting: entryusn # # amsharma111, People, replsuffix.com dn: uid=amsharma111,ou=People,dc=replsuffix,dc=com entryusn: 0 # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 ============ MMR case 2 ========== "nsslapd-entryusn-import-initval: 0" in cn=config. nsslapd-entryusn-global: on nsslapd-entryusn-import-initval: 0 [root@rhel61-ds90-amita ~]# /usr/lib64/dirsrv/slapd-M1/stop-slapd [root@rhel61-ds90-amita ~]# vim /etc/dirsrv/slapd-M1/dse.ldif [root@rhel61-ds90-amita ~]# /usr/lib64/dirsrv/slapd-M1/start-slapd [root@rhel61-ds90-amita ~]# ldapmodify -D "cn=directory manager" -w Secret123 -p 20100 -x -h localhost << EOF dn: cn=20100_to_20103,cn=replica,cn=dc\3Dreplsuffix\2Cdc\3Dcom,cn=mapping tree,cn=config changetype: modify replace: nsds5beginreplicarefresh nsds5beginreplicarefresh: start EOF modifying entry "cn=20100_to_20103,cn=replica,cn=dc\3Dreplsuffix\2Cdc\3Dcom,cn=mapping tree,cn=config" [root@rhel61-ds90-amita ~]# ldapsearch -x -p 20102 -h localhost -D "cn=Directory Manager" -w Secret123 -b "uid=amsharma111,ou=people,dc=replsuffix,dc=com" "(objectclass=*)" entryusn # extended LDIF # # LDAPv3 # base <uid=amsharma111,ou=people,dc=replsuffix,dc=com> with scope subtree # filter: (objectclass=*) # requesting: entryusn # # amsharma111, People, replsuffix.com dn: uid=amsharma111,ou=People,dc=replsuffix,dc=com entryusn: 0 # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 ========= MMR case 3 ========== "nsslapd-entryusn-import-initval: 8589934592" in cn=config (> 32bit uint). [root@rhel61-ds90-amita ~]# /usr/lib64/dirsrv/slapd-M1/stop-slapd [root@rhel61-ds90-amita ~]# vim /etc/dirsrv/slapd-M1/dse.ldif [root@rhel61-ds90-amita ~]# /usr/lib64/dirsrv/slapd-M1/start-slapd [root@rhel61-ds90-amita ~]# ldapmodify -D "cn=directory manager" -w Secret123 -p 20100 -x -h localhost << EOFdn: cn=20100_to_20103,cn=replica,cn=dc\3Dreplsuffix\2Cdc\3Dcom,cn=mapping tree,cn=config changetype: modify replace: nsds5beginreplicarefresh nsds5beginreplicarefresh: start EOF modifying entry "cn=20100_to_20103,cn=replica,cn=dc\3Dreplsuffix\2Cdc\3Dcom,cn=mapping tree,cn=config" [root@rhel61-ds90-amita ~]# ldapsearch -x -p 20102 -h localhost -D "cn=Directory Manager" -w Secret123 -b "uid=amsharma111,ou=people,dc=replsuffix,dc=com" "(objectclass=*)" entryusn # extended LDIF # # LDAPv3 # base <uid=amsharma111,ou=people,dc=replsuffix,dc=com> with scope subtree # filter: (objectclass=*) # requesting: entryusn # # search result search: 2 result: 10 Referral matchedDN: dc=replsuffix,dc=com ref: ldap://rhel61-ds90-amita.idm.lab.bos.redhat.com:20100 ref: ldap://rhel61-ds90-amita.idm.lab.bos.redhat.com:20106 ref: ldap://rhel61-ds90-amita.idm.lab.bos.redhat.com:20104 # numResponses: 1 [root@rhel61-ds90-amita ~]# ldapsearch -x -p 20102 -h localhost -D "cn=Directory Manager" -w Secret123 -b "uid=amsharma111,ou=people,dc=replsuffix,dc=com" "(objectclass=*)" entryusn # extended LDIF # # LDAPv3 # base <uid=amsharma111,ou=people,dc=replsuffix,dc=com> with scope subtree # filter: (objectclass=*) # requesting: entryusn # # amsharma111, People, replsuffix.com dn: uid=amsharma111,ou=People,dc=replsuffix,dc=com entryusn: 0 # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 [root@rhel61-ds90-amita ~]# ldapadd -x -h localhost -p 20100 -D "cn=Directory Manager" -w Secret123 << EOFdn: uid=amsharma112,ou=people,dc=replsuffix,dc=com cn: ams sn: ams givenname: ams objectclass: top objectclass: person objectclass: organizationalPerson objectclass: inetOrgPerson uid: ams mail: ams userpassword: amsamsams EOF adding new entry "uid=amsharma112,ou=people,dc=replsuffix,dc=com" [root@rhel61-ds90-amita ~]# ldapsearch -x -p 20102 -h localhost -D "cn=Directory Manager" -w Secret123 -b "uid=amsharma112,ou=people,dc=replsuffix,dc=com" "(objectclass=*)" entryusn # extended LDIF # # LDAPv3 # base <uid=amsharma112,ou=people,dc=replsuffix,dc=com> with scope subtree # filter: (objectclass=*) # requesting: entryusn # # amsharma112, People, replsuffix.com dn: uid=amsharma112,ou=People,dc=replsuffix,dc=com entryusn: 3 # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 [root@rhel61-ds90-amita ~]# ldapmodify -D "cn=directory manager" -w Secret123 -p 20100 -x -h localhost << EOFdn: cn=20100_to_20103,cn=replica,cn=dc\3Dreplsuffix\2Cdc\3Dcom,cn=mapping tree,cn=config changetype: modify replace: nsds5beginreplicarefresh nsds5beginreplicarefresh: start EOF modifying entry "cn=20100_to_20103,cn=replica,cn=dc\3Dreplsuffix\2Cdc\3Dcom,cn=mapping tree,cn=config" [root@rhel61-ds90-amita ~]# ldapsearch -x -p 20102 -h localhost -D "cn=Directory Manager" -w Secret123 -b "uid=amsharma112,ou=people,dc=replsuffix,dc=com" "(objectclass=*)" entryusn # extended LDIF # # LDAPv3 # base <uid=amsharma112,ou=people,dc=replsuffix,dc=com> with scope subtree # filter: (objectclass=*) # requesting: entryusn # # search result search: 2 result: 10 Referral matchedDN: dc=replsuffix,dc=com ref: ldap://rhel61-ds90-amita.idm.lab.bos.redhat.com:20100 ref: ldap://rhel61-ds90-amita.idm.lab.bos.redhat.com:20106 ref: ldap://rhel61-ds90-amita.idm.lab.bos.redhat.com:20104 # numResponses: 1 [root@rhel61-ds90-amita ~]# ldapsearch -x -p 20102 -h localhost -D "cn=Directory Manager" -w Secret123 -b "uid=amsharma112,ou=people,dc=replsuffix,dc=com" "(objectclass=*)" entryusn # extended LDIF # # LDAPv3 # base <uid=amsharma112,ou=people,dc=replsuffix,dc=com> with scope subtree # filter: (objectclass=*) # requesting: entryusn # # amsharma112, People, replsuffix.com dn: uid=amsharma112,ou=People,dc=replsuffix,dc=com entryusn: 0 # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 [root@rhel61-ds90-amita ~]# ldapsearch -x -p 20102 -h localhost -D "cn=Directory Manager" -w Secret123 -b "uid=amsharma112,ou=people,dc=replsuffix,dc=com" "(objectclass=*)" entryusn # extended LDIF # # LDAPv3 # base <uid=amsharma112,ou=people,dc=replsuffix,dc=com> with scope subtree # filter: (objectclass=*) # requesting: entryusn # # amsharma112, People, replsuffix.com dn: uid=amsharma112,ou=People,dc=replsuffix,dc=com entryusn: 0 # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 =========== MMR Case 4 ========= "nsslapd-entryusn-import-initval: next" in cn=config. [root@rhel61-ds90-amita ~]# vim /etc/dirsrv/slapd-M1/dse.ldif [root@rhel61-ds90-amita ~]# /usr/lib64/dirsrv/slapd-M1/stop-slapd [root@rhel61-ds90-amita ~]# vim /etc/dirsrv/slapd-M1/dse.ldif [root@rhel61-ds90-amita ~]# /usr/lib64/dirsrv/slapd-M1/start-slapd [root@rhel61-ds90-amita ~]# ldapmodify -D "cn=directory manager" -w Secret123 -p 20100 -x -h localhost << EOFdn: cn=20100_to_20103,cn=replica,cn=dc\3Dreplsuffix\2Cdc\3Dcom,cn=mapping tree,cn=config changetype: modify replace: nsds5beginreplicarefresh nsds5beginreplicarefresh: start EOF modifying entry "cn=20100_to_20103,cn=replica,cn=dc\3Dreplsuffix\2Cdc\3Dcom,cn=mapping tree,cn=config" [root@rhel61-ds90-amita ~]# ldapsearch -x -p 20102 -h localhost -D "cn=Directory Manager" -w Secret123 -b "uid=amsharma112,ou=people,dc=replsuffix,dc=com" "(objectclass=*)" entryusn # extended LDIF # # LDAPv3 # base <uid=amsharma112,ou=people,dc=replsuffix,dc=com> with scope subtree # filter: (objectclass=*) # requesting: entryusn # # search result search: 2 result: 10 Referral matchedDN: dc=replsuffix,dc=com ref: ldap://rhel61-ds90-amita.idm.lab.bos.redhat.com:20100 ref: ldap://rhel61-ds90-amita.idm.lab.bos.redhat.com:20106 ref: ldap://rhel61-ds90-amita.idm.lab.bos.redhat.com:20104 # numResponses: 1 [root@rhel61-ds90-amita ~]# ldapsearch -x -p 20102 -h localhost -D "cn=Directory Manager" -w Secret123 -b "uid=amsharma112,ou=people,dc=replsuffix,dc=com" "(objectclass=*)" entryusn # extended LDIF # # LDAPv3 # base <uid=amsharma112,ou=people,dc=replsuffix,dc=com> with scope subtree # filter: (objectclass=*) # requesting: entryusn # # amsharma112, People, replsuffix.com dn: uid=amsharma112,ou=People,dc=replsuffix,dc=com entryusn: 0 # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 [root@rhel61-ds90-amita ~]# For last case two cases of MMR, the results are not as expected, sending you the machine info in mail, Please guide.
> For last case two cases of MMR, the results are not as expected, > sending you the machine info in mail, Please guide. Could you put down the expected results? On the last 2 tests, ldapsearch was issued against the port 20102. The port belongs to the server M2: # egrep 20102 /etc/dirsrv/slapd-*/dse.ldif | egrep -i nsslapd-port /etc/dirsrv/slapd-M2/dse.ldif:nsslapd-port: 20102 I grepped nsslapd-entryusn-import-initval in all the config files and found just M1 has it. That being said, imported/initialized entries on M2 are supposed to have entryusn 0, aren't they? # egrep nsslapd-entryusn-import-initval /etc/dirsrv/slapd-*/dse.ldif /etc/dirsrv/slapd-M1/dse.ldif:nsslapd-entryusn-import-initval: next
oh okie, then my expectation was wrong. In all four cases in MMR when I am updating the nsslapd-entryusn-import-initval value (no,0,8589934592 and next) at one master M1. It gives entryusn 0. Hence marking the bug as verified.