Description of problem: This is a sample tombstone entry in id2entry.db4 id 11 rdn: nsuniqueid=042d8081-4e6111e0-ac8585ed-ca8fe9f7,uid=tuser0 uid;vucsn-4d7e5156000000010000;mdcsn-4d7e5156000000010000: tuser0 givenName;vucsn-4d7e5156000000010000: test objectClass;vucsn-4d7e5156000000010000: top objectClass;vucsn-4d7e5156000000010000: person objectClass;vucsn-4d7e5156000000010000: organizationalPerson objectClass;vucsn-4d7e5156000000010000: inetorgperson objectClass;vucsn-4d7e5163000000010000: nsTombstone sn;vucsn-4d7e5156000000010000: user0 cn;vucsn-4d7e5156000000010000: test user0 userPassword;vucsn-4d7e5156000000010000: {SSHA}QscFOIbdsvg6+qdW8Dy+MKwuvPdlIEu w0tChBg== creatorsName;vucsn-4d7e5156000000010000: uid=admin,ou=administrators,ou=topolo gymanagement,o=netscaperoot modifiersName;vucsn-4d7e5156000000010000: uid=admin,ou=administrators,ou=topol ogymanagement,o=netscaperoot createTimestamp;vucsn-4d7e5156000000010000: 20110314173310Z modifyTimestamp;vucsn-4d7e5156000000010000: 20110314173310Z nsUniqueId: 042d8081-4e6111e0-ac8585ed-ca8fe9f7 parentid: 4 entryid: 11 nsParentUniqueId: 89ad3283-4e5911e0-98dfa788-5bfd3127 nscpEntryDN: uid=tuser0,ou=people,dc=example,dc=com This is the same entry in the exported ldif file with db2ldif -r: # entry-id: 11 dn: nsuniqueid=042d8081-4e6111e0-ac8585ed-ca8fe9f7,uid=tuser0,ou=People,dc=exa mple,dc=com nsUniqueId: 042d8081-4e6111e0-ac8585ed-ca8fe9f7 uid;vucsn-4d7e5156000000010000;mdcsn-4d7e5156000000010000: tuser0 givenName;vucsn-4d7e5156000000010000: test objectClass;vucsn-4d7e5156000000010000: top objectClass;vucsn-4d7e5156000000010000: person objectClass;vucsn-4d7e5156000000010000: organizationalPerson objectClass;vucsn-4d7e5156000000010000: inetorgperson objectClass;vucsn-4d7e5163000000010000: nsTombstone sn;vucsn-4d7e5156000000010000: user0 cn;vucsn-4d7e5156000000010000: test user0 userPassword;vucsn-4d7e5156000000010000: {SSHA}QscFOIbdsvg6+qdW8Dy+MKwuvPdlIEu w0tChBg== creatorsName;vucsn-4d7e5156000000010000: uid=admin,ou=administrators,ou=topolo gymanagement,o=netscaperoot modifiersName;vucsn-4d7e5156000000010000: uid=admin,ou=administrators,ou=topol ogymanagement,o=netscaperoot createTimestamp;vucsn-4d7e5156000000010000: 20110314173310Z modifyTimestamp;vucsn-4d7e5156000000010000: 20110314173310Z nsParentUniqueId: 89ad3283-4e5911e0-98dfa788-5bfd3127 nscpEntryDN: uid=tuser0,ou=people,dc=example,dc=com Once this tombstone entry is imported, it loses the original rdn info (uid=tuser0) as well as the attribute parentid: id 11 rdn: nsuniqueid=042d8081-4e6111e0-ac8585ed-ca8fe9f7 nsUniqueId: 042d8081-4e6111e0-ac8585ed-ca8fe9f7 uid;vucsn-4d7e5156000000010000;mdcsn-4d7e5156000000010000: tuser0 givenName;vucsn-4d7e5156000000010000: test objectClass;vucsn-4d7e5156000000010000: top objectClass;vucsn-4d7e5156000000010000: person objectClass;vucsn-4d7e5156000000010000: organizationalPerson objectClass;vucsn-4d7e5156000000010000: inetorgperson objectClass;vucsn-4d7e5163000000010000: nsTombstone sn;vucsn-4d7e5156000000010000: user0 cn;vucsn-4d7e5156000000010000: test user0 userPassword;vucsn-4d7e5156000000010000: {SSHA}QscFOIbdsvg6+qdW8Dy+MKwuvPdlIEu w0tChBg== creatorsName;vucsn-4d7e5156000000010000: uid=admin,ou=administrators,ou=topolo gymanagement,o=netscaperoot modifiersName;vucsn-4d7e5156000000010000: uid=admin,ou=administrators,ou=topol ogymanagement,o=netscaperoot createTimestamp;vucsn-4d7e5156000000010000: 20110314173310Z modifyTimestamp;vucsn-4d7e5156000000010000: 20110314173310Z nsParentUniqueId: 89ad3283-4e5911e0-98dfa788-5bfd3127 nscpEntryDN: uid=tuser0,ou=people,dc=example,dc=com entryid: 10 This makes the tombstone search fail. $ ldapsearch ... -D 'cn=directory manager' -w password -b "dc=example,dc=com" "(&(objectclass=nstombstone)(uid=*))" dn $
Created attachment 486083 [details] git patch file (master) Description: When nsslapd-subtree-rename-switch is on, a tombstone entry has a special RDN which looks like this: nsuniqueid=042d8081-...-ca8fe9f7,<original_leaf_rdn> This special format was not treated properly. This patch adds the code to handle the special tombstone RDN, where an internal entry has the above RDN and points the correct parent entry.
Steps to verify 0. set up a replication topology (I used one master - one consumer) 1. add some entries and delete them. search them with ldapsearch: $ ldapsearch ... -D 'cn=directory manager' -w <pw> -b "<suffix>" "(objectclass=nstombstone)" dn You should get the search results something like this: dn: nsuniqueid=9a63a581-...-ca8fe9f7,uid=tuser1,ou=People,dc=example,dc=com dn: nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=example,dc=com 2. stop the master export the database with '-r' option /usr/lib[64]/dirsrv/slapd-ID/db2ldif -n <backend> -r -a /tmp/test.ldif 3. import back the ldif file to the master /usr/lib[64]/dirsrv/slapd-ID/ldif2db -n <backend> -i /tmp/test.ldif 4. start the master run the above ldapsearch to check if it returns the same result. 5. initialize the consumer run the above ldapsearch against the consumer to check if it returns the same result. 6. stop the master Open /etc/dirsrv/slapd-ID/dse.ldif and modify nsds5ReplicaPurgeDelay in the replica entry, e.g. (note: suffix is "dc=example,dc=com"): dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config so that: nsds5ReplicaPurgeDelay: 60 and add nsds5ReplicaTombstonePurgeInterval: 60 Restart the master Wait for a minute and add any entry to the master Run the above ldapsearch and check that the search result only shows an RUV: dn: nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=example,dc=com
Reviewed by Nathan (Thank you!!!) Pushed to master. $ git merge bug684996 Updating 2c8637c..bc40d7b Fast-forward ldap/servers/slapd/back-ldbm/import-threads.c | 52 ++++++++++++++++- ldap/servers/slapd/back-ldbm/ldif2ldbm.c | 24 ++++++++- ldap/servers/slapd/entry.c | 75 ++++++++++++++++++++++++- ldap/servers/slapd/rdn.c | 36 ++++++++++++ ldap/servers/slapd/slap.h | 3 + ldap/servers/slapd/slapi-plugin.h | 7 ++ 6 files changed, 189 insertions(+), 8 deletions(-) $ git push Counting objects: 23, done. Delta compression using up to 4 threads. Compressing objects: 100% (12/12), done. Writing objects: 100% (12/12), 3.76 KiB, done. Total 12 (delta 10), reused 0 (delta 0) To ssh://git.fedorahosted.org/git/389/ds.git 2c8637c..bc40d7b master -> master Note: pending the push to 389-ds-base-1.2.8 waiting for the approvals on https://bugzilla.redhat.com/show_bug.cgi?id=688730 (copy of this bug).
Cherry-picked commit bc40d7bf88c015c18ff6123c400119bd6b9b2966 and pushed to 389-ds-base-1.2.8 branch: $ git cherry-pick bc40d7bf88c015c18ff6123c400119bd6b9b2966 Finished one cherry-pick. [ds128-local 2ba240b] Bug 684996 - Exported tombstone cannot be imported correctly 6 files changed, 189 insertions(+), 8 deletions(-) $ git push origin ds128-local:389-ds-base-1.2.8 Counting objects: 23, done. Delta compression using up to 4 threads. Compressing objects: 100% (12/12), done. Writing objects: 100% (12/12), 3.76 KiB, done. Total 12 (delta 10), reused 0 (delta 0) To ssh://git.fedorahosted.org/git/389/ds.git fb7547f..2ba240b ds128-local -> 389-ds-base-1.2.8
Steps to verify 0. set up a replication topology (I used one master - one consumer) 1. add some entries and delete them. <Amita> [amsharma@rhel61-ds90-amita scripts]$ ldapmodify -x -h localhost -p 20100 -D "cn=Directory Manager" -w Secret123 << EOF > dn: uid=amita,ou=people,dc=replsuffix,dc=com > changetype: add > objectClass: top > objectClass: person > objectClass: inetorgperson > sn: testkrbuser > cn: kkk testkrbuser > userPassword: redhat > EOF adding new entry "uid=amita,ou=people,dc=replsuffix,dc=com" [amsharma@rhel61-ds90-amita scripts]$ ldapdelete -x -h localhost -p 20100 -D "cn=Directory Manager" -w Secret123 "uid=amita,ou=people,dc=replsuffix,dc=com" </Amita> search them with ldapsearch: $ ldapsearch ... -D 'cn=directory manager' -w <pw> -b "<suffix>" "(objectclass=nstombstone)" dn You should get the search results something like this: dn: nsuniqueid=9a63a581-...-ca8fe9f7,uid=tuser1,ou=People,dc=example,dc=com dn: nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=example,dc=com <Amita> ldapsearch -h localhost -p 20100 -D "cn=Directory Manager" -w Secret123 -b "dc=replsuffix,dc=com" "(objectclass=nstombstone)" dn # extended LDIF # # LDAPv3 # base <dc=replsuffix,dc=com> with scope subtree # filter: (objectclass=nstombstone) # requesting: dn # # ffffffff-ffffffff-ffffffff-ffffffff, replsuffix.com dn: nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=replsuffix,dc=com # 48b78481-723311e0-bb1bfc39-318906b0, amita, People, replsuffix.com dn: nsuniqueid=48b78481-723311e0-bb1bfc39-318906b0,uid=amita,ou=People,dc=repl suffix,dc=com # f5decc81-723411e0-bb1bfc39-318906b0, amita, People, replsuffix.com dn: nsuniqueid=f5decc81-723411e0-bb1bfc39-318906b0,uid=amita,ou=People,dc=repl suffix,dc=com # 86b8a681-723b11e0-bb1bfc39-318906b0, amita, People, replsuffix.com dn: nsuniqueid=86b8a681-723b11e0-bb1bfc39-318906b0,uid=amita,ou=People,dc=repl suffix,dc=com # search result search: 2 result: 0 Success # numResponses: 5 # numEntries: 4 </Amita> 2. stop the master export the database with '-r' option /usr/lib[64]/dirsrv/slapd-ID/db2ldif -n <backend> -r -a /tmp/test.ldif <Amita> /usr/lib64/dirsrv/slapd-M1/stop-slapd [amsharma@rhel61-ds90-amita slapd-M1]$ ./db2ldif -n repman20100 -r -a /tmp/test.ldif Exported ldif file: /tmp/test.ldif [29/Apr/2011:14:13:12 +051800] - I'm resizing my cache now...cache was 20000000 and is now 8000000 ldiffile: /tmp/test.ldif [29/Apr/2011:14:13:13 +051800] - export repman20100: Processed 2 entries (100%). [29/Apr/2011:14:13:13 +051800] - Waiting for 4 database threads to stop [29/Apr/2011:14:13:14 +051800] - All database threads now stopped </Amita> 3. import back the ldif file to the master /usr/lib[64]/dirsrv/slapd-ID/ldif2db -n <backend> -i /tmp/test.ldif <Amita> ./ldif2db -n repman20100 -i /tmp/test.ldif importing data ... [29/Apr/2011:14:16:51 +051800] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [29/Apr/2011:14:16:51 +051800] - check_and_set_import_cache: pagesize: 4096, pages: 255398, procpages: 50007 [29/Apr/2011:14:16:51 +051800] - WARNING: After allocating import cache 408636KB, the available memory is 612956KB, which is less than the soft limit 1048576KB. You may want to decrease the import cache size and rerun import. [29/Apr/2011:14:16:51 +051800] - Import allocates 408636KB import cache. [29/Apr/2011:14:16:51 +051800] - import repman20100: Beginning import job... [29/Apr/2011:14:16:51 +051800] - import repman20100: Index buffering enabled with bucket size 100 [29/Apr/2011:14:16:51 +051800] - import repman20100: Processing file "/tmp/test.ldif" [29/Apr/2011:14:16:51 +051800] - import repman20100: Finished scanning file "/tmp/test.ldif" (2 entries) [29/Apr/2011:14:16:52 +051800] - import repman20100: Workers finished; cleaning up... [29/Apr/2011:14:16:52 +051800] - import repman20100: Workers cleaned up. [29/Apr/2011:14:16:52 +051800] - import repman20100: Cleaning up producer thread... [29/Apr/2011:14:16:52 +051800] - import repman20100: Indexing complete. Post-processing... [29/Apr/2011:14:16:52 +051800] - import repman20100: Flushing caches... [29/Apr/2011:14:16:52 +051800] - import repman20100: Closing files... [29/Apr/2011:14:16:52 +051800] - All database threads now stopped [29/Apr/2011:14:16:52 +051800] - import repman20100: Import complete. Processed 2 entries in 1 seconds. (2.00 entries/sec) </Amita> 4. start the master run the above ldapsearch to check if it returns the same result. <Amita> /usr/lib64/dirsrv/slapd-M1/start-slapd ldapsearch -h localhost -p 20100 -D "cn=Directory Manager" -w Secret123 -b "dc=replsuffix,dc=com" "(objectclass=nstombstone)" dn # extended LDIF # # LDAPv3 # base <dc=replsuffix,dc=com> with scope subtree # filter: (objectclass=nstombstone) # requesting: dn # # ffffffff-ffffffff-ffffffff-ffffffff, replsuffix.com dn: nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=replsuffix,dc=com # 48b78481-723311e0-bb1bfc39-318906b0, amita, People, replsuffix.com dn: nsuniqueid=48b78481-723311e0-bb1bfc39-318906b0,uid=amita,ou=People,dc=repl suffix,dc=com # f5decc81-723411e0-bb1bfc39-318906b0, amita, People, replsuffix.com dn: nsuniqueid=f5decc81-723411e0-bb1bfc39-318906b0,uid=amita,ou=People,dc=repl suffix,dc=com # 86b8a681-723b11e0-bb1bfc39-318906b0, amita, People, replsuffix.com dn: nsuniqueid=86b8a681-723b11e0-bb1bfc39-318906b0,uid=amita,ou=People,dc=repl suffix,dc=com # search result search: 2 result: 0 Success # numResponses: 5 # numEntries: 4 </Amita> 5. initialize the consumer run the above ldapsearch against the consumer to check if it returns the same result. <Amita> ldapsearch -h localhost -p 20102 -D "cn=Directory Manager" -w Secret123 -b "dc=replsuffix,dc=com" "(objectclass=nstombstone)" dn # extended LDIF # # LDAPv3 # base <dc=replsuffix,dc=com> with scope subtree # filter: (objectclass=nstombstone) # requesting: dn # # ffffffff-ffffffff-ffffffff-ffffffff, replsuffix.com dn: nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=replsuffix,dc=com # 48b78481-723311e0-bb1bfc39-318906b0, amita, People, replsuffix.com dn: nsuniqueid=48b78481-723311e0-bb1bfc39-318906b0,uid=amita,ou=People,dc=repl suffix,dc=com # f5decc81-723411e0-bb1bfc39-318906b0, amita, People, replsuffix.com dn: nsuniqueid=f5decc81-723411e0-bb1bfc39-318906b0,uid=amita,ou=People,dc=repl suffix,dc=com # 86b8a681-723b11e0-bb1bfc39-318906b0, amita, People, replsuffix.com dn: nsuniqueid=86b8a681-723b11e0-bb1bfc39-318906b0,uid=amita,ou=People,dc=repl suffix,dc=com # search result search: 2 result: 0 Success # numResponses: 5 # numEntries: 4 </Amita> 6. stop the master Open /etc/dirsrv/slapd-ID/dse.ldif and modify nsds5ReplicaPurgeDelay in the replica entry, e.g. (note: suffix is "dc=example,dc=com"): dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config so that: nsds5ReplicaPurgeDelay: 60 and add nsds5ReplicaTombstonePurgeInterval: 60 Restart the master Wait for a minute and add any entry to the master Run the above ldapsearch and check that the search result only shows an RUV: dn: nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=example,dc=com <Amita> /usr/lib64/dirsrv/slapd-M1/stop-slapd vim /etc/dirsrv/slapd-M1/dse.ldif added nsds5ReplicaPurgeDelay: 60 nsds5ReplicaTombstonePurgeInterval: 60 /usr/lib64/dirsrv/slapd-M1/start-slapd ldapmodify -x -h localhost -p 20100 -D "cn=Directory Manager" -w Secret123 << EOF > dn: uid=amita,ou=people,dc=replsuffix,dc=com > changetype: add > objectClass: top > objectClass: person > objectClass: inetorgperson > sn: testkrbuser > cn: kkk testkrbuser > userPassword: redhat > EOF ????? Why I am not able to see the message from slapd "adding new entry "uid=amita,ou=people,dc=replsuffix,dc=com", It was showing before making changes in dse.ldif????? ldapsearch -h localhost -p 20100 -D "cn=Directory Manager" -w Secret123 -b "dc=replsuffix,dc=com" "(objectclass=nstombstone)" dn # extended LDIF # # LDAPv3 # base <dc=replsuffix,dc=com> with scope subtree # filter: (objectclass=nstombstone) # requesting: dn # # ffffffff-ffffffff-ffffffff-ffffffff, replsuffix.com dn: nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=replsuffix,dc=com # 86b8a681-723b11e0-bb1bfc39-318906b0, amita, People, replsuffix.com dn: nsuniqueid=86b8a681-723b11e0-bb1bfc39-318906b0,uid=amita,ou=People,dc=repl suffix,dc=com # search result search: 2 result: 0 Success # numResponses: 3 # numEntries: 2 ?????????? Check above, Is it what is expected?????????? ldapsearch -h localhost -p 20102 -D "cn=Directory Manager" -w Secret123 -b "dc=replsuffix,dc=com" "(objectclass=nstombstone)" dn # extended LDIF # # LDAPv3 # base <dc=replsuffix,dc=com> with scope subtree # filter: (objectclass=nstombstone) # requesting: dn # # ffffffff-ffffffff-ffffffff-ffffffff, replsuffix.com dn: nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=replsuffix,dc=com # 48b78481-723311e0-bb1bfc39-318906b0, amita, People, replsuffix.com dn: nsuniqueid=48b78481-723311e0-bb1bfc39-318906b0,uid=amita,ou=People,dc=repl suffix,dc=com # f5decc81-723411e0-bb1bfc39-318906b0, amita, People, replsuffix.com dn: nsuniqueid=f5decc81-723411e0-bb1bfc39-318906b0,uid=amita,ou=People,dc=repl suffix,dc=com # 86b8a681-723b11e0-bb1bfc39-318906b0, amita, People, replsuffix.com dn: nsuniqueid=86b8a681-723b11e0-bb1bfc39-318906b0,uid=amita,ou=People,dc=repl suffix,dc=com # search result search: 2 result: 0 Success # numResponses: 5 # numEntries: 4 </Amita> ?????????? Why Master and Relica are showing different results for this ldapserach now, which was not the case before making the changes in dse.ldif??????????
Rich responded to your NEEDINFO in [Bug 688730] Exported tombstone cannot be imported correctly.
VERIFIED.