Created attachment 366516 [details] description of problem by ssorce Description of problem: Requirement from IPA/Samba
Current implementation related to this topic a) EntryUSN per backend b) Db2ldif does not export entryUSN c) EntryUSN is internally a 64-bit unsigned long value. d) When the server starts, internal entryUSN counter picks up the largest entryUSN in the entryUSN index and starts from the value incremented by 1. Issues of the global entryUSN 1) possible inconsistency among multiple backends We could think of some out-of-control use cases. For instance, - Since DS's import/export is done per backend, if only one of the backends out of multiple is exported and imported, the backend loses the entryUSN info and the rest won't. 2) Exporting entryUSN should be suppressed or not? First of all, entryUSN is not a standard attribute. To maintain the compatibility of the ldif file with the other LDAP servers, it'd be safe not to have the entryUSN in the exported ldif file. But to avoid the issue (1) listed above, we may need to include entryUSN in the exported ldif file. This could cause more problems. * If the entryUSN values in the exported ldif file are manually modified and some unexpected value, e.g., -1, is set. Once the ldif file is imported, the entryUSN counter starts from 18446744073709551615 and the next value the counter provides is 0 (even if some entry already has the value). There is no way to prevent it. (Rather, I feel we don't want to do it. If entryUSN implementation is too strict -- e.g., making it a unique attribute, operation might fail just because of entryUSN's conflict. That does not sound preferable.) * If the entryUSN values in the exported ldif file are manually modified and if it's introduced any conflicts with the entryUSN values in the other backends, there is no way to avoid it. (Since they are in the different backends, even attribute unique plugin cannot be used.) * In the MMR topology, if an ldif file exported from the server A is imported to the server B, the entryUSNs on the server B will most likely have duplicated entryUSN values.
I don't think you should ever save entryUSN in an ldif as it should be considered a server generated attribute. You should probably error out if someone tries to do so. If a database is exported and then re-imported you'd loose all entry USN and get new ones, but that's fine because IMHO and export/import operation should be regarded as a modification to all entries of that database. Simo.
Thanks, Simo. BTW, import does not assign any entry USN. Only the ordinary operations -- add, modify, modrdn, delete.
Then I would recommend creating a slapi task in the entryUSN plugin that can "tag" all entries that miss the attribute, otherwise you can end up with trees where part of the entries do not have the entryUSN attribute creately reducing it's usefulnes.
We can add a code to add "entryusn: 0" to all the entries when an ldif is imported, I think. Then, the entryUSN counter starts from 1. Does it work? What if entries in the ldif file to be imported contain entryusn? (Most likely, manually added entryusn's) Should we override the value with 0?
(In reply to comment #5) > We can add a code to add "entryusn: 0" to all the entries when an ldif is > imported, I think. Then, the entryUSN counter starts from 1. Does it work? I like this approach. We would also want to make sure lastUSN is set to 0 in the root DSE at the end of import (maybe that is taken care of automatically by your code already?). > > What if entries in the ldif file to be imported contain entryusn? (Most > likely, manually added entryusn's) Should we override the value with 0? I think so. We don't support importing entryUSN anyway, and this gets rid of any chance of someone putting a bogus value like "-1" causing a problem.
Sounds good! Thank you for your comments! I think the additional spec is useful and doable.
Created attachment 441366 [details] git patch file (master) Fix description: 1. Introduced a config parameter nsslapd-entryusn-global: on|off to enable | disable the global mode. By default, off. In the global mode, search on root dse returns "lastusn: <num>" without the backend subtype (e.g., "lastusn;userroot: <num>") 2. Added slapi_get_next_suffix_ext to mapping_tree.c, which visits children as well as siblings in the mapping tree. (Note: slapi_get_next_suffix does just siblings.) 3. import (ldif2db) adds "entryusn: 0" to every entry unless the entry already contains the entryusn attribute. 4. ldbm_back_delete, ldbm_back_modify, ldbm_back_modrdn: set ldap_result_code to pblock so that bepost plugin could see if the operation was successful or not.
Reviewed by Rich (Thanks!!!) Pushed to master. $ git push Counting objects: 67, done. Delta compression using up to 4 threads. Compressing objects: 100% (41/41), done. Writing objects: 100% (41/41), 14.98 KiB, done. Total 41 (delta 33), reused 0 (delta 0) To ssh://git.fedorahosted.org/git/389/ds.git a2bcd81..cc36301 master -> master
Hi Noriko, Request you to please add the verification steps. Thanks, Amita
Steps to verify 0. create more than one backend. E.g., dc=example,dc=com (backend userRoot) dc=sub,dc=example,dc=com (backend subRoot; configure dc=example,dc=com a parent suffix) dc=test,dc=com (backnd testRoot) 1. stop the server 2. edit /etc/dirsrv/slapd-ID/dse.ldif to enable entryUSN plugin and its global mode so that ... cn=config nsslapd-entryusn-global: on ... dn: cn=USN,cn=plugins,cn=config nsslapd-pluginenabled: off 3. start the server 4. $ ldapsearch ... -b "" -s base "(objectclass=*)" lastusn the output is supposed to look like this: lastusn: <num> 5. import an ldif file to each backend # ldif2db.pl ... -n <backend> -i /path/to/ldif get the entryusn values $ ldapsearch ... -b "dc=example,dc=com" "(objectclass=*)" entryusn $ ldapsearch ... -b "dc=test,dc=com" "(objectclass=*)" entryusn the entryusn values are supposed to be 0. 6. do some modifications to the all backends (add, mod, del, etc.) 7. get the entryusn values again $ ldapsearch ... -b "dc=example,dc=com" "(objectclass=*)" entryusn $ ldapsearch ... -b "dc=test,dc=com" "(objectclass=*)" entryusn the entryusn values are supposed to be unique across the backends (except the original 0s).
dn: cn=USN,cn=plugins,cn=config objectClass: top objectClass: nsSlapdPlugin objectClass: extensibleObject cn: USN nsslapd-pluginPath: libusn-plugin nsslapd-pluginInitfunc: usn_init nsslapd-pluginType: object nsslapd-pluginEnabled: on nsslapd-plugin-depends-on-type: database AND dn: cn=config cn: config objectClass: top objectClass: extensibleObject objectClass: nsslapdConfig nsslapd-entryusn-global: on I have restarted the server. ldapsearch -x -h localhost -p 389 -D "cn=directory manager" -w Secret123 -b "" -s base "(objectclass=*)" lastusn # dn: lastusn: -1 ok, After import .... [root@testvm slapd-testvm]# ldapsearch -x -h localhost -p 389 -D "cn=directory manager" -w Secret123 -b "ou=people,dc=testnew,dc=com" "entryusn" # extended LDIF # # LDAPv3 # base <ou=people,dc=testnew,dc=com> with scope subtree # filter: (objectclass=*) # requesting: entryusn # # people, testnew.com dn: ou=people,dc=testnew,dc=com entryusn: 0 # tmorris, people, testnew.com dn: uid=tmorris,ou=people,dc=testnew,dc=com entryusn: 0 # search result search: 2 result: 0 Success # numResponses: 3 # numEntries: 2 root@testvm slapd-testvm]# ldapsearch -x -h localhost -p 389 -D "cn=directory manager" -w Secret123 -b "ou=people,dc=testnew,dc=com" "entryusn" # extended LDIF # # LDAPv3 # base <ou=people,dc=testnew,dc=com> with scope subtree # filter: (objectclass=*) # requesting: entryusn # # people, testnew.com dn: ou=people,dc=testnew,dc=com entryusn: 0 # tmorris, people, testnew.com dn: uid=tmorris,ou=people,dc=testnew,dc=com entryusn: 0 # hhmmm, people, testnew.com dn: uid=hhmmm,ou=people,dc=testnew,dc=com entryusn: 3 # search result search: 2 result: 0 Success # numResponses: 4 # numEntries: 3