Bug 531642 - EntryUSN: RFE: a configuration option to make entryusn "global"
Summary: EntryUSN: RFE: a configuration option to make entryusn "global"
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: 389
Classification: Retired
Component: Server - Plugins
Version: 1.2.1
Hardware: All
OS: Linux
high
medium
Target Milestone: ---
Assignee: Noriko Hosoi
QA Contact: Viktor Ashirov
URL:
Whiteboard:
Depends On:
Blocks: 389_1.2.7 639035
TreeView+ depends on / blocked
 
Reported: 2009-10-28 23:46 UTC by Noriko Hosoi
Modified: 2015-12-07 16:47 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-12-07 16:47:58 UTC


Attachments (Terms of Use)
description of problem by ssorce@redhat.com (1.52 KB, text/plain)
2009-10-28 23:46 UTC, Noriko Hosoi
no flags Details
git patch file (master) (25.92 KB, patch)
2010-08-27 00:52 UTC, Noriko Hosoi
nhosoi: review?
nhosoi: review?
rmeggins: review+
Details | Diff

Description Noriko Hosoi 2009-10-28 23:46:59 UTC
Created attachment 366516 [details]
description of problem by ssorce@redhat.com

Description of problem:
Requirement from IPA/Samba

Comment 1 Noriko Hosoi 2009-10-29 17:26:25 UTC
Current implementation related to this topic
a) EntryUSN per backend
b) Db2ldif does not export entryUSN
c) EntryUSN is internally a 64-bit unsigned long value.
d) When the server starts, internal entryUSN counter picks up the largest
entryUSN in the entryUSN index and starts from the value incremented by 1.

Issues of the global entryUSN
1) possible inconsistency among multiple backends
We could think of some out-of-control use cases.  For instance,
- Since DS's import/export is done per backend, if only one of the backends 
  out of multiple is exported and imported, the backend loses the entryUSN 
  info and the rest won't.

2) Exporting entryUSN should be suppressed or not?
First of all, entryUSN is not a standard attribute.  To maintain the compatibility of the ldif file with the other LDAP servers, it'd be safe not 
to have the entryUSN in the exported ldif file.

But to avoid the issue (1) listed above, we may need to include entryUSN in 
the exported ldif file.  This could cause more problems.
* If the entryUSN values in the exported ldif file are manually modified 
and some unexpected value, e.g., -1, is set.  Once the ldif file is imported,
the entryUSN counter starts from 18446744073709551615 and the next value
the counter provides is 0 (even if some entry already has the value).
There is no way to prevent it.  (Rather, I feel we don't want to do it.
If entryUSN implementation is too strict -- e.g., making it a unique 
attribute, operation might fail just because of entryUSN's conflict.
That does not sound preferable.)

* If the entryUSN values in the exported ldif file are manually modified 
and if it's introduced any conflicts with the entryUSN values in the other 
backends, there is no way to avoid it.  (Since they are in the different
backends, even attribute unique plugin cannot be used.)

* In the MMR topology, if an ldif file exported from the server A is imported
to the server B, the entryUSNs on the server B will most likely have duplicated
entryUSN values.

Comment 2 Simo Sorce 2009-10-29 17:44:09 UTC
I don't think you should ever save entryUSN in an ldif as it should be considered a server generated attribute. You should probably error out if someone tries to do so.
If a database is exported and then re-imported you'd loose all entry USN and get new ones, but that's fine because IMHO and export/import operation should be regarded as a modification to all entries of that database.

Simo.

Comment 3 Noriko Hosoi 2009-10-29 18:41:58 UTC
Thanks, Simo.  BTW, import does not assign any entry USN. Only the ordinary operations -- add, modify, modrdn, delete.

Comment 4 Simo Sorce 2009-10-29 19:16:20 UTC
Then I would recommend creating a slapi task in the entryUSN plugin that can "tag" all entries that miss the attribute, otherwise you can end up with trees where part of the entries do not have the entryUSN attribute creately reducing it's usefulnes.

Comment 5 Noriko Hosoi 2009-10-29 20:10:14 UTC
We can add a code to add "entryusn: 0" to all the entries when an ldif is imported, I think.  Then, the entryUSN counter starts from 1.  Does it work?

What if entries in the ldif file to be imported contain entryusn?  (Most likely, manually added entryusn's)  Should we override the value with 0?

Comment 6 Nathan Kinder 2009-10-29 20:28:11 UTC
(In reply to comment #5)
> We can add a code to add "entryusn: 0" to all the entries when an ldif is
> imported, I think.  Then, the entryUSN counter starts from 1.  Does it work?

I like this approach.  We would also want to make sure lastUSN is set to 0 in the root DSE at the end of import (maybe that is taken care of automatically by your code already?).

> 
> What if entries in the ldif file to be imported contain entryusn?  (Most
> likely, manually added entryusn's)  Should we override the value with 0?

I think so.  We don't support importing entryUSN anyway, and this gets rid of any chance of someone putting a bogus value like "-1" causing a problem.

Comment 7 Noriko Hosoi 2009-10-29 20:42:25 UTC
Sounds good!  Thank you for your comments!  I think the additional spec is
useful and doable.

Comment 10 Noriko Hosoi 2010-08-27 00:52:45 UTC
Created attachment 441366 [details]
git patch file (master)

Fix description:
1. Introduced a config parameter nsslapd-entryusn-global: on|off to
   enable | disable the global mode.  By default, off.
   In the global mode, search on root dse returns "lastusn: <num>"
   without the backend subtype (e.g., "lastusn;userroot: <num>")
2. Added slapi_get_next_suffix_ext to mapping_tree.c, which visits
   children as well as siblings in the mapping tree.
   (Note: slapi_get_next_suffix does just siblings.)
3. import (ldif2db) adds "entryusn: 0" to every entry unless the
   entry already contains the entryusn attribute.
4. ldbm_back_delete, ldbm_back_modify, ldbm_back_modrdn: set
   ldap_result_code to pblock so that bepost plugin could see if
   the operation was successful or not.

Comment 11 Noriko Hosoi 2010-08-31 17:42:10 UTC
Reviewed by Rich (Thanks!!!)

Pushed to master.
$ git push
Counting objects: 67, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (41/41), done.
Writing objects: 100% (41/41), 14.98 KiB, done.
Total 41 (delta 33), reused 0 (delta 0)
To ssh://git.fedorahosted.org/git/389/ds.git
   a2bcd81..cc36301  master -> master

Comment 12 Amita Sharma 2011-05-25 13:13:47 UTC
Hi Noriko,

Request you to please add the verification steps.

Thanks,
Amita

Comment 13 Noriko Hosoi 2011-05-25 17:22:34 UTC
Steps to verify
0. create more than one backend.  E.g.,
   dc=example,dc=com (backend userRoot)
   dc=sub,dc=example,dc=com
   (backend subRoot; configure dc=example,dc=com a parent suffix)
   dc=test,dc=com (backnd testRoot)
1. stop the server
2. edit /etc/dirsrv/slapd-ID/dse.ldif to enable entryUSN plugin 
   and its global mode so that ...
     cn=config
     nsslapd-entryusn-global: on
     ...

     dn: cn=USN,cn=plugins,cn=config
     nsslapd-pluginenabled: off
3. start the server
4. $ ldapsearch ... -b "" -s base "(objectclass=*)" lastusn
   the output is supposed to look like this:
     lastusn: <num>
5. import an ldif file to each backend
   # ldif2db.pl ... -n <backend> -i /path/to/ldif
   get the entryusn values
   $ ldapsearch ... -b "dc=example,dc=com" "(objectclass=*)" entryusn
   $ ldapsearch ... -b "dc=test,dc=com" "(objectclass=*)" entryusn
   the entryusn values are supposed to be 0.
6. do some modifications to the all backends (add, mod, del, etc.)
7. get the entryusn values again
   $ ldapsearch ... -b "dc=example,dc=com" "(objectclass=*)" entryusn
   $ ldapsearch ... -b "dc=test,dc=com" "(objectclass=*)" entryusn
   the entryusn values are supposed to be unique across the backends
   (except the original 0s).

Comment 16 Amita Sharma 2011-06-01 09:07:31 UTC
dn: cn=USN,cn=plugins,cn=config
objectClass: top
objectClass: nsSlapdPlugin
objectClass: extensibleObject
cn: USN
nsslapd-pluginPath: libusn-plugin
nsslapd-pluginInitfunc: usn_init
nsslapd-pluginType: object
nsslapd-pluginEnabled: on
nsslapd-plugin-depends-on-type: database
AND
dn: cn=config
cn: config
objectClass: top
objectClass: extensibleObject
objectClass: nsslapdConfig
nsslapd-entryusn-global: on

I have restarted the server.

ldapsearch -x -h localhost -p 389 -D "cn=directory manager" -w Secret123 -b "" -s base "(objectclass=*)" lastusn
#
dn:
lastusn: -1

ok, After import ....

[root@testvm slapd-testvm]# ldapsearch -x -h localhost -p 389 -D "cn=directory manager" -w Secret123 -b "ou=people,dc=testnew,dc=com" "entryusn"
# extended LDIF
#
# LDAPv3
# base <ou=people,dc=testnew,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: entryusn 
#

# people, testnew.com
dn: ou=people,dc=testnew,dc=com
entryusn: 0

# tmorris, people, testnew.com
dn: uid=tmorris,ou=people,dc=testnew,dc=com
entryusn: 0

# search result
search: 2
result: 0 Success

# numResponses: 3
# numEntries: 2


root@testvm slapd-testvm]# ldapsearch -x -h localhost -p 389 -D "cn=directory manager" -w Secret123 -b "ou=people,dc=testnew,dc=com" "entryusn"
# extended LDIF
#
# LDAPv3
# base <ou=people,dc=testnew,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: entryusn 
#

# people, testnew.com
dn: ou=people,dc=testnew,dc=com
entryusn: 0

# tmorris, people, testnew.com
dn: uid=tmorris,ou=people,dc=testnew,dc=com
entryusn: 0

# hhmmm, people, testnew.com
dn: uid=hhmmm,ou=people,dc=testnew,dc=com
entryusn: 3

# search result
search: 2
result: 0 Success

# numResponses: 4
# numEntries: 3


Note You need to log in before you can comment on or make changes to this bug.