Bug 2075017

Summary: RFE - performance improvement - store full entry DN in database record
Product: Red Hat Directory Server Reporter: mreynolds
Component: 389-ds-baseAssignee: mreynolds
Status: CLOSED ERRATA QA Contact: LDAP QA Team <idm-ds-qe-bugs>
Severity: high Docs Contact: Evgenia Martynyuk <emartyny>
Priority: high    
Version: 12.0CC: bsmejkal, gfialova, idm-ds-dev-bugs, kperrier, pasik, pcech, vashirov
Target Milestone: DS12.1Keywords: FutureFeature, TestCaseProvided, Triaged
Target Release: dirsrv-12.2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: sync-to-jira
Fixed In Version: redhat-ds-12-9020020221130212339.1674d57 Doc Type: Enhancement
Doc Text:
.Default behavior change: Directory Server now returns a DN in exactly the same spelling as it was added to the database With the new `nsslapd-return-original-entrydn` parameter under the `cn=config` entry, you can manage how Directory Server returns the distinguished name (DN) of entries during search operations. By default, the `nsslapd-return-original-entrydn` parameter is set to `on`, and Directory Server returns the DN exactly how it was originally added to the database. For example, you added or modified the entry `uid=User,ou=PEople,dc=ExaMPlE,DC=COM`, and with the setting turned on, Directory Server returns the same spelling of the DN for the entry: `uid=User,ou=PEople,dc=ExaMPlE,DC=COM`. When the `nsslapd-return-original-entrydn` parameter is set to `off`, Directory Server generates the entry DN by putting together a Relative DN (RDN) of the entry and the base DN that is stored in the database suffix configuration under `cn=userroot,cn=ldbm database,cn=plugins,cn=config`. If you set the base DN to `ou=people,dc=example,dc=com`, and the `nsslapd-return-original-entrydn` setting is `off`, Directory Server returns `uid=User,ou=people,dc=example,dc=com` during searches and not the spelling of the DN when you added the entry to the database.
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-05-30 09:40:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description mreynolds 2022-04-13 12:33:56 UTC
Description of problem:

While testing "nsslapd-subtree-rename-switch=off" there was noticeable performance improvement.  The current implementation generates the DN dynamically by using the entryrdn attribute in the database record, and then obtaining the parent portion of the DN from other database indexes.  

Disabling subtree-rename (nsslapd-subtree-rename-switch=off) improves throughput/response_time performance especially when the number of clients increases.

The range of improvement on highest load (high number of clients) ranges up to ~10%


subtree-rename=OFF 
----------------------------------------------------
Workers         Clients           SRCH/s        GAIN
15              40              142302.868      + 0%
15              45              157062.354      + 4%    
15              50              165993.278      +12%
15              55              168967.106      +15%

20              40              133749.551      + 2%
20              45              148529.026      + 0%
20              50              150216.182      + 1%
20              55              114247.201      +13%

25              40              123818.029      + 1%
25              45              141934.678      + 1%
25              50              153096.810      + 0%
25              55              109355.695      +10%


subtree-rename=ON
----------------------------------------------------
Workers         Clients         SRCH/s
15              40              142297.329
15              45              150931.420
15              50              147570.526
15              55              146769.262

20              40              130967.392
20              45              148365.746
20              50              148903.385
20              55              121286.399

25              40              122408.771
25              45              140797.527
25              50              151812.222
25              55               98683.023


We could also maintain the existing operational attribute "entrydn" and only use it for DN "generation" on searches.  It would need to be updated on adds, and modrdns.  There could also be an impact on tombstones queries as well.

Comment 1 mreynolds 2022-04-13 22:21:12 UTC
Upstream ticket:

https://github.com/389ds/389-ds-base/issues/5267

Comment 3 mreynolds 2022-09-14 16:02:04 UTC
*** Bug 2069744 has been marked as a duplicate of this bug. ***

Comment 8 mreynolds 2023-02-16 15:58:04 UTC
Developer verification:

[root@kvm-02-guest21 basic]# py.test ds_entrydn_test.py 
========================================================== test session starts ==========================================================
platform linux -- Python 3.9.16, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
389-ds-base: 2.2.6-1.module+el9dsrv+17949+63c5b04e
nss: 3.79.0-14.el9_0
nspr: 4.34.0-14.el9_0
openldap: 2.6.2-3.el9
cyrus-sasl: 2.1.27-21.el9
FIPS: disabled
rootdir: /root/source/389-ds-base/dirsrvtests, configfile: pytest.ini
collected 1 item                                                                                                                        

ds_entrydn_test.py .                                                                                                              [100%]

========================================================== 1 passed in 20.42s ===========================================================

Comment 9 Viktor Ashirov 2023-02-20 13:39:16 UTC
Build tested:
389-ds-base-2.2.6-1.module+el9dsrv+17949+63c5b04e.x86_64

I did some changes to the test, because I don't think the test was valid.

diff --git a/dirsrvtests/tests/suites/basic/ds_entrydn_test.py b/dirsrvtests/tests/suites/basic/ds_entrydn_test.py
index 7decc121d..9aa458a13 100644
--- a/dirsrvtests/tests/suites/basic/ds_entrydn_test.py
+++ b/dirsrvtests/tests/suites/basic/ds_entrydn_test.py
@@ -73,17 +73,17 @@ def test_dsentrydn(topo):
     assert user.get_attr_val_utf8('dsentrydn') == NEW_USER_DN

     # Check DN returned to client matches "dsEntryDN"
-    users = UserAccounts(inst, SUFFIX).list()
+    users = UserAccounts(inst, SUFFIX, rdn="ou=humans").list()
     for user in users:
-        if user.dn.startswith("tUser"):
+        if user.rdn.startswith("tUser"):
             assert user.dn == NEW_USER_DN
             break

-    # Disable 'nsslapd-return-original-entrydn' andcheck DN is normalized
-    inst.config.replace('nsslapd-return-original-entrydn', 'on')
-    users = UserAccounts(inst, SUFFIX).list()
+    # Disable 'nsslapd-return-original-entrydn' and check DN is normalized
+    inst.config.replace('nsslapd-return-original-entrydn', 'off')
+    users = UserAccounts(inst, SUFFIX, rdn="ou=humans").list()
     for user in users:
-        if user.dn.startswith("tUser"):
+        if user.rdn.startswith("tUser"):
             assert user.dn == NEW_USER_NORM_DN
             break

With these changes it fails:
>               assert user.dn == NEW_USER_NORM_DN
E               AssertionError: assert 'uid=tUser,ou...xample,DC=COM' == 'uid=tUser,ou...xample,dc=com'
E                 - uid=tUser,ou=humans,dc=example,dc=com
E                 ?                        ^       ^^ ^^^
E                 + uid=tUser,ou=humans,dc=Example,DC=COM
E                 ?                        ^       ^^ ^^^

dirsrvtests/tests/suites/basic/ds_entrydn_test.py:87: AssertionError

Looks like the server returns original entryDN even though nsslapd-return-original-entrydn was set to off.
But apparently the test can't even set it to off. When I tried to do that manually, I got an error:

ldap_modify: Server is unwilling to perform (53)
        additional info: nsslapd-subtree-rename-switch can't be modified while the server is running.

And why I edit dse.ldif manually, the server fails to start:
ERR - bdb_instance_start - nsslapd-subtree-rename-switch is off, while the instance userRoot is in the RDN format. Please change the value to on in dse.ldif.

So it doesn't look like I can change the server's behaviour by using this attribute, since it's always on.

Moving to ASSIGNED.

Comment 10 mreynolds 2023-02-20 13:52:33 UTC
FYI nsslapd-subtree-rename-switch should not be used/changed.  IIRC That setting no longer works in 2.2 with the changes for lmdb.  I'll take a look at your other findings though...

Comment 11 Viktor Ashirov 2023-02-20 14:07:43 UTC
If it can't be changed when lmdb is used, then I think we should modify additional info for the error 53 message to reflect that.

Comment 12 mreynolds 2023-02-20 14:41:58 UTC
(In reply to Viktor Ashirov from comment #11)
> If it can't be changed when lmdb is used, then I think we should modify
> additional info for the error 53 message to reflect that.

Agreed.

Also, I think the issue here is that you need to restart the server after changing the config value, but the fix changes how the entry is loaded into the entry cache.  So if you change the setting and entry is already in the cache then it has no impact.

[mareynol@fedora servers]$ ldapsearch -xLLL -D cn=dm -w XXXXXX -H ldap://localhost -b dc=example,dc=com cn=mark dsentrydn
dn: cn=mark,ou=People,dc=EXample,dc=COM
dsentrydn: cn=mark,ou=People,dc=EXample,dc=COM

Set nsslapd-return-original-entrydn to "off" and restart server:

[mareynol@fedora servers]$ ldapsearch -xLLL -D cn=dm -w XXXXXX -H ldap://localhost -b dc=example,dc=com cn=mark dsentrydn
dn: cn=mark,ou=people,dc=example,dc=com
dsentrydn: cn=mark,ou=People,dc=EXample,dc=COM


So the CI test needs some adjusting, but to me the fix is working as expected...

Comment 13 mreynolds 2023-02-20 14:55:09 UTC
(In reply to mreynolds from comment #10)
> FYI nsslapd-subtree-rename-switch should not be used/changed.  IIRC That
> setting no longer works in 2.2 with the changes for lmdb.  I'll take a look
> at your other findings though...

Ok I might have hastily misspoken.  This setting will not work in LMDB. However, it can be set in non-lmdb env, but you need to rebuild the database for it to work correctly.  I don't even see this setting in the docs anymore.  Anyway we are planning on removing this setting so I'm not sure how much more work should go into it.  Especially since it has not been used in years, it's not documented, and in fact it hasn't worked since 1.3.10 (or maybe 1.2.x)  I recently had to fix it for a customer as a workaround for their client needs in RHEL 8 which is why I know it hasn't worked in a very long time.

Comment 14 Viktor Ashirov 2023-02-20 15:56:44 UTC
I just realized I was trying to change the wrong attribute...
If I restart the instance, the test with my changes passes.
Moving to VERIFIED.

Comment 16 errata-xmlrpc 2023-05-30 09:40:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (redhat-ds:12 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:3344

Comment 17 Evgenia Martynyuk 2023-07-21 16:24:27 UTC
Added RN text, reviews by Mark https://issues.redhat.com/browse/IDMDS-3453
Moved to peer review.

Comment 18 Evgenia Martynyuk 2023-09-05 09:45:25 UTC
RN text has passed all reviews.