Upstream ticket:

https://github.com/389ds/389-ds-base/issues/5267

*** Bug 2069744 has been marked as a duplicate of this bug. ***

Design Doc:

https://www.port389.org/docs/389ds/design/store-case-sensitive-dn-design.html

Developer verification:

[root@kvm-02-guest21 basic]# py.test ds_entrydn_test.py 
========================================================== test session starts ==========================================================
platform linux -- Python 3.9.16, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
389-ds-base: 2.2.6-1.module+el9dsrv+17949+63c5b04e
nss: 3.79.0-14.el9_0
nspr: 4.34.0-14.el9_0
openldap: 2.6.2-3.el9
cyrus-sasl: 2.1.27-21.el9
FIPS: disabled
rootdir: /root/source/389-ds-base/dirsrvtests, configfile: pytest.ini
collected 1 item                                                                                                                        

ds_entrydn_test.py .                                                                                                              [100%]

========================================================== 1 passed in 20.42s ===========================================================

Build tested:
389-ds-base-2.2.6-1.module+el9dsrv+17949+63c5b04e.x86_64

I did some changes to the test, because I don't think the test was valid.

diff --git a/dirsrvtests/tests/suites/basic/ds_entrydn_test.py b/dirsrvtests/tests/suites/basic/ds_entrydn_test.py
index 7decc121d..9aa458a13 100644
--- a/dirsrvtests/tests/suites/basic/ds_entrydn_test.py
+++ b/dirsrvtests/tests/suites/basic/ds_entrydn_test.py
@@ -73,17 +73,17 @@ def test_dsentrydn(topo):
     assert user.get_attr_val_utf8('dsentrydn') == NEW_USER_DN

     # Check DN returned to client matches "dsEntryDN"
-    users = UserAccounts(inst, SUFFIX).list()
+    users = UserAccounts(inst, SUFFIX, rdn="ou=humans").list()
     for user in users:
-        if user.dn.startswith("tUser"):
+        if user.rdn.startswith("tUser"):
             assert user.dn == NEW_USER_DN
             break

-    # Disable 'nsslapd-return-original-entrydn' andcheck DN is normalized
-    inst.config.replace('nsslapd-return-original-entrydn', 'on')
-    users = UserAccounts(inst, SUFFIX).list()
+    # Disable 'nsslapd-return-original-entrydn' and check DN is normalized
+    inst.config.replace('nsslapd-return-original-entrydn', 'off')
+    users = UserAccounts(inst, SUFFIX, rdn="ou=humans").list()
     for user in users:
-        if user.dn.startswith("tUser"):
+        if user.rdn.startswith("tUser"):
             assert user.dn == NEW_USER_NORM_DN
             break

With these changes it fails:
>               assert user.dn == NEW_USER_NORM_DN
E               AssertionError: assert 'uid=tUser,ou...xample,DC=COM' == 'uid=tUser,ou...xample,dc=com'
E                 - uid=tUser,ou=humans,dc=example,dc=com
E                 ?                        ^       ^^ ^^^
E                 + uid=tUser,ou=humans,dc=Example,DC=COM
E                 ?                        ^       ^^ ^^^

dirsrvtests/tests/suites/basic/ds_entrydn_test.py:87: AssertionError

Looks like the server returns original entryDN even though nsslapd-return-original-entrydn was set to off.
But apparently the test can't even set it to off. When I tried to do that manually, I got an error:

ldap_modify: Server is unwilling to perform (53)
        additional info: nsslapd-subtree-rename-switch can't be modified while the server is running.

And why I edit dse.ldif manually, the server fails to start:
ERR - bdb_instance_start - nsslapd-subtree-rename-switch is off, while the instance userRoot is in the RDN format. Please change the value to on in dse.ldif.

So it doesn't look like I can change the server's behaviour by using this attribute, since it's always on.

Moving to ASSIGNED.

FYI nsslapd-subtree-rename-switch should not be used/changed.  IIRC That setting no longer works in 2.2 with the changes for lmdb.  I'll take a look at your other findings though...

If it can't be changed when lmdb is used, then I think we should modify additional info for the error 53 message to reflect that.

(In reply to Viktor Ashirov from comment #11)
> If it can't be changed when lmdb is used, then I think we should modify
> additional info for the error 53 message to reflect that.

Agreed.

Also, I think the issue here is that you need to restart the server after changing the config value, but the fix changes how the entry is loaded into the entry cache.  So if you change the setting and entry is already in the cache then it has no impact.

[mareynol@fedora servers]$ ldapsearch -xLLL -D cn=dm -w XXXXXX -H ldap://localhost -b dc=example,dc=com cn=mark dsentrydn
dn: cn=mark,ou=People,dc=EXample,dc=COM
dsentrydn: cn=mark,ou=People,dc=EXample,dc=COM

Set nsslapd-return-original-entrydn to "off" and restart server:

[mareynol@fedora servers]$ ldapsearch -xLLL -D cn=dm -w XXXXXX -H ldap://localhost -b dc=example,dc=com cn=mark dsentrydn
dn: cn=mark,ou=people,dc=example,dc=com
dsentrydn: cn=mark,ou=People,dc=EXample,dc=COM


So the CI test needs some adjusting, but to me the fix is working as expected...

(In reply to mreynolds from comment #10)
> FYI nsslapd-subtree-rename-switch should not be used/changed.  IIRC That
> setting no longer works in 2.2 with the changes for lmdb.  I'll take a look
> at your other findings though...

Ok I might have hastily misspoken.  This setting will not work in LMDB. However, it can be set in non-lmdb env, but you need to rebuild the database for it to work correctly.  I don't even see this setting in the docs anymore.  Anyway we are planning on removing this setting so I'm not sure how much more work should go into it.  Especially since it has not been used in years, it's not documented, and in fact it hasn't worked since 1.3.10 (or maybe 1.2.x)  I recently had to fix it for a customer as a workaround for their client needs in RHEL 8 which is why I know it hasn't worked in a very long time.

I just realized I was trying to change the wrong attribute...
If I restart the instance, the test with my changes passes.
Moving to VERIFIED.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (redhat-ds:12 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:3344

Added RN text, reviews by Mark https://issues.redhat.com/browse/IDMDS-3453
Moved to peer review.