The workaround proposed above of using "dbm:" prefix is not an option unfortunately... I succeeded to install the COTS java application by modifying its install scripts, but the application fails to launch and displays following error:

Exception in thread "main" java.security.ProviderException: NSS module not available: fips
	at jdk.crypto.cryptoki/sun.security.pkcs11.SunPKCS11.<init>(SunPKCS11.java:277)
	at jdk.crypto.cryptoki/sun.security.pkcs11.SunPKCS11$1.run(SunPKCS11.java:112)
	at jdk.crypto.cryptoki/sun.security.pkcs11.SunPKCS11$1.run(SunPKCS11.java:109)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at jdk.crypto.cryptoki/sun.security.pkcs11.SunPKCS11.configure(SunPKCS11.java:109)

I verified that "modutil -fips true -dbdir dbm:~/xxx" (executed from install script) did indeed succeed during installation.

I can't reproduce this issue:

[root@ci-vm-10-0-139-167 ~]# mkdir xxx
[root@ci-vm-10-0-139-167 ~]# modutil -create -dbdir xxx -force
[root@ci-vm-10-0-139-167 ~]# ls xxx
cert8.db  key3.db  secmod.db

[root@ci-vm-10-0-139-167 ~]# rpm -q nss
nss-3.53.1-3.el7_9.x86_64

same for databases created with certutil:
[root@ci-vm-10-0-139-167 ~]# mkdir test
[root@ci-vm-10-0-139-167 ~]# certutil -N -d test --empty-password
[root@ci-vm-10-0-139-167 ~]# ls test/
cert8.db  key3.db  secmod.db

nss-3.53.1.11 isn't in rhel7. It looks like you have a rhel8 build on your rhel-7 system, which is why the default is sql (it's always been sql on rhel8). Even when we rebase on rhel7, we change that default so nss-3.53.1-x_el7 has a different default than nss-3.53.1-x_el8, and our test cases verify this, (which is why hubert jumped in. If RHEL 7 had sql as the default, it means our test case was broken and giving us a false negative).

If you are seeing the issue on rhel7 with a rhel7 build, the next thing to check is your environment. Setting NSS_DEFAULT_DB_TYPE changes what the default is on your system. So if you set:
export NSS_DEFAULT_DB_TYPE=sql

Then your default database type will be sql on any version of RHEL (RHEL5, RHEL6, RHEL7, RHEL8, RHEL9). You can even set RHEL8 back to dbm with:
export NSS_DEFAULT_DB_TYPE=dbm
(though this won't work in RHEL 9 because dbm has been officially retired and isn't even compiled anymore).

Can you verify that you aren't tripping over something in comment 4. Thanks.

Apologies for delayed response AND for misidentification of the host platform:. It is rhel8.

I appreciate your very timely and informed responses.  I only ask that the man page(s) be updated to note that the default is "sql".

I have also informed the application vendor of the retiring of "dbm" format.

Thank you!

Confirmed that with nss-tools-3.53.1-17.el8_3.x86_64, the man page for certutil states
```
If no prefix is specified the default type is retrieved from NSS_DEFAULT_DB_TYPE. If NSS_DEFAULT_DB_TYPE is not set then dbm: is the default.
```
Similarly, the modutil man page states:
```
           modutil supports two types of databases: the legacy security databases (cert8.db, key3.db, and secmod.db) and new SQLite databases (cert9.db, key4.db, and pkcs11.txt). If the
           prefix sql: is not used, then the tool assumes that the given databases are in the old format.
```
So at least those two instances should be fixed, there may be other references to the dbm format being the default.

Agreed, dev_ack+, targetting next NSS rebase for the fix.

This needs an upstream fix for RHEL 8 (upstream has the same defaults as rhel 8) with:
       rhel7 reversion when we rebase.
       rhel9/fedora remove comments about dbm support.

Affected man pages:
certutil
modutil
pk12util
signver

upstream bug created because the upstream defaults match RHEL-8. RHEL-7 needs to revert the documentation patch when it updates (as the default in RHEL-7 is still dbm). RHEL-9/Fedora needs to remove all dbm references since dbm is disabled in RHEL-9/Fedora.

Need zstream+ to attach to the errrata.

Done