Certmonger works as expected when I use sql:/tmp/nssdb instead of /tmp/nssdb:

# rm -f /tmp/nssdb/cert8.db /tmp/nssdb/key3.db
# selfsign-getcert request -d sql:/tmp/nssdb -n Test-Cert -p /tmp/nssdb/pwdfile.txt -N CN=`hostname --fqdn` -D `hostname` -U id-kp-serverAuth -w
# ls /tmp/nssdb
cert9.db  key4.db  pkcs11.txt  pwd.txt
# getcert list
Request ID '20180111082637':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='sql:/tmp/nssdb',nickname='Test-Cert',token='NSS Certificate DB',pinfile='/tmp/nssdb/pwdfile.txt'
        certificate: type=NSSDB,location='sql:/tmp/nssdb',nickname='Test-Cert',token='NSS Certificate DB'
        CA: SelfSign
        issuer: CN=host
        subject: CN=host
        expires: 2019-01-11 09:26:37 CET
        dns: host
        eku: id-kp-serverAuth
        pre-save command: 
        post-save command: 
        track: yes
        auto-renew: yes
# certutil -d sql:/tmp/nssdb/ -L

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

Test-Cert                                                    ,,

It's going to be really tricky to get this right in FreeIPA and other users of certmonger. Certmonger considers /tmp/nssdb, dbm:/tmp/nssdb, and sql:/tmp/nssdb as three different entities, although /tmp/nssdb is either equivalent to dbm:/tmp/nssdb (Fedora 27) or to sql:/tmp/nssdb (Fedora 28).

On Fedora 27:

# certutil -d dbm:/tmp/nssdb/ -f /tmp/nssdb/pwdfile.txt -N
# selfsign-getcert request -d dbm:/tmp/nssdb -n Test-Cert-DBM -p /tmp/nssdb/pwdfile.txt -N CN=`hostname --fqdn` -D `hostname` -U id-kp-serverAuth -w
# getcert status -v -d dbm:/tmp/nssdb -n Test-Cert-DBM
State MONITORING, stuck: no.
# getcert status -v -d /tmp/nssdb -n Test-Cert-DBM
No request found that matched arguments.

Every caller has to know the NSSDB format *and* if certmonger tracks the cert with an explicit or implicit prefix. That's a big burden. It's likely that a caller get this wrong. For implicit prefix (no dbm: or sql:) qualifier, the auto-migration in Fedora 28 will change the NSSDB format, too.

I suggest that certmonger is changed to auto-detect the format for DIR without a prefix.

1) If DIR starts with either "dbm:" or "sql:", certmonger simply uses the given format.
2) If DIR has no prefix and DIR/cert9.db exists, it uses "sql:" prefix.
3) If DIR has no prefix and DIR/cert8.db exists, it uses "dbm:" prefix.
4) If DIR has no prefix and both DIR/cert9.db and DIR/cert8.db exist, it uses "sql:" prefix.
5) For any given path or tracking information, a DIR without a prefix is equivalent to a DIR with auto-detected prefix.
6) "getcert list" always includes the explicit prefix.

Example:
"getcert status -v -d /tmp/nssdb -n Test-Cert" automatically matches "dbm:/tmp/nssdb" if /tmp/nssdb/cert8.db exists. In case /tmp/nssdb/cert9.db exists, then the command is equivalent to "getcert status -v -d sql:/tmp/nssdb -n Test-Cert".

I'm not sure certmonger can or should try to pick the winner. It supports multiple distributions, each of which can have a different default.

NSS has a private function to do exactly this. I'll see if they can make it public. I filed https://bugzilla.redhat.com/show_bug.cgi?id=1533644

It occurs to me that we could also create an update task that would shut down certmonger, update the NSSDB path appropriately, and restart it.

And ensure that a db prefix is used on all subsequent tracking.

The issue might be with user-generated certs in the future. I'm sure they would work but the update task would catch up to the eventually and add the db prefix. Not sure if this would be surprising or not.

We also have to consider the situation where someone has an existing cert8/key3 database and want to do a stop/start tracking on it.

certmonger-0.79.5-3.fc28.x86_64 from scratch build https://koji.fedoraproject.org/koji/taskinfo?taskID=24231170 solves the issue of wrong default certdb.

# mkdir -p /tmp/nssdb
# echo "Secret.123" > /tmp/nssdb/pwdfile.txt
# certutil -d sql:/tmp/nssdb -f /tmp/nssdb/pwdfile.txt -N
# selfsign-getcert request -d /tmp/nssdb -n Test-Cert -p /tmp/nssdb/pwdfile.txt -N CN=`hostname --fqdn` -D `hostname` -U id-kp-serverAuth -w
New signing request "20180116171206" added.
# ls /tmp/nssdb/
cert9.db  key4.db  pkcs11.txt  pwdfile.txt
# certutil -d sql:/tmp/nssdb/ -L

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

Test-Cert                                                    u,u,u



The new certmonger build also migrated an existing DBM to SQL database correctly, even a password protected one:

# mkdir -p /tmp/nssdb
# echo "Secret.123" > /tmp/nssdb/pwdfile.txt
# certutil -d dbm:/tmp/nssdb -f /tmp/nssdb/pwdfile.txt -N
# certutil -d dbm:/tmp/nssdb -f /tmp/nssdb/pwdfile.txt -S -s "CN=selfsigned" -n selfsigned -x -t ',,'
# ls /tmp/nssdb/
cert8.db  key3.db  pwdfile.txt  secmod.db


# selfsign-getcert request -d /tmp/nssdb -n Test-Cert -p /tmp/nssdb/pwdfile.txt -N CN=`hostname --fqdn` -D `hostname` -U id-kp-serverAuth -w
New signing request "20180116172702" added.
# ls /tmp/nssdb/
cert8.db  cert9.db  key3.db  key4.db  pwdfile.txt  secmod.db
# certutil -d sql:/tmp/nssdb/ -f /tmp/nssdb/pwdfile.txt -K
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
< 0> rsa      9664bf2f72d6d8827d6ce2d1cde7d87828db7dd2   NSS Certificate DB:selfsigned
< 1> rsa      2b27364cb4628364f15555dc7d8be3e7544c00e1   NSS Certificate DB:Test-Cert

# certutil -d sql:/tmp/nssdb/ -f /tmp/nssdb/pwdfile.txt -L

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

selfsigned                                                   u,u,u
Test-Cert                                                    u,u,u