Created attachment 1378442 [details]
installation logs

Created attachment 1378443 [details]
tar.gz of NSSDB

Created attachment 1378484 [details]
ltrace output

I think it's a bug in certmonger. Certmonger uses the wrong key nickname. When I create a self-signed cert with certutil, the nick name is prefixed with the database name:

# certutil -d /etc/dirsrv/slapd-IPA-EXAMPLE -f /etc/dirsrv/slapd-IPA-EXAMPLE/pwdfile.txt -S -x -k rsa -g 2048 -Z SHA256 -t CT,C,C -s 'CN=Test-Cert' -n Test-Cert -m 365
# certutil -d /etc/dirsrv/slapd-IPA-EXAMPLE -f /etc/dirsrv/slapd-IPA-EXAMPLE/pwdfile.txt -L

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

IPA.EXAMPLE IPA CA                                           CT,C,C
Server-Cert                                                  ,,   
Test-Cert                                                    CTu,Cu,Cu
# certutil -d /etc/dirsrv/slapd-IPA-EXAMPLE -f /etc/dirsrv/slapd-IPA-EXAMPLE/pwdfile.txt -K
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
< 0> rsa      0320a8df63765c9e4d135bb2fbd4b32d6e3ec57f   Server-Cert
< 1> rsa      4c51580113b5296bb75d5ce780b59eb3118ca640   NSS Certificate DB:Test-Cert

(In reply to Christian Heimes from comment #0)
> 
> certutil shows that 389-DS' NSSDB contains a public cert and private key for
> nickname 'Server-Cert'. But 389-DS is not able to find the private key (no
> matching key). The output of certutil is also missing the dynamic 'u' flags
> and pk12util is not able to extract the private key. I suspect that the
> issue may be related to switch of NSS to SQL format.

If the certutil output is missing 'u', then how do you know that the DB really contains the private key?

I cannot run "certutil -K" on the attached NSSDB, because you didn't provide the password file.

It would be helpful to know what NSS commands are executed that produce the database.

(In reply to Christian Heimes from comment #4)
> Certmonger uses the wrong key nickname.

Does that mean you no longer require analysis from the NSS team?

The tar.gz ball contains the password file:

$ tar tfz slapd-nssdb.tar.gz 
cert9.db
key4.db
pkcs11.txt
pwdfile.txt

certutil -K shows a private key with nickname "Server-Cert" -- the same nickname as the public cert. The NSSDB, key and cert have been created by certmonger, https://pagure.io/certmonger/blob/master/f/src/keygen-n.c

When I create a key and self-signed cert with certutil directly, then the private key nick is prefixed with "NSS Certificate DB:"

We still need assistance from the NSS team. The certmonger code for NSS key generation hasn't changed in a while. Fedora 27 and earlier are working fine. I'm suspecting a regression.

(In reply to Christian Heimes from comment #8)
> The tar.gz ball contains the password file:

thanks

> The NSSDB, key and cert have been created by certmonger

Could you provide the full command, including parameters, that was used?

No, because certmonger doesn't use certutil. It uses NSS' C API directly, https://pagure.io/certmonger/blob/master/f/src/keygen-n.c

Are you saying that certmonger is executed without any parameters?

If certmonger is started with parameters, please provide what is used, as that will simplify debugging.

Please don't exepct us to debug through ipa and certmonger. Please extract the commands that are used, or a minimal environment that can be used to reproduce.

If you're saying that the latest NSS produces an unexpected cert database, then please also provide an equivalent good cert database from a previous version, so we can compare.

To summarize:

You claim that you are executing code through certmonger, that produces an apparently corrupt NSS SQL database.

You claim it's corrupt, because there is a certificate and a private key, but NSS is unable to match them, and believes they are unrelated.

Using the attached NSS SQL db, I confirm that NSS doesn't list them as related. certutil -L reports the certificate as having no private key. pk12util -o complains that no private key could be found.

I see these listing and exporting failures with old and current NSS.

This means, the bug is probably related to the creation of the data inside the NSS database.

In order to analyze the issue, we need to know what commands are used that produce the attached database. It doesn't matter what kind of commands are used. If the database is produced using certmonger, then please provide a minimal environment that can be used to run certmonger and that produces the database. Please don't ask us to figure out from the big certmonger source code, or from the big IPA code base, what commands are probably being executed. You should figure the code path that your software uses to produce the given database, and provide us an easy way to reproduce that. We need that, because we'll likely require to setup a minimal environment for reproducing the failure with debugging enabled. Optimized binaries often don't allow to debug correctly.

I'm OK to debug through certmonger to trace a specific execution flow that produces a corrupt database, but I don't want to figure myself out how certmonger is started to trigger that execution flow.

I discussed the issue with Alexander Bokokoy and Rob Crittenden. Rob is having issues with certmonger's test suite on rawhide, too. Alexander suggested to use certmonger's selfsign-getcert tool instead of IPA to reproduce the issue. selfsign-getcert uses certmonger to request a self-signed certificate. I'll attach more debug infos from certmonger in a minute.

Here is a simple reproducer:

# mkdir /tmp/nssdb
# echo 'Secret.123' > /tmp/nssdb/pwdfile.txt
# certutil -d /tmp/nssdb/ -f /tmp/nssdb/pwdfile.txt -N
# ls /tmp/nssdb/
cert9.db  key4.db  pkcs11.txt  pwdfile.txt
# selfsign-getcert request -d /tmp/nssdb -n Test-Cert -p /tmp/nssdb/pwdfile.txt -N CN=`hostname --fqdn` -D `hostname` -U id-kp-serverAuth -w

Diagnostic information from certmonger and certutil:

# getcert list
...
 Request ID '20180108131048':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/tmp/nssdb',nickname='Test-Cert',token='NSS Certificate DB',pinfile='/tmp/nssdb/pwdfile.txt'
        certificate: type=NSSDB,location='/tmp/nssdb',nickname='Test-Cert',token='NSS Certificate DB'
        CA: SelfSign
        issuer: CN=master.ipa.example
        subject: CN=master.ipa.example
        expires: 2019-01-08 08:10:48 EST
        dns: master.ipa.example
        eku: id-kp-serverAuth
        pre-save command: 
        post-save command: 
        track: yes
        auto-renew: yes
# certutil -d /tmp/nssdb -f  /tmp/nssdb/pwdfile.txt -L

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

Test-Cert                                                    ,,   
# certutil -d /tmp/nssdb -f  /tmp/nssdb/pwdfile.txt -K
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
< 0> rsa      7dbf7b28e417b6b0359cb35109df22dba1e06f08   Test-Cert

Created attachment 1378520 [details]
certmonger debug output

output of certmonger -S -d6 while issuing Test-Cert3

Thanks for the reduced reproducer. I'm able to reproduce.
I don't have a solution yet, but want to give you an update.

The regression is related to NSS now using the SQL database format by default.

When using the same software version, but explicitly requesting to use the DBM database format, it's working as expected.

But going back to DBM isn't an option, we must fix the use of the SQL database.

As Christian noticed, the listing produced by certutil -K is unusual, as the SQL database misses the token name prefix for the key nickname.

With the code that certmonger uses to create the key and the certificate, when using the SQL format, NSS is unable to associate the private key to the certificate.

I have already prepared an environment that allows me to debug, by building certmonger RPMs and NSS RPMs that have debugging enabled.

I see certmonger uses the PK11_SetPrivateKeyNickname API, which was my first suspicion of requiring a fix. Interestingly, none of the NSS tools or tests call this API...

As an experiment, I added a call to PK11_SetPrivateKeyNickname to certutil, just after having generated the key. This influences the token prefix shown by certutil -K. However, I get the same behavior with both DBM and SQL database, the token prefix is absent for both database types, and nevertheless, certutil -L shows the u flag, indicating that it's still able to associate key and cert correctly.

We'll have to debug more thorougly to find out what's going wrong in your scenario.

(Side note: I built nspr, nss-softokn and nss with optimization disabled. If I do that for nss-util, we run into a deadlock, I haven't figured out why. As a workaround, I currently use the regular, optimized RPM for nss-util.)

(In reply to Kai Engert (:kaie) from comment #17)
> I have already prepared an environment that allows me to debug, by building
> certmonger RPMs and NSS RPMs that have debugging enabled.

... that have optimization disabled.

I have a workaround for you, while we investigate the real cause.

I'll attach a patch that made certmonger work for me in my test environment.

certmonger uses CERT_ImportCerts for importing, because it only has the encoding of the certificate. Then certmonger calls PK11_FindKeyByAnyCert, but doesn't find the cert.

I cannot explain why the cert isn't found. We need to analyze that. I'll file an upstream bug and try to create a test that doesn't need the certmonger code.

I see that certutil uses PK11_ImportCerts, which requires to already have a CERTCertificate.

The workaround patch changes certmonger to call PK11_ImportCerts after having successfully called CERT_ImportCerts.

Please let us know if this workaround works for you.

Created attachment 1379253 [details]
workaround patch v1

With this patch and some other necessary changes I was able to get all the certmonger unit tests passing in rawhide.

Kai, awesome job. Thanks for diving into the bug!

Rob, Nalin, could you please provide a fixed build of certmonger for rawhide ASAP? I'd like to verify IPA installation before the next release of IPA.

The underlying bug in NSS has been reported at https://bugzilla.mozilla.org/show_bug.cgi?id=1429591

Workaround submitted to certmonger upstream in https://pagure.io/certmonger/pull-request/92

This bug appears to have been reported against 'rawhide' during the Fedora 28 development cycle.
Changing version to '28'.

The bug also affects JSS, Dogtag and CA replica installation of FreeIPA on Fedora 28. The installation fails when importing the CA PKCS12 file into Dogtag's NSS database.

Failing command:

$ pki -d /etc/pki/pki-tomcat/alias -C /etc/pki/pki-tomcat/pfile pkcs12-import --pkcs12-file /tmp/ca.p12 --pkcs12-password-file /tmp/password.txt 
<cheimes> NoSuchItemOnTokenException: Expected user cert but no matching key?: (-8157) Certificate extension not found.


The same command works fine after I modified Dogtag to use DBM format and try to import the file into a DBM NSS DB:

$ mkdir /tmp/dbm
$ certutil -d dbm:/tmp/dbm -f /etc/pki/pki-tomcat/pfile -N
$ sed -i s/sql/dbm/g /usr/share/pki/etc/pki.conf
$ pki -v -d /tmp/dbm/ -C /etc/pki/pki-tomcat/pfile pkcs12-import --pkcs12-file /tmp/ca.p12 --pkcs12-password-file /tmp/password.txt --debug

pki-ca-10.6.0-0.2.20180327184846.353de17f.fc28.noarch
jss-4.4.2-10.fc28.x86_64
nss-3.36.0-1.0.fc28.x86_64

Here is a simple reproducer.

# dnf install pki-ca nss-tools

# certutil -d /tmp/nssdb_sql/ -N -f /tmp/reproducer/pfile 
# pki -v -d /tmp/nssdb_sql/ -C /tmp/reproducer/pfile pkcs12-import --pkcs12-file /tmp/reproducer/ca.p12 --pkcs12-password-file /tmp/reproducer/password.txt --debug
org.mozilla.jss.crypto.NoSuchItemOnTokenException: Expected user cert but no matching key?: (-8157) Certificate extension not found.

# sed -i s/sql/dbm/g /usr/share/pki/etc/pki.conf
# mkdir /tmp/nssdb_dbm
# certutil -d dbm:/tmp/nssdb_dbm -N -f /tmp/reproducer/pfile
# pki -v -d /tmp/nssdb_dbm/ -C /tmp/reproducer/pfile pkcs12-import --pkcs12-file /tmp/reproducer/ca.p12 --pkcs12-password-file /tmp/reproducer/password.txt --debug

Created attachment 1414307 [details]
ca.p12, password.txt and pfile for reproducer

Extract the files in /tmp

Christian, are you saying the upstream fix doesn't work?

Let me ask differently.

Please clarify if the original issue was fixed for certmonger, by using NSS 3.36 which includes the fix from the attached upstream bug.

If this issue is fixed for certmonger, but other applications have a similar version with the fixed NSS version, I'd conclude those other applications are using a different scenario or code path, and you are reporting a new bug.

It's unclear if the fix works for certmonger. certmonger-0.79.5-6.fc28.x86_64 still contains the workaround https://pagure.io/certmonger/blob/master/f/src/certsave-n.c#_477 .

Christian re-tested certmonger without the workaround and the tests pass so this particular bug seems to be fixed in nss.

I opened a new nss bug to track the newly reported issue at https://bugzilla.redhat.com/show_bug.cgi?id=1561730

Marking this one as closed.