I tried and duplicated:

Here is the more detailed output near where it fails:

FINE: Importing user key for DOM-171-075.ABC.IDM.LAB.ENG.BRQ.REDHAT.COM IPA CA
FINE: Importing private key DOM-171-075.ABC.IDM.LAB.ENG.BRQ.REDHAT.COM IPA CA
FINE: Importing user certificate DOM-171-075.ABC.IDM.LAB.ENG.BRQ.REDHAT.COM IPA CA
FINE: Importing user key for ocspSigningCert cert-pki-ca
FINE: Importing private key ocspSigningCert cert-pki-ca
FINE: Importing user certificate ocspSigningCert cert-pki-ca
org.mozilla.jss.crypto.NoSuchItemOnTokenException: Expected user cert but no matching key?: (-8157) Certificate extension not found.
	at org.mozilla.jss.CryptoManager.importCertPackageNative(Native Method)
	at org.mozilla.jss.CryptoManager.importUserCACertPackage(CryptoManager.java:1151)
	at netscape.security.pkcs.PKCS12Util.storeCertIntoNSS(PKCS12Util.java:886)
	at netscape.security.pkcs.PKCS12Util.storeCertIntoNSS(PKCS12Util.java:902)
	at com.netscape.cmstools.pkcs12.PKCS12ImportCLI.execute(PKCS12ImportCLI.java:127)
	at com.netscape.cmstools.cli.CLI.execute(CLI.java:345)
	at com.netscape.cmstools.cli.CLI.execute(CLI.java:345)
	at com.netscape.cmstools.cli.MainCLI.execute(MainCLI.java:617)
	at com.netscape.cmstools.cli.MainCLI.main(MainCLI.java:653)

It looks like the first cert is successfully imported but it fails on the ocsp signing cert for some reason.

The pki command fails after it could not import the private key into the SQL NSSDB.

# certutil -d /tmp/nssdb_sql -f /tmp/reproducer/pfile -L

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

Certificate Authority - DOM-171-075.ABC.IDM.LAB.ENG.BRQ.REDHAT.COM CT,C,C
# certutil -d /tmp/nssdb_sql -f /tmp/reproducer/pfile -K
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
certutil: no keys found



# certutil -d dbm:/tmp/nssdb_dbm -f /tmp/reproducer/pfile -L

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

ocspSigningCert cert-pki-ca                                  u,u,u
subsystemCert cert-pki-ca                                    u,u,u
DOM-171-075.ABC.IDM.LAB.ENG.BRQ.REDHAT.COM IPA CA            CTu,Cu,Cu
auditSigningCert cert-pki-ca                                 u,u,u
# certutil -d dbm:/tmp/nssdb_dbm -f /tmp/reproducer/pfile -K
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
< 0> rsa      b02d30475a887b4b0467d552501f49b431717f17   auditSigningCert cert-pki-ca
< 1> rsa      f3bed2615ca1428de74a522e5fe9b5ce975e0d9b   subsystemCert cert-pki-ca
< 2> rsa      ef6e0d6c4d6c860f244b95ad6e6cfd753c6eafbc   ocspSigningCert cert-pki-ca
< 3> rsa      ceea71f117f373d374feb080541ded6327027e47   DOM-171-075.ABC.IDM.LAB.ENG.BRQ.REDHAT.COM IPA CA

Here is a more isolated command with DBM and SQL database:

# NSS_DEFAULT_DB_TYPE="sql" /usr/lib/jvm/jre-1.8.0-openjdk/bin/java -Djava.ext.dirs=/usr/share/pki/lib -Djava.util.logging.config.file=/usr/share/pki/etc/logging.properties com.netscape.cmstools.cli.MainCLI -d /tmp/nssdb_sql/ -C /tmp/reproducer/pfile --verbose pkcs12-import --pkcs12-file 
/tmp/reproducer/ca.p12 --pkcs12-password-file /tmp/reproducer/password.txt --debug "DOM-171-075.ABC.IDM.LAB.ENG.BRQ.REDHAT.COM IPA CA" "ocspSigningCert cert-pki-ca" "auditSigningCert cert-pki-ca" "subsystemCert cert-pki-ca"
Server URL: http://vm-247.abc.idm.lab.eng.brq.redhat.com:8080
Loading NSS password from /tmp/reproducer/pfile
NSS database: /tmp/nssdb_sql
Message format: null
Command: pkcs12-import --pkcs12-file /tmp/reproducer/ca.p12 --pkcs12-password-file /tmp/reproducer/password.txt --debug "DOM-171-075.ABC.IDM.LAB.ENG.BRQ.REDHAT.COM IPA CA" "ocspSigningCert cert-pki-ca" "auditSigningCert cert-pki-ca" "subsystemCert cert-pki-ca"
Initializing NSS
Logging into internal token
Using internal token
Module: pkcs12
Module: import
INFO: Loading PKCS #12 file
FINE: Load encrypted private keys:
FINE:  - Private key:
FINE:    ID: f38b50cbfd1a93ff9605effdd953657b1a6dd272
FINE:    Friendly name: DOM-171-075.ABC.IDM.LAB.ENG.BRQ.REDHAT.COM IPA CA
FINE:  - Private key:
FINE:    ID: bddc2052def38300517c45257421f81dac201c29
FINE:    Friendly name: ocspSigningCert cert-pki-ca
FINE:  - Private key:
FINE:    ID: 30a6055168c1ee5e6b7444e7f74624cc9c8f8a44
FINE:    Friendly name: auditSigningCert cert-pki-ca
FINE:  - Private key:
FINE:    ID: 0f23b797933fab64568fd0da931d3185f459fb8e
FINE:    Friendly name: subsystemCert cert-pki-ca
FINE: Loading certificates:
FINE:  - Certificate:
FINE:    Subject DN: CN=Certificate Authority,O=DOM-171-075.ABC.IDM.LAB.ENG.BRQ.REDHAT.COM
FINE:    ID: f38b50cbfd1a93ff9605effdd953657b1a6dd272
FINE:    Trust flags: CTu,Cu,Cu
FINE:    Friendly name: DOM-171-075.ABC.IDM.LAB.ENG.BRQ.REDHAT.COM IPA CA
FINE:  - Certificate:
FINE:    Subject DN: CN=OCSP Subsystem,O=DOM-171-075.ABC.IDM.LAB.ENG.BRQ.REDHAT.COM
FINE:    Trust flags: u,u,u
FINE:    ID: bddc2052def38300517c45257421f81dac201c29
FINE:    Friendly name: ocspSigningCert cert-pki-ca
FINE:  - Certificate:
FINE:    Subject DN: CN=CA Audit,O=DOM-171-075.ABC.IDM.LAB.ENG.BRQ.REDHAT.COM
FINE:    Trust flags: u,u,u
FINE:    ID: 30a6055168c1ee5e6b7444e7f74624cc9c8f8a44
FINE:    Friendly name: auditSigningCert cert-pki-ca
FINE:  - Certificate:
FINE:    Subject DN: CN=CA Subsystem,O=DOM-171-075.ABC.IDM.LAB.ENG.BRQ.REDHAT.COM
FINE:    Trust flags: u,u,u
FINE:    ID: 0f23b797933fab64568fd0da931d3185f459fb8e
FINE:    Friendly name: subsystemCert cert-pki-ca
FINE: Importing user key for DOM-171-075.ABC.IDM.LAB.ENG.BRQ.REDHAT.COM IPA CA
FINE: Importing private key DOM-171-075.ABC.IDM.LAB.ENG.BRQ.REDHAT.COM IPA CA
FINE: Importing user certificate DOM-171-075.ABC.IDM.LAB.ENG.BRQ.REDHAT.COM IPA CA
FINE: Importing user key for ocspSigningCert cert-pki-ca
FINE: Importing private key ocspSigningCert cert-pki-ca
FINE: Importing user certificate ocspSigningCert cert-pki-ca
org.mozilla.jss.crypto.NoSuchItemOnTokenException: Expected user cert but no matching key?: (-8157) Certificate extension not found.
        at org.mozilla.jss.CryptoManager.importCertPackageNative(Native Method)
        at org.mozilla.jss.CryptoManager.importUserCACertPackage(CryptoManager.java:1151)
        at netscape.security.pkcs.PKCS12Util.storeCertIntoNSS(PKCS12Util.java:891)
        at netscape.security.pkcs.PKCS12Util.storeCertIntoNSS(PKCS12Util.java:907)
        at com.netscape.cmstools.pkcs12.PKCS12ImportCLI.execute(PKCS12ImportCLI.java:127)
        at com.netscape.cmstools.cli.CLI.execute(CLI.java:345)
        at com.netscape.cmstools.cli.CLI.execute(CLI.java:345)
        at com.netscape.cmstools.cli.MainCLI.execute(MainCLI.java:705)
        at com.netscape.cmstools.cli.MainCLI.main(MainCLI.java:741)



# NSS_DEFAULT_DB_TYPE="dbm" /usr/lib/jvm/jre-1.8.0-openjdk/bin/java -Djava.ext.dirs=/usr/share/pki/lib -Djava.util.logging.config.file=/usr/share/pki/etc/logging.properties com.netscape.cmstools.cli.MainCLI -d /tmp/nssdb_dbm/ -C /tmp/reproducer/pfile --verbose pkcs12-import --pkcs12-file /tmp/reproducer/ca.p12 --pkcs12-password-file /tmp/reproducer/password.txt --debug "DOM-171-075.ABC.IDM.LAB.ENG.BRQ.REDHAT.COM IPA CA" "ocspSigningCert cert-pki-ca" "auditSigningCert cert-pki-ca" "subsystemCert cert-pki-ca"
Server URL: http://vm-247.abc.idm.lab.eng.brq.redhat.com:8080
Loading NSS password from /tmp/reproducer/pfile
NSS database: /tmp/nssdb_dbm
Message format: null
Command: pkcs12-import --pkcs12-file /tmp/reproducer/ca.p12 --pkcs12-password-file /tmp/reproducer/password.txt --debug "DOM-171-075.ABC.IDM.LAB.ENG.BRQ.REDHAT.COM IPA CA" "ocspSigningCert cert-pki-ca" "auditSigningCert cert-pki-ca" "subsystemCert cert-pki-ca"
Initializing NSS
Logging into internal token
Using internal token
Module: pkcs12
Module: import
INFO: Loading PKCS #12 file
FINE: Load encrypted private keys:
FINE:  - Private key:
FINE:    ID: f38b50cbfd1a93ff9605effdd953657b1a6dd272
FINE:    Friendly name: DOM-171-075.ABC.IDM.LAB.ENG.BRQ.REDHAT.COM IPA CA
FINE:  - Private key:
FINE:    ID: bddc2052def38300517c45257421f81dac201c29
FINE:    Friendly name: ocspSigningCert cert-pki-ca
FINE:  - Private key:
FINE:    ID: 30a6055168c1ee5e6b7444e7f74624cc9c8f8a44
FINE:    Friendly name: auditSigningCert cert-pki-ca
FINE:  - Private key:
FINE:    ID: 0f23b797933fab64568fd0da931d3185f459fb8e
FINE:    Friendly name: subsystemCert cert-pki-ca
FINE: Loading certificates:
FINE:  - Certificate:
FINE:    Subject DN: CN=Certificate Authority,O=DOM-171-075.ABC.IDM.LAB.ENG.BRQ.REDHAT.COM
FINE:    ID: f38b50cbfd1a93ff9605effdd953657b1a6dd272
FINE:    Trust flags: CTu,Cu,Cu
FINE:    Friendly name: DOM-171-075.ABC.IDM.LAB.ENG.BRQ.REDHAT.COM IPA CA
FINE:  - Certificate:
FINE:    Subject DN: CN=OCSP Subsystem,O=DOM-171-075.ABC.IDM.LAB.ENG.BRQ.REDHAT.COM
FINE:    Trust flags: u,u,u
FINE:    ID: bddc2052def38300517c45257421f81dac201c29
FINE:    Friendly name: ocspSigningCert cert-pki-ca
FINE:  - Certificate:
FINE:    Subject DN: CN=CA Audit,O=DOM-171-075.ABC.IDM.LAB.ENG.BRQ.REDHAT.COM
FINE:    Trust flags: u,u,u
FINE:    ID: 30a6055168c1ee5e6b7444e7f74624cc9c8f8a44
FINE:    Friendly name: auditSigningCert cert-pki-ca
FINE:  - Certificate:
FINE:    Subject DN: CN=CA Subsystem,O=DOM-171-075.ABC.IDM.LAB.ENG.BRQ.REDHAT.COM
FINE:    Trust flags: u,u,u
FINE:    ID: 0f23b797933fab64568fd0da931d3185f459fb8e
FINE:    Friendly name: subsystemCert cert-pki-ca
---------------
Import complete
---------------

The interesting steps for import happen in Java_org_mozilla_jss_CryptoManager_importCertPackageNative and Java_org_mozilla_jss_pkcs11_PK11Store_importEncryptedPrivateKeyInfo. I stepped through the code with gdb. The problem manifests on line https://github.com/dogtagpki/jss/blob/2acb5ca33714eee46965737684a22d3a1daa3c43/org/mozilla/jss/PK11Finder.c#L918 . With SQL database, the NSS function call PK11_KeyForDERCertExists(&theDerCert, &keyID, NULL) returns no slot "ocspSigningCert cert-pki-ca". The same function call returns a slot with DBM.

In order to trigger the code, you have to create a new NSS DBM and set "handle SIGSEGV nostop noprint pass" in gdb.

Adam, Stephen,

this is another release blocker for FreeIPA on Fedora 28. The bug prevents installation of a CA or KRA replica.

(In reply to Christian Heimes from comment #4)
> Adam, Stephen,
> 
> this is another release blocker for FreeIPA on Fedora 28. The bug prevents
> installation of a CA or KRA replica.

Just to clarify, because the phrasing was unclear: This prevents installation of a CA *replica* or a KRA *replica*, right?

If so, this is actually *not* a strict blocker, since the criteria only mandates (intentionally) that a single master FreeIPA server works.

We've *talked* about adding criteria around clustering, but at present it's not treated as blocking for Fedora. We'd likely consider it as a freeze exception, however.

I don't think this is worth blocking the Beta for, though. I'll propose it so that it goes through the proper approval or refusal process.

Actually, I don't think it's even meaningful to propose as a blocker at all. I can't think of any part of https://fedoraproject.org/wiki/Domain_controller_role_requirements that this would impact.

Christian, if you feel differently, please use https://qa.fedoraproject.org/blockerbugs/propose_bug to propose it for consideration prior to the Go/No-Go meeting taking place in approximately four hours from now.

(In reply to Stephen Gallagher from comment #5)
> (In reply to Christian Heimes from comment #4)
> > Adam, Stephen,
> > 
> > this is another release blocker for FreeIPA on Fedora 28. The bug prevents
> > installation of a CA or KRA replica.
> 
> Just to clarify, because the phrasing was unclear: This prevents
> installation of a CA *replica* or a KRA *replica*, right?

Correct, it prevents the creation of a CA replica or a KRA replica. Sorry for the ambiguity.

I wasn't aware that replica is not part of the requirements for Fedora. FreeIPA highly recommends to have at least one replica with CA. An installation with just one master is discouraged because it is a single point of failure.

(In reply to Christian Heimes from comment #7)
> (In reply to Stephen Gallagher from comment #5)
> > (In reply to Christian Heimes from comment #4)
> > > Adam, Stephen,
> > > 
> > > this is another release blocker for FreeIPA on Fedora 28. The bug prevents
> > > installation of a CA or KRA replica.
> > 
> > Just to clarify, because the phrasing was unclear: This prevents
> > installation of a CA *replica* or a KRA *replica*, right?
> 
> Correct, it prevents the creation of a CA replica or a KRA replica. Sorry
> for the ambiguity.
> 
> I wasn't aware that replica is not part of the requirements for Fedora.
> FreeIPA highly recommends to have at least one replica with CA. An
> installation with just one master is discouraged because it is a single
> point of failure.

We're going through an exercise to update our blocking criteria for Fedora 29; if you (and the FreeIPA team) feels that there should be criteria around replica functionality, I think that's perfectly reasonable. We should be certain to bound it sanely though; I don't want it to block Fedora release if for example having exactly 17 replicas causes it to collapse or something.

/me has been getting around to setting up a replica for the last 8 releases...

anyway, yeah, Stephen is right: it's not currently in the criteria but it seems reasonable to add it. That would mean I get to make the tests yet more complex though, happy fun times!

We should probably start a thread on server@ to thrash out the details of what we should put in the criteria. I'd say, the simplest replication scenario the upstream docs recommend, or so.

We *can* throw this on the prioritized list and propose it as an FE for Final right now, though, which should help us make sure we fix it for Final, and pull in the fix if we don't fix it by freeze time.

Discussed during the 2018-04-02 blocker review meeting: [1]

The decision to classify this bug as a RejectedFreezeException was made as this can be fixed with an update, and as most F28 Server users will update their systems before attempting replication, we have decided that no freeze exception status is warranted.

[1] https://meetbot.fedoraproject.org/fedora-blocker-review/2018-04-02/f28-blocker-review.2018-04-02-16.00.txt

I've started to look into this issue.

I see that your application causes NSS to import several objects into the database, but soon afterwards, all those data objects are deleted from the database.

My initial impression is, at the time the application runs into the failure,  all objects have been removed.

I see your application calls Java_org_mozilla_jss_pkcs11_PK11Store_deleteCert.

Could you find out why you are doing that?

Can you suggest a way for me to see the call stack in dogtag, which called that function? Below that call, the stack in gdb only shows ?????.

I'm debugging on my local Fedora 27 system.

Please look at this code:
https://github.com/dogtagpki/pki/blob/master/base/util/src/netscape/security/pkcs/PKCS12Util.java#L853

The documentation of store.deleteCert says "delete cert and key".

Is that really what you want here?

Do you potentially want to call deleteCertOnly() ?

Is it possible that with the DBM database, the key was somehow still cached in memory, and with the SQL db it gets purged immediately?

If my suggestion makes sense, could you please try that change?

Your call is inside a function called importKey.

The implement calls PK11_DeleteTokenCertAndKey.

I really think this is incorrect.

However, in order to change it, I believe you'll have to also change JSS, because the API deleteCertOnly isn't part of the CryptoStore interface.

You'll have to add a CryptoStore.deleteCertOnly method to the interface.

Class PK11Store already implements that function.

As a quick hack, I changed Java_org_mozilla_jss_pkcs11_PK11Store_deleteCert in JSS to call Java_org_mozilla_jss_pkcs11_PK11Store_deleteCertOnly.

With that change, running your command succeeds without error.

Unfortunately, it still isn't working perfectly.

Looking at certutil -L and certutil -K, all 4 certificates were imported, and all 4 private keys were imported.

However, they weren't associated to each either, all trust flags are listed as ",,", similar as seen in bug 1532188.

We must continue the debugging.

Thanks Kai,

I made the changes as you suggested. The import now works, and I observed the second problem that you mentioned (i.e. missing u flags).

$ certutil -L -d nssdb

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

DOM-171-075.ABC.IDM.LAB.ENG.BRQ.REDHAT.COM IPA CA            CT,C,C
ocspSigningCert cert-pki-ca                                  ,,   
auditSigningCert cert-pki-ca                                 ,,   
subsystemCert cert-pki-ca                                    ,,   

$ certutil -K -d nssdb -f pfile 
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
< 0> rsa      ceea71f117f373d374feb080541ded6327027e47   DOM-171-075.ABC.IDM.LAB.ENG.BRQ.REDHAT.COM IPA CA
< 1> rsa      ef6e0d6c4d6c860f244b95ad6e6cfd753c6eafbc   ocspSigningCert cert-pki-ca
< 2> rsa      b02d30475a887b4b0467d552501f49b431717f17   auditSigningCert cert-pki-ca
< 3> rsa      f3bed2615ca1428de74a522e5fe9b5ce975e0d9b   subsystemCert cert-pki-ca

I cannot provide an F27 build yet since apparently the COPR infrastructure is very slow/down right now.

OK, the PKI builds for F27, F28, and F29 are available here:
https://copr.fedorainfracloud.org/coprs/edewata/pki-10.6/build/739975/

It's now calling PK11Store.deleteCertOnly() instead of CryptoStore.deleteCert().

I have tested pki-ca-10.6.0-0.3.20180411205313.dcbf3837.fc28 on a new replica. CA replication is still failing, but later and with a different error message:


# ipa-ca-install
Directory Manager (existing master) password:

Run connection check to master
Connection check OK
Waiting for keys to appear on host: master.ipa.example, please wait until this has completed.
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/24]: creating certificate server db
  [2/24]: setting up initial replication
Starting replication, please wait until this has completed.
Update in progress, 4 seconds elapsed
Update succeeded

  [3/24]: creating installation admin user
  [4/24]: configuring certificate server instance
ipaserver.install.dogtaginstance: CRITICAL Failed to configure CA instance: CalledProcessError(Command ['/usr/sbin/pkispawn', '-s', 'CA', '-f', '/tmp/tmpgls3uefb'] returned non-zero exit status 1: 'certutil: could not find certificate named "caSigningCert cert-pki-ca": SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.\n')
ipaserver.install.dogtaginstance: CRITICAL See the installation logs and the following files/directories for more information:
ipaserver.install.dogtaginstance: CRITICAL   /var/log/pki/pki-tomcat
  [error] RuntimeError: CA configuration failed.

Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

CA configuration failed.   


Logs:
2018-04-12 10:55:50 pki.nssdb   : DEBUG    Command: pki -d /etc/pki/pki-tomcat/alias -C /etc/pki/pki-tomcat/pfile pkcs12-import --pkcs12-file /tmp/ca.p12 --pkcs12-password-file /tmp/tmpkt79_h7u/password.txt
2018-04-12 10:55:52 pki.nssdb   : DEBUG    Command: certutil -M -d /etc/pki/pki-tomcat/alias -f /etc/pki/pki-tomcat/pfile -n caSigningCert cert-pki-ca -t CTu,Cu,Cu
2018-04-12 10:55:52 pkispawn    : DEBUG    ....... Error Type: CalledProcessError
2018-04-12 10:55:52 pkispawn    : DEBUG    ....... Error Message: Command '['certutil', '-M', '-d', '/etc/pki/pki-tomcat/alias', '-f', '/etc/pki/pki-tomcat/pfile', '-n', 'caSigningCert cert-pki-ca', '-t', 'CTu,Cu,Cu']' returned non-zero exit status 255.
2018-04-12 10:55:52 pkispawn    : DEBUG    .......   File "/usr/lib/python3.6/site-packages/pki/server/pkispawn.py", line 534, in main
    scriptlet.spawn(deployer)
  File "/usr/lib/python3.6/site-packages/pki/server/deployment/scriptlets/security_databases.py", line 160, in spawn
    trust_attributes='CTu,Cu,Cu')
  File "/usr/lib/python3.6/site-packages/pki/nssdb.py", line 405, in modify_cert
    subprocess.check_call(cmd)
  File "/usr/lib64/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)


# certutil -d /etc/pki/pki-tomcat/alias/ -f /etc/pki/pki-tomcat/pfile -L

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

IPA.EXAMPLE IPA CA                                           CT,C,C
ocspSigningCert cert-pki-ca                                  ,,
auditSigningCert cert-pki-ca                                 ,,
subsystemCert cert-pki-ca                                    ,,

# certutil -d /etc/pki/pki-tomcat/alias/ -f /etc/pki/pki-tomcat/pfile -K
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
< 0> rsa      e91db0933d1c775cdb1492f72f2f79c0d7827341   IPA.EXAMPLE IPA CA
< 1> rsa      07cbbacc767f3637c6d50709d63d5a56b9126d2a   ocspSigningCert cert-pki-ca
< 2> rsa      d2cb9e6fd3a70577dfdf21bca0a109579e0dfdd8   auditSigningCert cert-pki-ca
< 3> rsa      d93bf162f76f6615ff9ace27140db25577aba8e4   subsystemCert cert-pki-ca

For comparision, this is Dogtag's NSSDB on the master (CA and KRA):

# certutil -d /etc/pki/pki-tomcat/alias/ -f /etc/pki/pki-tomcat/pfile -L

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

caSigningCert cert-pki-ca                                    CTu,Cu,Cu
ocspSigningCert cert-pki-ca                                  u,u,u
subsystemCert cert-pki-ca                                    u,u,u
auditSigningCert cert-pki-ca                                 u,u,Pu
Server-Cert cert-pki-ca                                      u,u,u
transportCert cert-pki-kra                                   u,u,u
storageCert cert-pki-kra                                     u,u,u
auditSigningCert cert-pki-kra                                u,u,Pu

# certutil -d /etc/pki/pki-tomcat/alias/ -f /etc/pki/pki-tomcat/pfile -K
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
< 0> rsa      1a475725cbc56f529022f068416d6bd8a5c8aac9   NSS Certificate DB:Server-Cert cert-pki-ca
< 1> rsa      e91db0933d1c775cdb1492f72f2f79c0d7827341   caSigningCert cert-pki-ca
< 2> rsa      07cbbacc767f3637c6d50709d63d5a56b9126d2a   ocspSigningCert cert-pki-ca
< 3> rsa      d93bf162f76f6615ff9ace27140db25577aba8e4   subsystemCert cert-pki-ca
< 4> rsa      d2cb9e6fd3a70577dfdf21bca0a109579e0dfdd8   auditSigningCert cert-pki-ca
< 5> rsa      ae3c66ea7dc8965dde85d6c76a2e5f86a2cd6023   transportCert cert-pki-kra
< 6> rsa      244461a094ed05b02a6c8f351878440a23cc7ed4   storageCert cert-pki-kra
< 7> rsa      7457caaaa634d5c5cf98c766cde86dc111819f35   auditSigningCert cert-pki-kra

"IPA.EXAMPLE IPA CA" on the new replica and "caSigningCert cert-pki-ca" on the master are one and the same certificate and key. The tool is importing the certificate with wrong nick name.

Endi, please take a look.

The tool imports the CA signing cert, but uses the friendly name from the PKCS#12 file instead of the correct name "caSigningCert cert-pki-ca" this breaks the replication installer. I don't know if the import is broken or the export is broken.

Christian,

How was the PKCS #12 file exported? The exporter need to store the nicknames as friendly names in the PKCS #12 file, that way the the certs & keys can be imported back into an NSS database with the same nicknames.

For example, export with this command:

 $ pki -d nssdb1 -c nsspw pkcs12-export --pkcs12-file test.p12 --pkcs12-password p12pw

Verify the friendly names in PKCS #12 file with this command:

 $ pki pkcs12-cert-find --pkcs12-file test.p12 --pkcs12-password p12pw

Import with this command:

 $ pki -d nssdb2 -c nsspw pkcs12-import --pkcs12-file test.p12 --pkcs12-password p12pw

The content of nssdb1 and nssdb2 should be identical.

The PKCS.12 file is created with PKCS12Export command from a temporary NSS DB, see https://github.com/freeipa/freeipa/blob/9762bd12797e1f01b48ec903c2dcb40649c1e484/ipaserver/install/custodiainstance.py#L180-L238

1) A temporary NSSDB is created
2) Replica downloads PKCS#12 files for each of 'caSigningCert cert-pki-ca', 'ocspSigningCert cert-pki-ca', 'auditSigningCert cert-pki-ca', and 'subsystemCert cert-pki-ca'
3) The PKCS#12 files are imported with pk12util into the temporary NSSDB.
4) All IPA CA certs are imported into the temporary NSS DB as well.
5) The temporary NSSDB is exported into one PKCS#12 file with PKCS12Export from Dogtag.

It's actually a problem with NSS. The SQL backend handles duplicate certificates differently than DBM. Step (4) imports the 'caSigningCert cert-pki-ca' public key a second time. With DBM format, the duplicate certificate was ignored. However with SQL format, step (4) changes the nickname of 'caSigningCert cert-pki-ca' into the nick name of the duplicate CA cert 'IPA.EXAMPLE IPA CA'. I'll file an issue.

See https://pagure.io/freeipa/issue/7498 for upstream bug report

I filed https://bugzilla.redhat.com/show_bug.cgi?id=1568271 for the NSS issue.

Endi, feel free to close this issue. With workaround https://github.com/freeipa/freeipa/pull/1819/files I was able to install a CA replica with ipa-replica-install.

Hello Christian, thanks for your work on this issue.

I'm glad to hear that you have a workaround, because I couldn't yet identify why NSS is behaving wrongly. I'm glad that you were able to duplicate the behavior using the NSS tools and have filed bug 1568271, this will simplify the further investitation.

I suggest that we assign this bug back to the pki component, because it was used to identify the bug that you had deleted the keys but expected them to remain. You said you have fixed that bug, so it seems right to mark this bug as fixed.

Hi, there seems to be several issues found in this bug:

* PKI issue about incorrect cert deletion in comment #14 which was fixed in comment #16
* NSS issue about incorrect trust flags in comment #15
* NSS issue about importing duplicate certs in comment #23 which is now tracked in bug #1568271

To my understanding the second issue is not yet addressed. Does it no longer need to be fixed due to other changes in PKI/IPA?

The trust flags from comment 15 aren't really trust flags. The trust flag "u" is an abuse of the trust flag system, and it indicates if an associated private key is present for the display certificate.

After the incorrect cert deletion, my understanding of the remaining issue was "private keys are not correctly associated to the certificate, and therefore the import is incomplete".

It hasn't been said explicitly, so let's ask Christian for confirmation:

Does your workaround fix the incomplete import, and correctly associate private key and certificate?

No, it doesn't fix all problems.

The workaround https://github.com/freeipa/freeipa/pull/1819/ fixes the case of "ipa-replica-install --setup-ca", that is installing a FreeIPA replica with Dogtag replica at the same time. I still cannot upgrade a simple replica to CA replica. Dogtag is hitting RHBZ#1568271, see https://pagure.io/freeipa/issue/7516.