We have two separate issues here.

Little introduction to both of them
  I built scripts to simplify CA installation and to avoid human errors.
  In this case there are two separate scripts, one for installing Root CAs and the second to install Sub CAs.
  Each of them follows the same pattern: 
    - prepares configs to run setup-ds.pl and pkispawn
    - runs setup-ds.pl and pkispawn
    - uses pki cli to do post installation updates


The fist issue: 
  In both scripts, the first pki command executed after pkispawn and setting up nss db for admin is
    pki -v -d /root/.dogtag/pki-tomcat/ca/alias/ -C /root/.dogtag/pki-tomcat/ca/password.conf -n 'PKI Administrator for hostname.com' ca-cert-find

  This command was always working in both root and sub CAs till rpm version 10.5.18-15.

  In case of rpm versions 10.5.18-16 and 10.5.18-17, execution of the following pki command:

    pki -v -d /root/.dogtag/pki-tomcat/ca/alias/ -C /root/.dogtag/pki-tomcat/ca/password.conf -n 'PKI Administrator for hostname.com' ca-cert-find

  fails ONLY during installation of SUB CAs with the following error:
    Server certificate: CN=vm-4.hostname.com,OU=pki-tomcat,O=hostname.com Security Domain
    ERROR: UNKNOWN_ISSUER encountered on 'CN=vm-4.hostname.com,OU=pki-tomcat,O=hostname.com Security Domain' results in a denied SSL server cert!
    FATAL: SSL alert sent: BAD_CERTIFICATE

  To summarize, this command fails only during sub-CA installation and only for rpm versions 10.5.18-16 and 10.5.18-17 and it fails for both RSA and ECC CAs.
  I added log from sub CA installation in verbose mode (sub-ca-inst-log.txt).


The second issue: 
  I discover the second issue, when I decided to install ECC CAs using older RPMs with version 10.5.18-15.
  In this case Root CA installed fine but Sub CA failed in pkispawn. I saw no such failure in using 10.5.18-17 RPMs.
  I will provide log from pkispawn failure.


Both issues create perfect trap making impossible to install any ECC sub CAs.

Created attachment 1842643 [details]
sub-ca-inst-log.txt - logs from sub CA installation

Main issue in short, after installing sub CA, the following command:

    pki -v -d /root/.dogtag/pki-tomcat/ca/alias/ -C /root/.dogtag/pki-tomcat/ca/password.conf -n 'PKI Administrator for hostname.com' ca-cert-find

is causing the following error:

Server certificate: CN=vm-4.hostname.com,OU=pki-tomcat,O=hostname.com Security Domain
ERROR: UNKNOWN_ISSUER encountered on 'CN=vm-4.hostname.com,OU=pki-tomcat,O=hostname.com Security Domain' results in a denied SSL server cert!
FATAL: SSL alert sent: BAD_CERTIFICATE
javax.ws.rs.ProcessingException: Unable to invoke request
	at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:287)
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:407)
	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invoke(ClientInvoker.java:102)
	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientProxy.invoke(ClientProxy.java:62)
	at com.sun.proxy.$Proxy32.login(Unknown Source)
	at com.netscape.certsrv.account.AccountClient.login(AccountClient.java:45)
	at com.netscape.certsrv.client.SubsystemClient.login(SubsystemClient.java:50)
	at com.netscape.cmstools.cli.SubsystemCLI.login(SubsystemCLI.java:46)
	at com.netscape.cmstools.cli.SubsystemCLI.execute(SubsystemCLI.java:64)
	at com.netscape.cmstools.cli.CLI.execute(CLI.java:345)
	at com.netscape.cmstools.cli.MainCLI.execute(MainCLI.java:630)
	at com.netscape.cmstools.cli.MainCLI.main(MainCLI.java:666)
Caused by: java.io.IOException: SocketException cannot write on socket
	at org.mozilla.jss.ssl.SSLSocket.write(SSLSocket.java:1503)
	at org.mozilla.jss.ssl.SSLOutputStream.write(SSLOutputStream.java:24)
	at org.apache.http.impl.io.AbstractSessionOutputBuffer.flushBuffer(AbstractSessionOutputBuffer.java:147)
	at org.apache.http.impl.io.AbstractSessionOutputBuffer.flush(AbstractSessionOutputBuffer.java:154)
	at org.apache.http.impl.AbstractHttpClientConnection.doFlush(AbstractHttpClientConnection.java:278)
	at org.apache.http.impl.AbstractHttpClientConnection.flush(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.flush(ManagedClientConnectionImpl.java:175)
	at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:260)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
	at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:283)
	... 11 more

Created attachment 1843258 [details]
logs from pkispawn failure during ecc sub-ca installation

I've attached logs from pkispawn failure during ecc sub-ca installation in sub-ca-ecc-pkispawn-failure-log.txt.

Here is my investigation result.

The issue has to do with the "pki_import_admin_cert" parameter mentioned here:
https://bugzilla.redhat.com/show_bug.cgi?id=1984431 (ok, this bug was for 10.x instead of 9.7.x, but the confusion is the same)

Its existence has caused confusions as well as errors we are to sort out later (very likely not in 7.9.x).  At present time, the recommended procedure is to always leave out the "pki_import_admin_cert" parameter in the pkispawn config file. Leaving it out would lead to a successful installation. After pkispawn is run successfully, you then create your own admin nssdb by importing the cert chain and the admin cert p12 .

I have tested creating ECC subCA instances in the following scenarios with success:
* ECC subCA with hsm using two-step pkispawn installation
* ECC subCA without HSM using two-step pkispawn installation
* ECC subCA without HSM using single step pkispawn installation
* RSA subCA with HSM using two-step pkispawn installation


In the future, in helping the responsible engineer a speedier understanding of the reported issue, in addition to logs, please also provide info on how you run pkispawn (complete with your pkispawn config files).  Thanks.

Endi, please review and ack the above recommendation?

I should add to my investigation result that the observed issue was caused by the fact that the bootstrap admins' nssdb is no longer created automatically:
e.g. /root/.dogtag/pki-tomcat/ca/alias/
And the recommendation is to create it manually after pkispawn is done.

One more thing. The ECC installation issue was resolved from a different bug and should be available in the next batch update.

As discussed on IRC, I'm not sure why the BAD_CERTIFICATE started to appear,
but in general, instead of downloading the admin cert from CA, I'd recommend
to import the admin cert from the PKCS #12 file provided by pkispawn:

https://github.com/dogtagpki/pki/wiki/Importing-Admin-Certificate-into-PKI-CLI
https://github.com/dogtagpki/pki/wiki/Importing-Admin-Certificate-into-Firefox

Just FYI, the admin NSS database at ~/.dogtag/pki-tomcat/ca/alias should be
automatically created if pki_client_database_purge=False, but this is not the
default and it's mainly used internally by pkispawn, so I don't recommend
using it. It's better to import the PKCS #12 file as described above.

Also, about the pki_import_admin_cert param, it's also mainly used internally
by pkispawn so I don't recommend using it either.

Thanks Endi.  So I just tried out "pki_client_database_purge=False" and now the nss dbs are created!
I was just doing a simply case of single step pkispawn without hsm.

Hi Christina,
Hi Endi,

Thank you for your detailed update. 

Here is what I see:

  I never defined "pki_import_admin_cert" in my pkispawn config file.
  If this counts for "leaving out" then SUB-CA installation still fails in RPM versions:
    10.5.18-14, 10.5.18-15, and 10.5.18-18.
  I've already attached logs from pkispawn failure (in sub-ca-ecc-pkispawn-failure-log.txt). 

  I see NSSDB still being created on RHEL/CentOS 7.9 and created NSSDB still includes admin key and certificate.
  Above NSSDB never contained CA chain, which I download from CAs and subCAs itself. 
  This worked for PKI CLI 
    - RHEL/CentOS 7.7 - RPMs 10.5.16-6 
    - RHEL/CentOS 7.8 - RPMs 10.5.17-6, 10.5.17-7
    - RHEL/CentOS 7.9 - RPMs 10.5.18-7, 10.5.18-12, 10.5.18-14, 10.5.18-15
  It stoped working in
    - RHEL/CentOS 7.9 - RPMs 10.5.18-16, 10.5.18-17, 10.5.18-18

Hi Andrew, so did you try adding "pki_client_database_purge=False" ?  It did the trick for me.  And again, for ECC subCA, please wait for next batch release.

Hi Christina,
I don't mind trying "pki_client_database_purge=False", but which issue that should solve?
At this point, pki cli failure is the primary show stopper.

Hi Andrew, as Endi pointed out in comment #12, "the admin NSS database at ~/.dogtag/pki-tomcat/ca/alias should be
automatically created if pki_client_database_purge=False", which should then resolve the issue you indicated for
    pki -v -d /root/.dogtag/pki-tomcat/ca/alias/ -C /root/.dogtag/pki-tomcat/ca/password.conf -n 'PKI Administrator for hostname.com' ca-cert-find

Hi Christina, sorry if I was not clear enough.

In pki-cli case, nothing changed in the content of nssdb located at at ~/.dogtag/pki-tomcat/ca/alias (so there is no need to fix nssdb content).

I belive that there was change between versions 10.5.18-15 and 10.5.18-16,
that is causing failure during establishing of TLS connection by pki-cli,
which is done on the same proper set of certificates and the key.

Hi Andrew.  Again, it takes too much time guessing from my side.  If you could provide the exact pkispawn file you used so I could reproduce your issue, or let's just wait till the next BU release is available and we can see if it fixes the problem you encountered and we can go from there.  Thanks.

Hi Christina,
Hi Hack,

I've sent zipped tar (ca-scripts.tar-zipped) via email, that should help you see the issue.

Here is what to do:
 * You need 2 VMs with RHEL/CentOS 7.9.

 * Copy  zipped tar file to each of them and unpack it by running the following commands:
       unzip ca-scripts.tar-zipped
       tar xfv ca-scripts.tar
   (By the way, Matt knows the magic word and you know it too ;) .)

 * On the first VM install rootCA by running the following command (as root):
       cd ca-scripts
       ./create-ca.sh - - ' Test 1' rsa 2048

 * On the second VM install subCA by running the following command (as root):
       cd ca-scripts
       ./create-sub-ca.sh hostname-of-your-root-ca - - root-ca-admin-password ' Test 1' rsa 2048
   You can find the rootCA admin password on the first VM in /root/ca-scripts/.tmp/ca.cfg.

 * To repeat the test on subCA run the following commands (as root):
       ./remove-ca.s
       ./create-sub-ca.sh hostname-of-your-root-ca - - root-ca-admin-password ' Test 1' rsa 2048

Search in create-sub-ca.sh for string "Issue:" to find location in the code.

Hi Andrew. Thanks for provide the info. I only needed the pkispawn cfg files.  But I was able to gather info from your scripts to create both ca and subCA pkispawn config files .  That helps.

I was able to install ca and subCA on two separate VMS, per your latest info from comment #20

The pki cli command with ca-cert-find succeeded on the root CA;  The same command failed on the subCA with the same issue you reported ("UNKNOWN_ISSUER").
In other words, I am able to reproduce the issue!

Do you mind if I change the subject of this bug to something closer to the issue at hand?  You can file another bug for ECC if the other issue persists after you try out the next BU. How's that?

The subject I intend to change it to is something like "Failed to connect to subCA with pki cli client nssdb created from pkispawn"
Let me know.

Hi Christina,

This is a great news that you could reproduce pki-cli case. Thank you.

I don't mind if you would update subject of this bug, but could please create another bug for ecc case.
In some strange way both case are a bit similar since both are failing during subCA creation.

I saw ecc case in rpm version 10.5.18-14, 10.5.18-15, and 10.5.18-18. pkispawn failure logs are already attached.

Here are steps to reproduce ecc case:
 * You need 2 VMs with RHEL/CentOS 7.9.

 * Copy  zipped tar file to each of them and unpack it by running the following commands:
       unzip ca-scripts.tar-zipped
       tar xfv ca-scripts.tar

 * On the first VM install rootCA by running the following command (as root):
       cd ca-scripts
       ./create-ca.sh - - ' ECC Test' ecc nistp256

 * On the second VM install subCA by running the following command (as root):
       cd ca-scripts
       ./create-sub-ca.sh hostname-of-your-root-ca - - root-ca-admin-password ' ECC Test' ecc nistp256
   You can find the rootCA admin password on the first VM in /root/ca-scripts/.tmp/ca.cfg.

 * To repeat the test on subCA run the following commands (as root):
       ./remove-ca.s
       ./create-sub-ca.sh hostname-of-your-root-ca - - root-ca-admin-password ' ECC Test' ecc nistp256

*** Bug 2022071 has been marked as a duplicate of this bug. ***

*** Bug 2033108 has been marked as a duplicate of this bug. ***

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (pki-core bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:0070