Bug 1301546 - pkispawn ignores 3rd party CA certs in pki_clone_pkcs12_path
pkispawn ignores 3rd party CA certs in pki_clone_pkcs12_path
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: pki-core (Show other bugs)
All Linux
high Severity high
: rc
: 7.3
Assigned To: Endi Sukma Dewata
Asha Akkiangady
: ZStream
Depends On:
Blocks: 1301687 1318302 1323836
  Show dependency treegraph
Reported: 2016-01-25 06:19 EST by Jan Cholasta
Modified: 2016-11-04 01:22 EDT (History)
16 users (show)

See Also:
Fixed In Version: pki-core-10.3.1-1.el7, pki-core-10.3.3-5.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1318302 (view as bug list)
Last Closed: 2016-11-04 01:22:02 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
bz1301546.tar.gz (377.93 KB, application/x-gzip)
2016-08-29 09:05 EDT, Abhijeet Kasurde
no flags Details

  None (edit)
Description Jan Cholasta 2016-01-25 06:19:38 EST
Description of problem:
If pki_clone_uri in pkispawn config file points to a host which uses a server cert signed by a 3rd party CA, pkispawn will fail even if the 3rd party CA cert is present in the PKCS#12 file specified by pki_clone_pkcs12_path.

This causes a failure in IPA replica install: https://fedorahosted.org/freeipa/ticket/5602

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Run pkispawn with the config described above

Actual results:
pkispawn fails

Expected results:
pkispawn succeeds

Additional info:
Comment 2 Matthew Harmsen 2016-01-25 17:13:55 EST
Upstream ticket:
Comment 3 Jan Cholasta 2016-02-01 12:11:27 EST
Steps to reproduce:

1. Install CA-less IPA server with:

root@server# ipa-server-install --dirsrv-cert-file=/path/to/server-cert.p12 --http-cert-file=/path/to/server-cert.p12

(You have to provide private key and a server certificate issued by an arbitrary 3rd party CA in server-cert.p12.)

2. Install the CA on the IPA server with:

root@server# ipa-ca-install

3. Prepare a replica with:

root@server# ipa-replica-prepare replica.example.com

(Copy the resulting replica file to the replica host.)

4. Install IPA replica with:

root@replica# ipa-replica-install replica-info-replica.example.com.gpg

5. Install CA clone on the replica with:

root@replica# ipa-ca-install replica-info-replica.example.com.gpg
Comment 4 Jan Cholasta 2016-02-01 12:20:11 EST
You can use the makepki.sh script from <https://www.redhat.com/archives/freeipa-devel/2015-November/msg00487.html> to easily generate server-cert.p12.
Comment 6 Endi Sukma Dewata 2016-02-01 20:24:48 EST
The makepki.sh does not generate a server-cert.p12 file. I tried using the nssdb/ca1/server.p12, the ipa-replica-install failed with this output:

  [17/38]: configuring ssl for ds instance
  [error] RuntimeError: Could not find a CA cert in /tmp/tmpIjLRdIipa/realm_info/dscert.p12
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

ipa.ipapython.install.cli.install_tool(Replica): ERROR    Could not find a CA cert in /tmp/tmpIjLRdIipa/realm_info/dscert.p12

This was tested on F23. Could you provide the proper steps to generate server-cert.p12?
Comment 7 Jan Cholasta 2016-02-02 01:26:44 EST
The file name does not matter, I assumed you would be able to figure that out.

I forgot one step in comment 3, between steps 2 and 3:

2.5. Update local filesystem with the new CA certificate on the IPA server:

root@server# ipa-certupdate

(Note that you need IPA build with the fix for https://fedorahosted.org/freeipa/ticket/5595 in order for ipa-certupdate to work properly in this case.)
Comment 14 Endi Sukma Dewata 2016-03-17 05:02:53 EDT
Does IPA keep track the nicknames of the 3rd-party CA certificates? Where are the nicknames stored in IPA?
Comment 15 Jan Cholasta 2016-03-17 05:31:22 EDT
Yes, /etc/ipa/nssdb.
Comment 16 Endi Sukma Dewata 2016-03-18 18:43:41 EDT
Fixed in master:

* aa613fa272defcc8eebd4b9ef2556e61683b4e97
* 709457876a6d5e4aea281a35350667492bc34df8
* 54849505729d3f6345bc7b530e5a40c14ff36116
* 6947854a3ab6ee4f296a5f97850f5521572683a1
* 0d44556fa78203121a24224d4733b89c36ef9cc9
* a96ecbae1bfa27223bbebc7a67f695b643c4aebe
* 67a0c95b8622b18c9803b2bfe0f708be8747f896
* 67402ac16d2635ab3464568ca007cf81c4db73e6
* b74bf9b82102715e08fa3fd3bd5ce9462312aded
* b48889a2ef41fd45ca69c3926c36ef075777447c
* 1d58b883ff9d0056d89d74d30f1375ab12d01f03
* 935633c5ea9f2b5c4321d924af166367008ac4b3
* 0dadf421c327bc32d220405208031a9f7e1bb097
* 9c6b53ac8f6eee2eb8ed8f47a4b26be828626841
* 20a70830961f532e9483baefb64cc92af7cda8b2
* 1b15c725b6e9c5d9057b66e0a2806a7813a8d61b
* 04055a9bc40486950a3288acf610522e767c1e27
* c14e8c52ae7a2c15433fe9568c393c1d0e7a1301

Note: there is no change required in IPA.
Comment 18 Matthew Harmsen 2016-03-31 11:45:19 EDT
PKI TRAC Ticket #2247 fixes checked into master:
    * 58b78bd1602e3efeb33a73f8d07a6edaaee104ba
    * 061bec70264c2c7a601ffe80846ef1fa5497c15c 

Note that there is another error similar to ticket #2226 that blocks KRA installation on IPA replica.
Comment 19 Matthew Harmsen 2016-03-31 12:07:15 EDT
Steps to verify changes for PKI TRAC Ticket #2247:

    1.) setup a FreeIPA master w/ KRA

    2.) install a replica with CA

    3.) install KRA clone on the replica

KRA clone should be installed and functional.
Comment 20 Matthew Harmsen 2016-04-21 14:38:52 EDT
*** Bug 1328522 has been marked as a duplicate of this bug. ***
Comment 22 Martin Bašti 2016-07-27 08:17:34 EDT
*** Bug 1358752 has been marked as a duplicate of this bug. ***
Comment 23 Martin Bašti 2016-07-27 08:19:34 EDT
Hello, we (IPA team) are afraid that this bug is not fixed completely, please see bz1358752
Comment 24 Endi Sukma Dewata 2016-08-05 16:38:00 EDT
Fixed in master:
* f726f9a668b523c4e5a9438d8ea301f4b556efd4
* da66600e8ae07fa4169d24909c7d04ed69d2906c
* b7d4f6e9efd8b2e7d26a001f6c18a10b82df6b56

PKI has been fixed to import the PKCS #12 generated by Custodia properly. However, to fix IPA cloning properly it looks like Custodia needs to include the 3rd-party DS/HTTP certificates into the PKCS #12 file.
Comment 26 Geetika Kapoor 2016-08-16 05:48:57 EDT

IPa did some testing on new build and it fails in their setup.below is the location of logs.They follow same procedure as mentioned in comment#3.

location of logs and other details: http://file.pnq.redhat.com/~akasurde/logs/bz1301546/
Comment 27 Endi Sukma Dewata 2016-08-17 20:02:29 EDT

Most of the files in the above location is inaccessible. Could you fix the permission? Please also provide the exact commands and the input files (e.g. PKCS #12 files) used in the test. Thanks.
Comment 28 Abhijeet Kasurde 2016-08-29 09:05 EDT
Created attachment 1195378 [details]
Comment 29 Endi Sukma Dewata 2016-08-29 10:13:22 EDT

According to ipareplica1-ca-debug the replica does not trust the server certificate provide by the master:

[16/Aug/2016:07:54:10][http-bio-8443-exec-3]: Server certificate:
[16/Aug/2016:07:54:10][http-bio-8443-exec-3]:  - subject: CN=ipamaster73v2.testrelm.test,O=Example Organization
[16/Aug/2016:07:54:10][http-bio-8443-exec-3]:  - issuer: CN=CA,O=Example Organization
[16/Aug/2016:07:54:10][http-bio-8443-exec-3]: ERROR: UNTRUSTED_ISSUER

Is the master's server certificate generated by the CA on master, or is it a 3rd-party certificate? Does the PKCS #12 file provided to the replica contain this certificate?

Please note that as mentioned in comment #23 and comment #24 in order to complete the installation the IPA bug #1358752 needs to be fixed as well. Please try again with an IPA build that fixes the IPA issue when it becomes available. Thanks.
Comment 30 Asha Akkiangady 2016-08-31 14:05:33 EDT
This bug can be verified when IPA bug #1358752 gets fixed and lands ON_QA. 

Marking this bug ON_QA.
Comment 31 Endi Sukma Dewata 2016-09-22 11:05:49 EDT
I think this bug can actually be verified without IPA. Bug #1358752 is only required to verify with IPA.

Here are the steps:

1. Run "external CA" installation step 1 on PKI master.
2. Sign the CSR as usual with the external CA (i.e. external.crt).
3. Create another external CA (i.e. third-party.crt) unrelated to #2.
4. Import the third-party.crt into PKI:

 $ certutil -A -d /var/lib/pki/pki-tomcat/alias -n "Third-Party CA" -t "CT,C,C" -i third-party.crt

5. Run "external CA" installation step 2 on PKI master.
6. Clone PKI master to PKI replica.

Expected result:

Both PKI master's and and PKI replica's NSS databases should contain the third-party certificate, the external CA certificate, and the normal PKI certificates with the correct trust flags.

$ certutil -L -d /etc/pki/pki-tomcat/alias

Certificate Nickname                                         Trust Attributes

Third-Party CA                                               CT,C,C
caSigningCert External CA                                    CT,C,C
caSigningCert cert-pki-tomcat CA                             CTu,Cu,Cu
ocspSigningCert cert-pki-tomcat CA                           u,u,u
subsystemCert cert-pki-tomcat                                u,u,u
auditSigningCert cert-pki-tomcat CA                          u,u,Pu
Server-Cert cert-pki-tomcat                                  u,u,u
Comment 32 Geetika Kapoor 2016-09-26 00:21:22 EDT
Hi Jan.

This bugzilla that you have raised in which use case you have observed this bahavior? I mean this is with IPA or with pki alone.If would be great if you shared this information because in IPA case we are still observing failures .

Comment 33 Petr Vobornik 2016-09-26 04:19:51 EDT
Jan is not available this week.

It failed in conversion of self-signed setup on RHEL 6. Self-signed is obsoleted. It is similar to CA-less setup but with self-signed http, directory server certs.

This issue can be probably reproduced in CA-less to CA conversion if conditions in comment 0 are met.

Bug 1358752 blocks the conversion only if domain level 1 is used (default). For testing installation can be switched to domain level 0: ipa-server-install has --domain-level (or without the dash) The option is undocumented, because we want new installs to use domain level 1.
Comment 34 Geetika Kapoor 2016-09-26 05:24:01 EDT
Hello Petr,

So do you think it makes use to test this without IPA because i am aware about the Bug 1358752. I too believe for proper testing with IPA this bug needs to be fixed before.

Comment 35 Petr Vobornik 2016-09-26 06:27:22 EDT
The use case "old self-signed" -> "IPA with CA" is a basically a migration case where master is on RHEL 6 then a new replica is created. This means domain level 0. Domain level 1 is available only if all replicas are 7.3. 

So in theory you should be able to test it (assuming there isn't any other unknown bug).
Comment 36 Geetika Kapoor 2016-09-27 05:27:12 EDT
Hi Endi,

I tried to do the installation.I have noticed that my master External CA alias looks like ::

Master :

[root@pki1 externalCA]# certutil -L -d /etc/pki/TestExternal/alias
Certificate Nickname                                         Trust Attributes

Server-Cert cert-TestExternal                                u,u,u
auditSigningCert cert-TestExternal CA                        u,u,Pu
caSigningCert cert-TestExternal CA                           CTu,Cu,Cu
ocspSigningCert cert-TestExternal CA                         u,u,u
subsystemCert cert-TestExternal                              u,u,u
caSigningCert External CA                                    CT,C,C

I took backup of this nssdb and moved it in clone.

Clone side failed as it could not find "caSigningCert cert-TestExternal CA" in the nssdb.When i check nssdb i could see:

"caSigningCert cert-TestExternal CA" is replaced by "CA Signing Certificate - EXTERNAL" not sure what is happening there??

[root@pki2 ~]# certutil -L -d /var/lib/pki/TestExternal/alias

Certificate Nickname                                         Trust Attributes

Server-Cert cert-TestExternal                                u,u,u
CA Signing Certificate - EXTERNAL                            CT,C,C
auditSigningCert cert-TestExternal CA                        u,u,Pu
caSigningCert External CA                                    CT,C,C
ocspSigningCert cert-TestExternal CA                         u,u,u
subsystemCert cert-TestExternal                              u,u,u

I could see 

27/Sep/2016:04:39:03][http-bio-31142-exec-3]: SystemConfigService: verify certificates
[27/Sep/2016:04:39:03][http-bio-31142-exec-3]: ConfigurationUtils.verifySystemCertificates(): checking certificate caSigningCert cert-TestExternal CA
java.lang.Exception: Missing system certificate: caSigningCert cert-TestExternal CA
        at com.netscape.cms.servlet.csadmin.ConfigurationUtils.verifySystemCertificates(ConfigurationUtils.java:997)
        at org.dogtagpki.server.rest.SystemConfigService.configureClone(SystemConfigService.java:898)
        at org.dogtagpki.server.rest.SystemConfigService.configureSubsystem(SystemConfigService.java:1019)
        at org.dogtagpki.server.rest.SystemConfigService.configure(SystemConfigService.java:164)
        at org.dogtagpki.server.rest.SystemConfigService.configure(SystemConfigService.java:121)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Comment 37 Geetika Kapoor 2016-09-27 05:40:49 EDT
On clone machine during install nssdb cert nick changes::

[root@pki2 ~]# certutil -L -d /etc/pki/TestExternal/alias

Certificate Nickname                                         Trust Attributes

caSigningCert cert-TestExternal CA                           CTu,Cu,Cu
auditSigningCert cert-TestExternal CA                        u,u,Pu
Server-Cert cert-TestExternal                                CTu,Cu,Cu
caSigningCert External CA                                    CT,C,C
subsystemCert cert-TestExternal                              u,u,u
ocspSigningCert cert-TestExternal CA                         u,u,u

[root@pki2 ~]# certutil -L -d /etc/pki/TestExternal/alias

Certificate Nickname                                         Trust Attributes

Server-Cert cert-TestExternal                                CTu,Cu,Cu
CA Signing Certificate - EXTERNAL                            CT,C,C
auditSigningCert cert-TestExternal CA                        u,u,Pu
caSigningCert External CA                                    CT,C,C
subsystemCert cert-TestExternal                              u,u,u
ocspSigningCert cert-TestExternal CA                         u,u,u
[root@pki2 ~]#
Comment 39 Endi Sukma Dewata 2016-09-27 11:55:49 EDT

There's a known problem using the backup PKCS #12 file created during install for cloning a RHEL 6.8 machine (bug #1374877). Could you try again with a new PKCS #12 file exported after installation? Thanks.
Comment 40 Geetika Kapoor 2016-09-27 18:38:34 EDT
Hello Ade,

While doing clone installation , wherein master CA is an External CA , we could see below exceptions in clone debug logs.

[28/Sep/2016:11:26:59][http-bio-31142-exec-3]: setupReplication: Waiting for replication to complete
[28/Sep/2016:11:27:00][http-bio-31142-exec-3]: replicationDone: dn: cn=masterAgreement1-pki2.example.com-TestExternal_master,cn=replica,cn="dc=camaster,dc=pki,dc=example,dc=com",cn=mapping tree,cn=config
[28/Sep/2016:11:27:01][http-bio-31142-exec-3]: replicationStatus: dn: cn=masterAgreement1-pki2.example.com-TestExternal_master,cn=replica,cn="dc=camaster,dc=pki,dc=example,dc=com",cn=mapping tree,cn=config
[28/Sep/2016:11:27:01][http-bio-31142-exec-3]: setupReplication: consumer initialization failed. -1  - LDAP error: Can't contact LDAP server
[28/Sep/2016:11:27:01][http-bio-31142-exec-3]: setupReplication: java.io.IOException: consumer initialization failed. -1  - LDAP error: Can't contact LDAP server
java.io.IOException: Failed to setup the replication for cloning.
        at com.netscape.cms.servlet.csadmin.ConfigurationUtils.setupReplication(ConfigurationUtils.java:2038)
        at org.dogtagpki.server.rest.SystemConfigService.initializeDatabase(SystemConfigService.java:779)
        at org.dogtagpki.server.ca.rest.CAInstallerService.initializeDatabase(CAInstallerService.java:100)
        at org.dogtagpki.server.rest.SystemConfigService.configure(SystemConfigService.java:179)
        at org.dogtagpki.server.rest.SystemConfigService.configure(SystemConfigService.java:121)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)

Could you please  look into this.
Comment 42 Geetika Kapoor 2016-09-28 01:01:05 EDT
I am moving it to verified from pki side. We still need to try once  again when Bug 1358752 gets fixed.
Comment 44 errata-xmlrpc 2016-11-04 01:22:02 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.