Bug 1301546

Summary: pkispawn ignores 3rd party CA certs in pki_clone_pkcs12_path
Product: Red Hat Enterprise Linux 7 Reporter: Jan Cholasta <jcholast>
Component: pki-coreAssignee: Endi Sukma Dewata <edewata>
Status: CLOSED ERRATA QA Contact: Asha Akkiangady <aakkiang>
Severity: high Docs Contact:
Priority: high    
Version: 7.3CC: akasurde, alee, arubin, cfu, edewata, ekeck, gkapoor, jcholast, jmagne, lfriedma, mbasti, mharmsen, mkolaja, ndehadra, nkinder, pvoborni
Target Milestone: rcKeywords: ZStream
Target Release: 7.3   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: pki-core-10.3.1-1.el7, pki-core-10.3.3-5.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1318302 (view as bug list) Environment:
Last Closed: 2016-11-04 05:22:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1301687, 1318302, 1323836    
Attachments:
Description Flags
bz1301546.tar.gz none

Description Jan Cholasta 2016-01-25 11:19:38 UTC
Description of problem:
If pki_clone_uri in pkispawn config file points to a host which uses a server cert signed by a 3rd party CA, pkispawn will fail even if the 3rd party CA cert is present in the PKCS#12 file specified by pki_clone_pkcs12_path.

This causes a failure in IPA replica install: https://fedorahosted.org/freeipa/ticket/5602

Version-Release number of selected component (if applicable):
pki-core-10.2.5-6.el7

How reproducible:
Always.

Steps to Reproduce:
1. Run pkispawn with the config described above
2.
3.

Actual results:
pkispawn fails

Expected results:
pkispawn succeeds

Additional info:

Comment 2 Matthew Harmsen 2016-01-25 22:13:55 UTC
Upstream ticket:
https://fedorahosted.org/pki/ticket/1742

Comment 3 Jan Cholasta 2016-02-01 17:11:27 UTC
Steps to reproduce:

1. Install CA-less IPA server with:

root@server# ipa-server-install --dirsrv-cert-file=/path/to/server-cert.p12 --http-cert-file=/path/to/server-cert.p12

(You have to provide private key and a server certificate issued by an arbitrary 3rd party CA in server-cert.p12.)

2. Install the CA on the IPA server with:

root@server# ipa-ca-install

3. Prepare a replica with:

root@server# ipa-replica-prepare replica.example.com

(Copy the resulting replica file to the replica host.)

4. Install IPA replica with:

root@replica# ipa-replica-install replica-info-replica.example.com.gpg

5. Install CA clone on the replica with:

root@replica# ipa-ca-install replica-info-replica.example.com.gpg

Comment 4 Jan Cholasta 2016-02-01 17:20:11 UTC
You can use the makepki.sh script from <https://www.redhat.com/archives/freeipa-devel/2015-November/msg00487.html> to easily generate server-cert.p12.

Comment 6 Endi Sukma Dewata 2016-02-02 01:24:48 UTC
The makepki.sh does not generate a server-cert.p12 file. I tried using the nssdb/ca1/server.p12, the ipa-replica-install failed with this output:

  [17/38]: configuring ssl for ds instance
  [error] RuntimeError: Could not find a CA cert in /tmp/tmpIjLRdIipa/realm_info/dscert.p12
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

ipa.ipapython.install.cli.install_tool(Replica): ERROR    Could not find a CA cert in /tmp/tmpIjLRdIipa/realm_info/dscert.p12

This was tested on F23. Could you provide the proper steps to generate server-cert.p12?

Comment 7 Jan Cholasta 2016-02-02 06:26:44 UTC
The file name does not matter, I assumed you would be able to figure that out.

I forgot one step in comment 3, between steps 2 and 3:

2.5. Update local filesystem with the new CA certificate on the IPA server:

root@server# ipa-certupdate

(Note that you need IPA build with the fix for https://fedorahosted.org/freeipa/ticket/5595 in order for ipa-certupdate to work properly in this case.)

Comment 14 Endi Sukma Dewata 2016-03-17 09:02:53 UTC
Does IPA keep track the nicknames of the 3rd-party CA certificates? Where are the nicknames stored in IPA?

Comment 15 Jan Cholasta 2016-03-17 09:31:22 UTC
Yes, /etc/ipa/nssdb.

Comment 16 Endi Sukma Dewata 2016-03-18 22:43:41 UTC
Fixed in master:

* aa613fa272defcc8eebd4b9ef2556e61683b4e97
* 709457876a6d5e4aea281a35350667492bc34df8
* 54849505729d3f6345bc7b530e5a40c14ff36116
* 6947854a3ab6ee4f296a5f97850f5521572683a1
* 0d44556fa78203121a24224d4733b89c36ef9cc9
* a96ecbae1bfa27223bbebc7a67f695b643c4aebe
* 67a0c95b8622b18c9803b2bfe0f708be8747f896
* 67402ac16d2635ab3464568ca007cf81c4db73e6
* b74bf9b82102715e08fa3fd3bd5ce9462312aded
* b48889a2ef41fd45ca69c3926c36ef075777447c
* 1d58b883ff9d0056d89d74d30f1375ab12d01f03
* 935633c5ea9f2b5c4321d924af166367008ac4b3
* 0dadf421c327bc32d220405208031a9f7e1bb097
* 9c6b53ac8f6eee2eb8ed8f47a4b26be828626841
* 20a70830961f532e9483baefb64cc92af7cda8b2
* 1b15c725b6e9c5d9057b66e0a2806a7813a8d61b
* 04055a9bc40486950a3288acf610522e767c1e27
* c14e8c52ae7a2c15433fe9568c393c1d0e7a1301

Note: there is no change required in IPA.

Comment 18 Matthew Harmsen 2016-03-31 15:45:19 UTC
PKI TRAC Ticket #2247 fixes checked into master:
    * 58b78bd1602e3efeb33a73f8d07a6edaaee104ba
    * 061bec70264c2c7a601ffe80846ef1fa5497c15c 

Note that there is another error similar to ticket #2226 that blocks KRA installation on IPA replica.

Comment 19 Matthew Harmsen 2016-03-31 16:07:15 UTC
Steps to verify changes for PKI TRAC Ticket #2247:

    1.) setup a FreeIPA master w/ KRA

    2.) install a replica with CA

    3.) install KRA clone on the replica

KRA clone should be installed and functional.

Comment 20 Matthew Harmsen 2016-04-21 18:38:52 UTC
*** Bug 1328522 has been marked as a duplicate of this bug. ***

Comment 22 Martin Bašti 2016-07-27 12:17:34 UTC
*** Bug 1358752 has been marked as a duplicate of this bug. ***

Comment 23 Martin Bašti 2016-07-27 12:19:34 UTC
Hello, we (IPA team) are afraid that this bug is not fixed completely, please see bz1358752

Comment 24 Endi Sukma Dewata 2016-08-05 20:38:00 UTC
Fixed in master:
* f726f9a668b523c4e5a9438d8ea301f4b556efd4
* da66600e8ae07fa4169d24909c7d04ed69d2906c
* b7d4f6e9efd8b2e7d26a001f6c18a10b82df6b56

PKI has been fixed to import the PKCS #12 generated by Custodia properly. However, to fix IPA cloning properly it looks like Custodia needs to include the 3rd-party DS/HTTP certificates into the PKCS #12 file.

Comment 26 Geetika Kapoor 2016-08-16 09:48:57 UTC
Hello,

IPa did some testing on new build and it fails in their setup.below is the location of logs.They follow same procedure as mentioned in comment#3.

location of logs and other details: http://file.pnq.redhat.com/~akasurde/logs/bz1301546/

Comment 27 Endi Sukma Dewata 2016-08-18 00:02:29 UTC
Abhijeet,

Most of the files in the above location is inaccessible. Could you fix the permission? Please also provide the exact commands and the input files (e.g. PKCS #12 files) used in the test. Thanks.

Comment 28 Abhijeet Kasurde 2016-08-29 13:05:18 UTC
Created attachment 1195378 [details]
bz1301546.tar.gz

Comment 29 Endi Sukma Dewata 2016-08-29 14:13:22 UTC
Abhijeet,

According to ipareplica1-ca-debug the replica does not trust the server certificate provide by the master:

[16/Aug/2016:07:54:10][http-bio-8443-exec-3]: Server certificate:
[16/Aug/2016:07:54:10][http-bio-8443-exec-3]:  - subject: CN=ipamaster73v2.testrelm.test,O=Example Organization
[16/Aug/2016:07:54:10][http-bio-8443-exec-3]:  - issuer: CN=CA,O=Example Organization
[16/Aug/2016:07:54:10][http-bio-8443-exec-3]: ERROR: UNTRUSTED_ISSUER

Is the master's server certificate generated by the CA on master, or is it a 3rd-party certificate? Does the PKCS #12 file provided to the replica contain this certificate?

Please note that as mentioned in comment #23 and comment #24 in order to complete the installation the IPA bug #1358752 needs to be fixed as well. Please try again with an IPA build that fixes the IPA issue when it becomes available. Thanks.

Comment 30 Asha Akkiangady 2016-08-31 18:05:33 UTC
This bug can be verified when IPA bug #1358752 gets fixed and lands ON_QA. 

Marking this bug ON_QA.

Comment 31 Endi Sukma Dewata 2016-09-22 15:05:49 UTC
I think this bug can actually be verified without IPA. Bug #1358752 is only required to verify with IPA.

Here are the steps:

1. Run "external CA" installation step 1 on PKI master.
2. Sign the CSR as usual with the external CA (i.e. external.crt).
3. Create another external CA (i.e. third-party.crt) unrelated to #2.
4. Import the third-party.crt into PKI:

 $ certutil -A -d /var/lib/pki/pki-tomcat/alias -n "Third-Party CA" -t "CT,C,C" -i third-party.crt

5. Run "external CA" installation step 2 on PKI master.
6. Clone PKI master to PKI replica.

Expected result:

Both PKI master's and and PKI replica's NSS databases should contain the third-party certificate, the external CA certificate, and the normal PKI certificates with the correct trust flags.

$ certutil -L -d /etc/pki/pki-tomcat/alias

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

Third-Party CA                                               CT,C,C
caSigningCert External CA                                    CT,C,C
caSigningCert cert-pki-tomcat CA                             CTu,Cu,Cu
ocspSigningCert cert-pki-tomcat CA                           u,u,u
subsystemCert cert-pki-tomcat                                u,u,u
auditSigningCert cert-pki-tomcat CA                          u,u,Pu
Server-Cert cert-pki-tomcat                                  u,u,u

Comment 32 Geetika Kapoor 2016-09-26 04:21:22 UTC
Hi Jan.

This bugzilla that you have raised in which use case you have observed this bahavior? I mean this is with IPA or with pki alone.If would be great if you shared this information because in IPA case we are still observing failures .

Thanks
Geetika

Comment 33 Petr Vobornik 2016-09-26 08:19:51 UTC
Jan is not available this week.

It failed in conversion of self-signed setup on RHEL 6. Self-signed is obsoleted. It is similar to CA-less setup but with self-signed http, directory server certs.

This issue can be probably reproduced in CA-less to CA conversion if conditions in comment 0 are met.

Bug 1358752 blocks the conversion only if domain level 1 is used (default). For testing installation can be switched to domain level 0: ipa-server-install has --domain-level (or without the dash) The option is undocumented, because we want new installs to use domain level 1.

Comment 34 Geetika Kapoor 2016-09-26 09:24:01 UTC
Hello Petr,

So do you think it makes use to test this without IPA because i am aware about the Bug 1358752. I too believe for proper testing with IPA this bug needs to be fixed before.

Thanks

Comment 35 Petr Vobornik 2016-09-26 10:27:22 UTC
The use case "old self-signed" -> "IPA with CA" is a basically a migration case where master is on RHEL 6 then a new replica is created. This means domain level 0. Domain level 1 is available only if all replicas are 7.3. 

So in theory you should be able to test it (assuming there isn't any other unknown bug).

Comment 36 Geetika Kapoor 2016-09-27 09:27:12 UTC
Hi Endi,

I tried to do the installation.I have noticed that my master External CA alias looks like ::

Master :

[root@pki1 externalCA]# certutil -L -d /etc/pki/TestExternal/alias
Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

Server-Cert cert-TestExternal                                u,u,u
auditSigningCert cert-TestExternal CA                        u,u,Pu
caSigningCert cert-TestExternal CA                           CTu,Cu,Cu
ocspSigningCert cert-TestExternal CA                         u,u,u
subsystemCert cert-TestExternal                              u,u,u
caSigningCert External CA                                    CT,C,C


I took backup of this nssdb and moved it in clone.

Clone side failed as it could not find "caSigningCert cert-TestExternal CA" in the nssdb.When i check nssdb i could see:

"caSigningCert cert-TestExternal CA" is replaced by "CA Signing Certificate - EXTERNAL" not sure what is happening there??

[root@pki2 ~]# certutil -L -d /var/lib/pki/TestExternal/alias

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

Server-Cert cert-TestExternal                                u,u,u
CA Signing Certificate - EXTERNAL                            CT,C,C
auditSigningCert cert-TestExternal CA                        u,u,Pu
caSigningCert External CA                                    CT,C,C
ocspSigningCert cert-TestExternal CA                         u,u,u
subsystemCert cert-TestExternal                              u,u,u

I could see 

<logs>
27/Sep/2016:04:39:03][http-bio-31142-exec-3]: SystemConfigService: verify certificates
[27/Sep/2016:04:39:03][http-bio-31142-exec-3]: ConfigurationUtils.verifySystemCertificates(): checking certificate caSigningCert cert-TestExternal CA
java.lang.Exception: Missing system certificate: caSigningCert cert-TestExternal CA
        at com.netscape.cms.servlet.csadmin.ConfigurationUtils.verifySystemCertificates(ConfigurationUtils.java:997)
        at org.dogtagpki.server.rest.SystemConfigService.configureClone(SystemConfigService.java:898)
        at org.dogtagpki.server.rest.SystemConfigService.configureSubsystem(SystemConfigService.java:1019)
        at org.dogtagpki.server.rest.SystemConfigService.configure(SystemConfigService.java:164)
        at org.dogtagpki.server.rest.SystemConfigService.configure(SystemConfigService.java:121)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

Comment 37 Geetika Kapoor 2016-09-27 09:40:49 UTC
On clone machine during install nssdb cert nick changes::

[root@pki2 ~]# certutil -L -d /etc/pki/TestExternal/alias

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

caSigningCert cert-TestExternal CA                           CTu,Cu,Cu
auditSigningCert cert-TestExternal CA                        u,u,Pu
Server-Cert cert-TestExternal                                CTu,Cu,Cu
caSigningCert External CA                                    CT,C,C
subsystemCert cert-TestExternal                              u,u,u
ocspSigningCert cert-TestExternal CA                         u,u,u

[root@pki2 ~]# certutil -L -d /etc/pki/TestExternal/alias

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

Server-Cert cert-TestExternal                                CTu,Cu,Cu
CA Signing Certificate - EXTERNAL                            CT,C,C
auditSigningCert cert-TestExternal CA                        u,u,Pu
caSigningCert External CA                                    CT,C,C
subsystemCert cert-TestExternal                              u,u,u
ocspSigningCert cert-TestExternal CA                         u,u,u
[root@pki2 ~]#

Comment 39 Endi Sukma Dewata 2016-09-27 15:55:49 UTC
Geetika,

There's a known problem using the backup PKCS #12 file created during install for cloning a RHEL 6.8 machine (bug #1374877). Could you try again with a new PKCS #12 file exported after installation? Thanks.

Comment 40 Geetika Kapoor 2016-09-27 22:38:34 UTC
Hello Ade,

While doing clone installation , wherein master CA is an External CA , we could see below exceptions in clone debug logs.

[28/Sep/2016:11:26:59][http-bio-31142-exec-3]: setupReplication: Waiting for replication to complete
[28/Sep/2016:11:27:00][http-bio-31142-exec-3]: replicationDone: dn: cn=masterAgreement1-pki2.example.com-TestExternal_master,cn=replica,cn="dc=camaster,dc=pki,dc=example,dc=com",cn=mapping tree,cn=config
[28/Sep/2016:11:27:01][http-bio-31142-exec-3]: replicationStatus: dn: cn=masterAgreement1-pki2.example.com-TestExternal_master,cn=replica,cn="dc=camaster,dc=pki,dc=example,dc=com",cn=mapping tree,cn=config
[28/Sep/2016:11:27:01][http-bio-31142-exec-3]: setupReplication: consumer initialization failed. -1  - LDAP error: Can't contact LDAP server
[28/Sep/2016:11:27:01][http-bio-31142-exec-3]: setupReplication: java.io.IOException: consumer initialization failed. -1  - LDAP error: Can't contact LDAP server
java.io.IOException: Failed to setup the replication for cloning.
        at com.netscape.cms.servlet.csadmin.ConfigurationUtils.setupReplication(ConfigurationUtils.java:2038)
        at org.dogtagpki.server.rest.SystemConfigService.initializeDatabase(SystemConfigService.java:779)
        at org.dogtagpki.server.ca.rest.CAInstallerService.initializeDatabase(CAInstallerService.java:100)
        at org.dogtagpki.server.rest.SystemConfigService.configure(SystemConfigService.java:179)
        at org.dogtagpki.server.rest.SystemConfigService.configure(SystemConfigService.java:121)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)


Could you please  look into this.

Comment 42 Geetika Kapoor 2016-09-28 05:01:05 UTC
I am moving it to verified from pki side. We still need to try once  again when Bug 1358752 gets fixed.

Comment 44 errata-xmlrpc 2016-11-04 05:22:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2396.html