Bug 1358752
|
Description
Abhijeet Kasurde
2016-07-21 12:16:55 UTC
Created attachment 1182459 [details]
ipareplica1-ca_debug.log
Created attachment 1182461 [details]
ipamaster1-tomcat.log
Created attachment 1182463 [details]
ipareplica1-tomcat.log
Created attachment 1182464 [details]
ipamaster1-ca_debug.log
Created attachment 1182465 [details]
ipamaster1-ipaclient-install.log
Created attachment 1182466 [details]
ipamaster1-ipareplica-conncheck.log
Created attachment 1182468 [details]
ipamaster1-ipareplica-install.log
Created attachment 1182469 [details]
ipamaster1-ipaserver-ca-install.log
Created attachment 1182470 [details]
ipareplica1-ipaclient-install.log
Created attachment 1182474 [details]
ipareplica1-ipaserver-ca-install.log
Created attachment 1182475 [details]
ipareplica1-ipaserver-install.log
Created attachment 1182489 [details]
pkispawn_ca.cfg
seems that ipareplica1-ca_debug.log contains content of ipareplica1-tomcat.log. And ipamaster1-ipaserver-ca-install.log and ipareplica1-ipaserver-ca-install.log seems to be switched. Could you attach proper ipareplica1-ca_debug.log? Or ideally replica's pki spawn log. Created attachment 1182691 [details]
ipareplica1-ca_debug.log
Created attachment 1182692 [details]
ipamaster1-ipaserver-ca-install.log
Created attachment 1182694 [details]
ipareplica1-ipaserver-ca-install.log
*** This bug has been marked as a duplicate of bug 1301546 *** Hi, I'm fixing PKI to work with the PKCS #12 file generated by Custodia. However, it looks like the PKCS #12 file does not contain the 3rd-party SSL certificate (server.p12) which needs to be trusted by PKI replica in order to communicate with PKI master via IPA's HTTP proxy. Could someone confirm which SSL certificate is used by the HTTP proxy in this particular configuration? Also, since PKI master is configured to use an SSL connection to the DS, the PKI replica must also be configured with pki_ds_secure_connection=True. The attached ipareplica1-ipaserver-install.log indicates that that is not the case. With the new PKI packages the /var/log/pki/pki-tomcat/ca/debug shows that the master HTTP proxy is using the 3rd-party SSL certificate, not the IPA certificate generated by PKI: [04/Aug/2016:00:21:25][http-bio-8443-exec-3]: ConfigurationUtils: GET https://master.example.com:443/ca/admin/ca/getDomainXML ... [04/Aug/2016:00:21:25][http-bio-8443-exec-3]: Server certificate: [04/Aug/2016:00:21:25][http-bio-8443-exec-3]: - subject: CN=master.example.com [04/Aug/2016:00:21:25][http-bio-8443-exec-3]: - issuer: CN="Certificate Authority,O=EXTERNAL" [04/Aug/2016:00:21:25][http-bio-8443-exec-3]: ERROR: UNTRUSTED_ISSUER javax.ws.rs.ProcessingException: Unable to invoke request at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:287) ... Also the /var/log/ipareplica-ca-install.log shows that neither the 3rd-party SSL certificate or its CA certificate is included in the PKCS #12 file: Importing certificates from /tmp/ca.p12: --------------- 4 entries found --------------- Certificate ID: 7fa6641e55d9693b49207f64f5f8932a2eb3d9bb Serial Number: 0x1 Nickname: caSigningCert cert-pki-ca Subject DN: CN=Certificate Authority,O=EXAMPLE Issuer DN: CN=Certificate Authority,O=EXAMPLE Trust Flags: u,u,u Has Key: true Certificate ID: 124db33431d3e90dfb05a17a5b8a1b65160a983c Serial Number: 0x5 Nickname: auditSigningCert cert-pki-ca Subject DN: CN=CA Audit,O=EXAMPLE Issuer DN: CN=Certificate Authority,O=EXAMPLE Trust Flags: u,u,u Has Key: true Certificate ID: ba9330c3bedbbc0014203cd74082de1c539fa0e7 Serial Number: 0x2 Nickname: ocspSigningCert cert-pki-ca Subject DN: CN=OCSP Subsystem,O=EXAMPLE Issuer DN: CN=Certificate Authority,O=EXAMPLE Trust Flags: u,u,u Has Key: true Certificate ID: e2319cb39ff2a48e4a949685d161a712430f8d9c Serial Number: 0x4 Nickname: subsystemCert cert-pki-ca Subject DN: CN=CA Subsystem,O=EXAMPLE Issuer DN: CN=Certificate Authority,O=EXAMPLE Trust Flags: u,u,u Has Key: true --------------- Import complete --------------- Imported certificates in /etc/pki/pki-tomcat/alias: Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI caSigningCert cert-pki-ca CTu,Cu,Cu auditSigningCert cert-pki-ca u,u,Pu ocspSigningCert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u First ipa-ca-install uses LDAPS because of https://fedorahosted.org/freeipa/ticket/5570 The logic in IPA code is: """ if not self.create_ra_agent_db and not self.clone: self._use_ldaps_during_spawn(config) """ """ def _use_ldaps_during_spawn(self, config, ds_cacert=paths.IPA_CA_CRT): config.set(self.subsystem, "pki_ds_ldaps_port", "636") config.set(self.subsystem, "pki_ds_secure_connection", "True") config.set(self.subsystem, "pki_ds_secure_connection_ca_pem_file", ds_cacert) """ Does it mean that in CA-less -> CA scenario every CA replica will use LDAPS instead of TLS over port 389 as it is in installations where CA is by default? Can LDAPS be used only for spawning the CA db on first master and then migrated to TLS so that installations of additional replicas can use TLS? I'd suggest we investigate the second issue (i.e. LDAPS issue) after we fix the first one (i.e. missing 3rd-party certs in PKCS #12 file). I need to investigate the proper procedure to set up a CA replica when the CA master is configured with LDAPS. Hi Petr, Do you know which SSL certificate is supposed to be used by the HTTP proxy in this scenario (see comment #19)? Thanks. Hi Endi, as IPA was initially CA_less, HTTP is using a 3rd part certificate. When ipa-ca-install is run on the master, the cert for HTTP is not modified and is still the one signed by the 3rd part CA. I must check how we can provide this trust anchor to pkispawn. Upstream ticket: https://fedorahosted.org/freeipa/ticket/6207 Fixed upstream master: https://fedorahosted.org/freeipa/changeset/6581389ac3ac1c6a0dbeb18d80e3fef69b158cc8 bz1365858 must be fixed too for sucessful installation Fixed upstream master: https://fedorahosted.org/freeipa/changeset/17ea4ae6b9007e121ae1ea7748643394fec84ad7 Marking this BZ as FailedQA, because installer is not successful and bz1365858 must be fixed for successful installation. FAILED_QA Hit an issue which might be related to: https://pagure.io/freeipa/issue/6577 ipa-server-4.5.0-11.el7.x86_64 pki-ca-10.4.1-4.el7.noarch 1. Install IPA server [root@master ~]# ipa-server-install --ip-address $(ip addr|grep "global"|cut -d " " -f6|cut -d "/" -f1|head -n 1) -r TESTRELM.TEST -n testrelm.test -p 'Secret123' -a 'Secret123' --setup-dns --forwarder 192.168.222.1 -U --dirsrv-cert-file=./server.p12 --http-cert-file=./server.p12 --dirsrv-pin Secret123 --http-pin Secret123 --no-pkinit 2. Install CA-less replica [root@replica ~]# ipa-client-install -U --domain testrelm.test --realm TESTRELM.TEST -p admin -w Secret123 --server master.testrelm.test [root@replica ~]# ipa-replica-install -U --dirsrv-cert-file=./replica.p12 --http-cert-file=./replica.p12 --dirsrv-pin Secret123 --http-pin Secret123 -P admin -w Secret123 --no-pkinit 3. Promote IPA server from CA-less to CA-full [root@master ~]# ipa-ca-install 4. Promote IPA replica from CA-less to CA-full [root@replica1 ~]# ipa-ca-install Directory Manager (existing master) password: Run connection check to master Connection check OK /usr/lib/python2.7/site-packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) SecurityWarning Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes [1/27]: creating certificate server db [2/27]: setting up initial replication Starting replication, please wait until this has completed. Update in progress, 3 seconds elapsed Update succeeded [3/27]: creating installation admin user [4/27]: configuring certificate server instance [5/27]: exporting Dogtag certificate store pin [6/27]: stopping certificate server instance to update CS.cfg [7/27]: backing up CS.cfg [8/27]: disabling nonces [9/27]: set up CRL publishing [10/27]: enable PKIX certificate path discovery and validation [11/27]: destroying installation admin user [12/27]: starting certificate server instance [13/27]: configure certmonger for renewals [14/27]: Importing RA key [15/27]: setting up signing cert profile [16/27]: setting audit signing renewal to 2 years [17/27]: restarting certificate server [18/27]: authorizing RA to modify profiles [19/27]: authorizing RA to manage lightweight CAs [20/27]: Ensure lightweight CAs container exists [21/27]: configure certificate renewals [22/27]: configure Server-Cert certificate renewal [23/27]: Configure HTTP to proxy connections [24/27]: restarting certificate server [25/27]: updating IPA configuration [26/27]: enabling CA instance [27/27]: configuring certmonger renewal for lightweight CAs Done configuring certificate server (pki-tomcatd). Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up. CA did not start in 300.0s [root@replica1 ~]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED ipa-otpd Service: RUNNING [root@replica1 ~]# systemctl status pki-tomcatd ● pki-tomcatd - PKI Tomcat Server pki-tomcat Loaded: loaded (/usr/lib/systemd/system/pki-tomcatd@.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2017-05-16 07:02:14 EDT; 8min ago Process: 22844 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, status=0/SUCCESS) Main PID: 22970 (java) CGroup: /system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd └─22970 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java -DRESTEASY_LIB=/usr/share/java/resteasy-base -Djava.library.path=/usr/lib64/nuxwdog-jni -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tom... May 16 07:10:19 replica1 server[22970]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@5c47a92a background process May 16 07:10:19 replica1 server[22970]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable May 16 07:10:19 replica1 server[22970]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) May 16 07:10:19 replica1 server[22970]: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) May 16 07:10:19 replica1 server[22970]: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) May 16 07:10:19 replica1 server[22970]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) May 16 07:10:19 replica1 server[22970]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) May 16 07:10:19 replica1 server[22970]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) May 16 07:10:19 replica1 server[22970]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) May 16 07:10:19 replica1 server[22970]: at java.lang.Thread.run(Thread.java:748) Created attachment 1279275 [details]
pki-ca-spawn
Created attachment 1279276 [details]
ca_debug20170516
Created attachment 1279277 [details]
journalctl_pki-tomcat20170516
Additional info: [root@master ~]# ipa cert-find ipa: ERROR: cannot connect to 'https://master.testrelm.test/ipa/json': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579) [root@master ~]# python /usr/lib/python2.7/site-packages/requests/certs.py /etc/pki/tls/certs/ca-bundle.crt [root@master ~]# openssl crl2pkcs7 -nocrl -certfile /etc/pki/tls/certs/ca-bundle.crt | openssl pkcs7 -print_certs -noout | grep -i example subject=/O=Example Organization/CN=CA issuer=/O=Example Organization/CN=CA [root@replica1 ~]# ipa cert-find ipa: ERROR: an internal error has occurred Marking failed_qa as per comments 42-46 Temporarily withdrawing "failed_qa" as the error might have been caused by bad certificates. Will need to run the checks again. UPDATE: there were no issue with certificates as I originally thought. The issue seems to be the same as originally reported with pki-server-10.4.1-3.el7.noarch. [3/27]: creating installation admin user [4/27]: configuring certificate server instance ipa.ipaserver.install.cainstance.CAInstance: CRITICAL Failed to configure CA instance: Command '/usr/sbin/pkispawn -s CA -f /tmp/tmpzMUgcv' returned non-zero exit status 1 ipa.ipaserver.install.cainstance.CAInstance: CRITICAL See the installation logs and the following files/directories for more information: ipa.ipaserver.install.cainstance.CAInstance: CRITICAL /var/log/pki/pki-tomcat [error] RuntimeError: CA configuration failed. After upgrade to pki-server-10.4.1-4.el7.noarch we are hitting the issue as per comments 42-46. Attaching new logs. Created attachment 1279654 [details]
20170517_master_journaclt_pki-tomcat
Created attachment 1279655 [details]
20170517_master_ca-debug
Created attachment 1279656 [details]
20170517_master_pki-spawn071437
Created attachment 1279657 [details]
20170517_replica_journaclt_pki-tomcat
Created attachment 1279659 [details]
20170517_replica_pki-spawn071846
Created attachment 1279660 [details]
20170517_replica_ca-debug
Hi Michal, the procedure in comment 42 is not complete as it lacks the step running ipa-certupdate on all machines after ipa-ca-install. If this step is not run, then the replica does not download the new CA cert, and when Dogtag starts on the replica, it fails to connect to LDAP server (it connects through SSL port using subsystemCert, which is delivered by the new CA, and LDAP server does not trust this CA until ipa-certupdate has been run). This does not explain why ipa-ca-install fails on the master, though. I am still investigating the issue. Hi Michal, please ignore the last line of my comment 56: "This does not explain why ipa-ca-install fails on the master, though. I am still investigating the issue." I wrongly assumed that the error happened during ipa-ca-install on the master. If the error happens on the replica, then it is expected as the ipa-certupdate step is missing. But the failure should happen when pki-tomcat is restarted after step 27, not during pkispawn. The attached logs do not show any issue during pkispawn, can you check if they are the right ones, and also try the procedure with the ipa-certupdate step on the replica after ipa-ca-install on the master? Thank you Hi Flo, I was able to issue "ipa-certupdate" on both master and replica after [SSL: CERTIFICATE_VERIFY_FAILED] issue workaround (adding CA cert that signed the httpd cert to /etc/ipa/ca.crt). Then with pki-server-10.4.1-4.el7.noarch I was able to successfully proceed with all steps. Thank you for your assistance. Verified on: ipa-server-4.5.0-13.el7.x86_64 pki-server-10.4.1-4.el7.noarch selinux-policy-3.13.1-150.el7.noarch [root@master ~]# getenforce Enforcing [root@replica ~]# getenforce Enforcing 1. Install CA-less IPA server [root@master ~]# ipa-server-install --ip-address $(ip addr|grep "global"|cut -d " " -f6|cut -d "/" -f1|head -n 1) -r TESTRELM.TEST -n testrelm.test -p 'XXX' -a 'XXX' --setup-dns --forwarder 192.168.222.1 -U --dirsrv-cert-file=./server.p12 --http-cert-file=./server.p12 --dirsrv-pin XXX --http-pin XXX --no-pkinit 2. Install CA-less replica [root@replica ~]# ipa-client-install -U --domain testrelm.test --realm TESTRELM.TEST -p admin -w XXX --server master.testrelm.test [root@replica ~]# ipa-replica-install -U --dirsrv-cert-file=./replica.p12 --http-cert-file=./replica.p12 --dirsrv-pin XXX --http-pin XXX -P admin -w XXX --no-pkinit 3. Promote IPA server from CA-less to CA-full [root@master ~]# ipa-ca-install Workaround for [SSL: CERTIFICATE_VERIFY_FAILED] (not related to the tested issue): [root@master2 ~]# cat nssdb/ca1.pem >> /etc/ipa/ca.crt 4. Update local IPA certificate databases with certificates from the server [root@master ~]# ipa-certupdate trying https://master.testrelm.test/ipa/json Forwarding 'ca_is_enabled' to json server 'https://master.testrelm.test/ipa/json' Forwarding 'ca_find/1' to json server 'https://master.testrelm.test/ipa/json' Systemwide CA database updated. Systemwide CA database updated. The ipa-certupdate command was successful 5. Update local IPA certificate databases with certificates from the server [root@replica ~]# kinit admin Password for admin: [root@replica ~]# [root@replica ~]# ipa-certupdate trying https://replica.testrelm.test/ipa/json Forwarding 'ca_is_enabled' to json server 'https://replica.testrelm.test/ipa/json' Forwarding 'ca_find/1' to json server 'https://replica.testrelm.test/ipa/json' Systemwide CA database updated. Systemwide CA database updated. The ipa-certupdate command was successful 6. Promote IPA replica from CA-less to CA-full [root@replica ~]# ipa-ca-install Directory Manager (existing master) password: Run connection check to master Connection check OK /usr/lib/python2.7/site-packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) SecurityWarning Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes [1/27]: creating certificate server db [2/27]: setting up initial replication Starting replication, please wait until this has completed. Update in progress, 4 seconds elapsed Update succeeded [3/27]: creating installation admin user [4/27]: configuring certificate server instance [5/27]: exporting Dogtag certificate store pin [6/27]: stopping certificate server instance to update CS.cfg [7/27]: backing up CS.cfg [8/27]: disabling nonces [9/27]: set up CRL publishing [10/27]: enable PKIX certificate path discovery and validation [11/27]: destroying installation admin user [12/27]: starting certificate server instance [13/27]: configure certmonger for renewals [14/27]: Importing RA key [15/27]: setting up signing cert profile [16/27]: setting audit signing renewal to 2 years [17/27]: restarting certificate server [18/27]: authorizing RA to modify profiles [19/27]: authorizing RA to manage lightweight CAs [20/27]: Ensure lightweight CAs container exists [21/27]: configure certificate renewals [22/27]: configure Server-Cert certificate renewal [23/27]: Configure HTTP to proxy connections [24/27]: restarting certificate server [25/27]: updating IPA configuration [26/27]: enabling CA instance [27/27]: configuring certmonger renewal for lightweight CAs Done configuring certificate server (pki-tomcatd). Updating DNS system records Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2304 |