Red Hat Bugzilla – Bug 1114127
RHEL6.5 ipa-server-install fails configuring certificate server instance
Last modified: 2016-01-29 08:21:00 EST
Description of problem: I'm seeing ipa-server-install failures configuring the CA. Configuring certificate server (pki-cad): Estimated time 3 minutes 30 seconds [1/21]: creating certificate server user [2/21]: creating pki-ca instance [3/21]: configuring certificate server instance ipa : CRITICAL failed to configure ca instance Command '/usr/bin/perl /usr/bin/pkisilent ConfigureCA -cs_hostname pit-scenario2-7244a244-10af-4e0e-ac56-c715e79fd03f-rs-1.novalocal -cs_port 9445 -client_certdb_dir /tmp/tmp-gTN63u -client_certdb_pwd XXXXXXXX -preop_pin 8IhRgvChTfGUEzvFqteh -domain_name IPA -admin_user admin -admin_email root@localhost -admin_password XXXXXXXX -agent_name ipa-ca-agent -agent_key_size 2048 -agent_key_type rsa -agent_cert_subject CN=ipa-ca-agent,O=QE.PIT.REALM -ldap_host pit-scenario2-7244a244-10af-4e0e-ac56-c715e79fd03f-rs-1.novalocal -ldap_port 7389 -bind_dn cn=Directory Manager -bind_password XXXXXXXX -base_dn o=ipaca -db_name ipaca -key_size 2048 -key_type rsa -key_algorithm SHA256withRSA -save_p12 true -backup_pwd XXXXXXXX -subsystem_name pki-cad -token_name internal -ca_subsystem_cert_subject_name CN=CA Subsystem,O=QE.PIT.REALM -ca_subsystem_cert_subject_name CN=CA Subsystem,O=QE.PIT.REALM -ca_ocsp_cert_subject_name CN=OCSP Subsystem,O=QE.PIT.REALM -ca_server_cert_subject_name CN=pit-scenario2-7244a244-10af-4e0e-ac56-c715e79fd03f-rs-1.novalocal,O=QE.PIT.REALM -ca_audit_signing_cert_subject_name CN=CA Audit,O=QE.PIT.REALM -ca_sign_cert_subject_name CN=Certificate Authority,O=QE.PIT.REALM -external false -clone false' returned non-zero exit status 255 Configuration of CA failed Version-Release number of selected component (if applicable): ipa-server-3.0.0-39.el6.x86_64 pki-ca-9.0.3-32.el6.noarch How reproducible: Unknown. Steps to Reproduce: 1. ipa-server-install -r <REALM> -p Secret123 -a Secret123 -U Actual results: Failure listed above. Expected results: No failure and IPA installs cleanly. Additional info: Problem is we cannot reproduce this outside on an automated environment that generates and builds the VM it's failing on. If the VM is manually created (from the same image) it's not failing. I do not know the host systems in the two cases. ipaserver-install.log: 2014-06-27T19:42:10Z DEBUG Configuring certificate server (pki-cad): Estimated time 3 minutes 30 seconds 2014-06-27T19:42:10Z DEBUG [1/21]: creating certificate server user 2014-06-27T19:42:10Z DEBUG ca user pkiuser exists 2014-06-27T19:42:10Z DEBUG duration: 0 seconds 2014-06-27T19:42:10Z DEBUG [2/21]: creating pki-ca instance 2014-06-27T19:43:59Z DEBUG args=/usr/bin/pkicreate -pki_instance_root /var/lib -pki_instance_name pki-ca -subsystem_type ca -agent_secure_port 9443 -ee_secure_port 9444 -admin_secure_port 9445 -ee_secure_client_auth_port 9446 -unsecure_port 9180 -tomcat_server_port 9701 -redirect conf=/etc/pki-ca -redirect logs=/var/log/pki-ca -enable_proxy 2014-06-27T19:43:59Z DEBUG stdout=PKI instance creation Utility ... Capturing installation information in /var/log/pki-ca-install.log PKI instance creation completed ... Installation information recorded in /var/log/pki-ca-install.log. Before proceeding with the configuration, make sure the firewall settings of this machine permit proper access to this subsystem. Please start the configuration by accessing: https://pit-scenario2-7244a244-10af-4e0e-ac56-c715e79fd03f-rs-1.novalocal:9445/ca/admin/console/config/login?pin=pVNJx5tI2e9laIPwyV8w After configuration, the server can be operated by the command: /sbin/service pki-cad restart pki-ca 2014-06-27T19:43:59Z DEBUG stderr= 2014-06-27T19:43:59Z DEBUG duration: 109 seconds 2014-06-27T19:43:59Z DEBUG [3/21]: configuring certificate server instance 2014-06-27T19:44:00Z DEBUG args=/usr/bin/perl /usr/bin/pkisilent ConfigureCA -cs_hostname pit-scenario2-7244a244-10af-4e0e-ac56-c715e79fd03f-rs-1.novalocal -cs_port 9445 -client_certdb_dir /tmp/tmp-TpgvUw -client_certdb_pwd XXXXXXXX -preop_pin pVNJx5tI2e9laIPwyV8w -domain_name IPA -admin_user admin -admin_email root@localhost -admin_password XXXXXXXX -agent_name ipa-ca-agent -agent_key_size 2048 -agent_key_type rsa -agent_cert_subject CN=ipa-ca-agent,O=QE.PIT.REALM -ldap_host pit-scenario2-7244a244-10af-4e0e-ac56-c715e79fd03f-rs-1.novalocal -ldap_port 7389 -bind_dn cn=Directory Manager -bind_password XXXXXXXX -base_dn o=ipaca -db_name ipaca -key_size 2048 -key_type rsa -key_algorithm SHA256withRSA -save_p12 true -backup_pwd XXXXXXXX -subsystem_name pki-cad -token_name internal -ca_subsystem_cert_subject_name CN=CA Subsystem,O=QE.PIT.REALM -ca_subsystem_cert_subject_name CN=CA Subsystem,O=QE.PIT.REALM -ca_ocsp_cert_subject_name CN=OCSP Subsystem,O=QE.PIT.REALM -ca_server_cert_subject_name CN=pit-scenario2-7244a244-10af-4e0e-ac56-c715e79fd03f-rs-1.novalocal,O=QE.PIT.REALM -ca_audit_signing_cert_subject_name CN=CA Audit,O=QE.PIT.REALM -ca_sign_cert_subject_name CN=Certificate Authority,O=QE.PIT.REALM -external false -clone false 2014-06-27T19:44:00Z DEBUG stdout=libpath=/usr/lib64 ####################################################################### CRYPTO INIT WITH CERTDB:/tmp/tmp-TpgvUw tokenpwd:XXXXXXXX ############################################# Attempting to connect to: pit-scenario2-7244a244-10af-4e0e-ac56-c715e79fd03f-rs-1.novalocal:9445 Exception in LoginPanel(): java.lang.NullPointerException ERROR: ConfigureCA: LoginPanel() failure ERROR: unable to create CA ####################################################################### 2014-06-27T19:44:00Z DEBUG stderr=Exception: Unable to Send Request:java.io.IOException: java.io.IOException: SSL_ForceHandshake failed: (-5991) I/O function error. --> java.net.SocketException: Connection reset java.io.IOException: java.io.IOException: SSL_ForceHandshake failed: (-5991) I/O function error. --> java.net.SocketException: Connection reset at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.mozilla.jss.ssl.SocketBase.processExceptions(SocketBase.java:401) at org.mozilla.jss.ssl.SSLSocket.forceHandshake(Native Method) at HTTPClient.sslConnect(HTTPClient.java:331) at ConfigureCA.LoginPanel(ConfigureCA.java:244) at ConfigureCA.ConfigureCAInstance(ConfigureCA.java:1157) at ConfigureCA.main(ConfigureCA.java:1672) java.lang.NullPointerException at ConfigureCA.LoginPanel(ConfigureCA.java:245) at ConfigureCA.ConfigureCAInstance(ConfigureCA.java:1157) at ConfigureCA.main(ConfigureCA.java:1672) 2014-06-27T19:44:00Z CRITICAL failed to configure ca instance Command '/usr/bin/perl /usr/bin/pkisilent ConfigureCA -cs_hostname pit-scenario2-7244a244-10af-4e0e-ac56-c715e79fd03f-rs-1.novalocal -cs_port 9445 -client_certdb_dir /tmp/tmp-TpgvUw -client_certdb_pwd XXXXXXXX -preop_pin pVNJx5tI2e9laIPwyV8w -domain_name IPA -admin_user admin -admin_email root@localhost -admin_password XXXXXXXX -agent_name ipa-ca-agent -agent_key_size 2048 -agent_key_type rsa -agent_cert_subject CN=ipa-ca-agent,O=QE.PIT.REALM -ldap_host pit-scenario2-7244a244-10af-4e0e-ac56-c715e79fd03f-rs-1.novalocal -ldap_port 7389 -bind_dn cn=Directory Manager -bind_password XXXXXXXX -base_dn o=ipaca -db_name ipaca -key_size 2048 -key_type rsa -key_algorithm SHA256withRSA -save_p12 true -backup_pwd XXXXXXXX -subsystem_name pki-cad -token_name internal -ca_subsystem_cert_subject_name CN=CA Subsystem,O=QE.PIT.REALM -ca_subsystem_cert_subject_name CN=CA Subsystem,O=QE.PIT.REALM -ca_ocsp_cert_subject_name CN=OCSP Subsystem,O=QE.PIT.REALM -ca_server_cert_subject_name CN=pit-scenario2-7244a244-10af-4e0e-ac56-c715e79fd03f-rs-1.novalocal,O=QE.PIT.REALM -ca_audit_signing_cert_subject_name CN=CA Audit,O=QE.PIT.REALM -ca_sign_cert_subject_name CN=Certificate Authority,O=QE.PIT.REALM -external false -clone false' returned non-zero exit status 255 2014-06-27T19:44:00Z INFO File "/usr/lib/python2.6/site-packages/ipaserver/install/installutils.py", line 614, in run_script return_value = main_function() File "/usr/sbin/ipa-server-install", line 942, in main subject_base=options.subject) File "/usr/lib/python2.6/site-packages/ipaserver/install/cainstance.py", line 626, in configure_instance self.start_creation(runtime=210) File "/usr/lib/python2.6/site-packages/ipaserver/install/service.py", line 358, in start_creation method() File "/usr/lib/python2.6/site-packages/ipaserver/install/cainstance.py", line 888, in __configure_instance raise RuntimeError('Configuration of CA failed') 2014-06-27T19:44:00Z INFO The ipa-server-install command failed, exception: RuntimeError: Configuration of CA failed
Note, I got come help from dogtag guys troubleshooting this. Ade noticed that there wasn't a service check between 2/21 pkicreate and 3/21 pkisilent. A check to make sure the service is up and functioning may help here?
FYI, I did attempt a workaround like from bug#905064#c16: [root@pit1 install]# cat /usr/bin/pkicreate #!/bin/sh /usr/bin/pkicreate.real "$@" RET=$? sleep 100 exit $RET [root@pit1 install]# ls -l /usr/bin/pkicreate.real -rwxr-xr-x. 1 root root 150749 Sep 3 2013 /usr/bin/pkicreate.real [root@pit1 install]# It paused for the 100 seconds as you can see from the time but, it still failed. So, I'm not sure if there is a usable workaround here. Thoughts?
Ok, it looks like Bug 1112581 was really not caused by an interim package error. Ade, please help us with investigation, it looks like the problem is not in a wait for pkicreate (Comment 1).
The problem here is the length of the hostname. When we run pkicreate, we create a temporary cert in the certificate database to get SSL kickstarted. I looked on the system and noticed that the cert had not been generated (and therefore SSL connections failed). The command used to create the cert is something like the following: root@pit-scenario2-7244a244-10af-4e0e-ac56-c715e79fd03f-rs-1 ~]# certutil -S -d /tmp/foo -h 'internal' -m 0 -v 12 -x -s 'CN=pit-scenario2-7244a244-10af-4e0e-ac56-c715e79fd03f-rs-1.novalocal,O=2014-06-30 10:24:33' -c 'CN=pit-scenario2-7244a244-10af-4e0e-ac56-c715e79fd03f-rs-1.novalocal,O=2014-06-30 10:24:33' -t 'CTu,CTu,CTu' ... which results in: certutil -s: improperly formatted name: "CN=pit-scenario2-7244a244-10af-4e0e-ac56-c715e79fd03f-rs-1.novalocal,O=2014-06-30 10:24:33" The reason for this is that the CN exceeds 64 characters. According to RFC 5280, the CN cannot exceed 64 characters --> In Appendix A: -- Upper Bounds ub-common-name INTEGER ::= 64 Therefore, this is not a bug in Dogtag.
I also found that this is a limit for hostname max length: [root@pit-scenario2-7244a244-10af-4e0e-ac56-c715e79fd03f-rs-1 sysconfig]# getconf HOST_NAME_MAX 64 Since this is a known limit we're hitting for the CN for the cert, as well as a limit for hostname max length. So, I'm closing this one out as NOTABUG.
Would it make sense FreeIPA to check the hostname before the server/replica installer is run? This would require just one extra check to hostname validator. It may be beneficial, given how difficult it was to investigate the root cause.
Maybe. You're right. It would have helped here. So, reopening so installer can be updated to check hostname length. Thanks
Note that while CN is limited to 64 characters, it is not a host name but full FQDN that is passed there. In this particular case hostname is 54 character long but FQDN is 65 characters long, thus it fails. So a check would need to take FQDN and ensure that it is less than 64 characters. Now, this will be a sizable limitation as DNS entries can have 64 character *per label*, not per whole FQDN.
I think we may be using RFC5280 wrongly here. It states (and RFC 6818 which is an update to RFC 5280 doesn't change this dramatically): 4.1.2.4 [..] In addition, implementations of this specification MUST be prepared to receive the domainComponent attribute, as defined in [RFC4519]. The Domain Name System (DNS) provides a hierarchical resource labeling system. This attribute provides a convenient mechanism for organizations that wish to use DNs that parallel their DNS names. I think we should be packing FQDN into domainComponent attribute and use just a host name part for CN.
Upstream ticket: https://fedorahosted.org/freeipa/ticket/4415
Looking at nss code, I can see that with the help of bug https://bugzilla.mozilla.org/show_bug.cgi?id=970539 there is a change in upstream libnss to allow CN attribute value to be up to 640 bytes instead of 64. The code otherwise doesn't put any limitation on the CN value.
Thank you taking your time and submitting this request for Red Hat Enterprise Linux. Unfortunately, this bug was not given a priority and was deferred both in the upstream project and in Red Hat Enterprise Linux. Given that we are unable to fulfill this request in following Red Hat Enterprise Linux releases, I am closing the Bugzilla as WONTFIX. To request that Red Hat re-considers the decision, please re-open the Bugzilla via appropriate support channels and provide additional business and/or technical details about its importance to you. Note that you can still track this request or even contribute patches in the referred upstream Trac ticket.