Bug 1311280

Summary:	ipa-server-install Configuration of CA failed with crash in pki-tomcat signalHandler
Product:	Red Hat Enterprise Linux 7	Reporter:	Scott Poore <spoore>
Component:	pki-core	Assignee:	Matthew Harmsen <mharmsen>
Status:	CLOSED WORKSFORME	QA Contact:	Asha Akkiangady <aakkiang>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	7.2	CC:	kbanerje, mharmsen, mmuehlfe, nsoman, pvoborni, rcritten
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:	Cause: Consequence: Workaround (if any) Result:	Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-03-10 15:55:01 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Scott Poore 2016-02-23 19:56:31 UTC

Description of problem:

We see ipa-server-install fail and crash here:

ipa.ipaserver.install.cainstance.CAInstance: CRITICAL Failed to configure CA instance: Command ''/usr/sbin/pkispawn' '-s' 'CA' '-f' '/tmp/tmpeEIxcO'' returned non-zero exit status 1


This generates abrt report email:

time:           Tue 23 Feb 2016 01:46:59 PM EST
cmdline:        /usr/lib/jvm/jre/bin/java -DRESTEASY_LIB=/usr/share/java/resteasy-base -Djava.library.path=/usr/lib64/nuxwdog-jni -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.manager -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy org.apache.catalina.startup.Bootstrap start
uid:            17 (pkiuser)
abrt_version:   2.1.11
backtrace_rating: 3
crash_function: signalHandler
event_log:      
executable:     /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.65-3.b17.el7.x86_64/jre/bin/java
global_pid:     15505
hostname:       auto-hv-02-guest07.testrelm.test
kernel:         3.10.0-327.el7.x86_64
last_occurrence: 1456253219
pid:            15505
pkg_arch:       x86_64
pkg_epoch:      1
pkg_name:       java-1.8.0-openjdk-headless
pkg_release:    3.b17.el7
pkg_version:    1.8.0.65
pwd:            /usr/share/tomcat
runlevel:       N 3
username:       pkiuser

sosreport.tar.xz: Binary file, 5747972 bytes


Version-Release number of selected component (if applicable):

[root@auto-hv-02-guest07 pki]# rpm -q ipa-server pki-ca tomcat java-1.8.0-openjdk-headless
ipa-server-4.2.0-15.el7_2.7.x86_64
pki-ca-10.2.5-7.el7_2.noarch
tomcat-7.0.54-2.el7_1.noarch
java-1.8.0-openjdk-headless-1.8.0.65-3.b17.el7.x86_64

How reproducible:
Unknown

Steps to Reproduce:
1.  yum install ipa-server bind-dyndb-ldap
2.  ipa-server-install --setup-dns --forwarder=$DNSFORWARDER --hostname=$HOSTNAME --ip-address=$IPADDRESS -n $DNSDOMAIN -r $REALM -p $ADMINPW -a $ADMINPW -U

Actual results:

Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes 30 seconds
  [1/27]: creating certificate server user
  [2/27]: configuring certificate server instance

MARK-LWD-LOOP -- 2016-02-23 13:47:39 --
ipa.ipaserver.install.cainstance.CAInstance: CRITICAL Failed to configure CA instance: Command ''/usr/sbin/pkispawn' '-s' 'CA' '-f' '/tmp/tmpeEIxcO'' returned non-zero exit status 1
ipa.ipaserver.install.cainstance.CAInstance: CRITICAL See the installation logs and the following files/directories for more information:
ipa.ipaserver.install.cainstance.CAInstance: CRITICAL   /var/log/pki-ca-install.log
ipa.ipaserver.install.cainstance.CAInstance: CRITICAL   /var/log/pki/pki-tomcat
  [error] RuntimeError: CA configuration failed.
ipa.ipapython.install.cli.install_tool(Server): ERROR    CA configuration failed.
:: [   FAIL   ] :: Command ' /usr/sbin/ipa-server-install --setup-dns --forwarder=<DNS_FORWARDER> --hostname=auto-hv-02-guest07.testrelm.test -r TESTRELM.TEST -n testrelm.test -p Secret123 -a Secret123 --ip-address=<IP_ADDRESS> -U' (Expected 0, got 1)

Expected results:

no crash or failure

Additional info:

Comment 9 Petr Vobornik 2016-02-24 10:22:16 UTC

I fails quite early in pkispawn. Seems to me as PKI error, changing component.

From PKI logs:

catalina.out:
INFO: Starting ProtocolHandler ["ajp-bio-127.0.0.1-8009"]
Feb 23, 2016 1:46:59 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 6426 ms
Feb 23, 2016 1:47:03 PM org.apache.catalina.startup.Catalina stopServer
SEVERE: Could not contact localhost:8005. Tomcat may not be running.
Feb 23, 2016 1:47:03 PM org.apache.catalina.startup.Catalina stopServer
SEVERE: Catalina.stop: 
java.net.ConnectException: Connection refused
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at java.net.Socket.connect(Socket.java:538)
	at java.net.Socket.<init>(Socket.java:434)
	at java.net.Socket.<init>(Socket.java:211)
	at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:498)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:370)
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:457)


I don't see any error in debug.log

pki-ca-spawn.log:

2016-02-23 13:46:51 pkispawn    : INFO     ....... modifying '/root/.dogtag/pki-tomcat/ca/pkcs12_password.conf'
2016-02-23 13:46:51 pkispawn    : DEBUG    ........... chmod 660 /root/.dogtag/pki-tomcat/ca/pkcs12_password.conf
2016-02-23 13:46:51 pkispawn    : DEBUG    ........... chown 17:17 /root/.dogtag/pki-tomcat/ca/pkcs12_password.conf
2016-02-23 13:46:51 pkispawn    : INFO     ....... executing 'certutil -N -d /tmp/tmp-p7htha -f /root/.dogtag/pki-tomcat/ca/password.conf'
2016-02-23 13:46:51 pkispawn    : INFO     ....... executing 'systemctl daemon-reload'
2016-02-23 13:46:51 pkispawn    : INFO     ....... executing 'systemctl start pki-tomcatd'
2016-02-23 13:46:51 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:46:51 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:46:52 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:46:52 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:03 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:03 pkispawn    : DEBUG    ........... No connection - exception thrown: EOF occurred in violation of protocol (_ssl.c:765)
2016-02-23 13:47:04 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:04 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:05 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:05 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:06 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:06 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:07 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:07 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:08 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:08 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:09 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:09 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:10 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:10 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:11 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:11 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:12 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:12 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:13 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:13 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:14 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:14 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:15 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:15 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:16 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:16 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:17 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:17 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:18 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:18 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:19 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:19 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:20 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:20 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:21 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:21 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:22 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:22 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:23 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:23 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:24 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:24 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:25 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:25 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:26 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:26 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:27 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:27 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:28 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:28 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:29 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:29 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:30 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:30 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:31 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:31 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:32 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:32 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:33 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:33 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:34 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:34 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:35 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:35 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:36 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:36 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:37 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:37 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:38 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:38 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:39 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:39 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:40 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:40 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:41 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:41 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:42 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:42 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:43 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:43 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:44 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:44 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:45 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:45 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:46 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:46 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:47 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:47 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:48 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:48 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:49 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:49 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:50 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:50 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:51 pkispawn    : DEBUG    ........... No connection - server may still be down
2016-02-23 13:47:51 pkispawn    : DEBUG    ........... No connection - exception thrown: ('Connection aborted.', error(111, 'Connection refused'))
2016-02-23 13:47:52 pkispawn    : ERROR    ....... server failed to restart
2016-02-23 13:47:52 pkispawn    : DEBUG    ....... Error Type: Exception
2016-02-23 13:47:52 pkispawn    : DEBUG    ....... Error Message: server failed to restart
2016-02-23 13:47:52 pkispawn    : DEBUG    .......   File "/usr/sbin/pkispawn", line 597, in main
    rv = instance.spawn(deployer)
  File "/usr/lib/python2.7/site-packages/pki/server/deployment/scriptlets/configuration.py", line 232, in spawn
    raise Exception("server failed to restart")

Comment 11 Scott Poore 2016-02-24 17:15:10 UTC

FYI, this is the pkispawn config used in ipa install (from log):

2016-02-23T18:46:50Z DEBUG Contents of pkispawn configuration file (/tmp/tmpeEIxcO):
[CA]
pki_security_domain_name = IPA
pki_enable_proxy = True
pki_restart_configured_instance = False
pki_backup_keys = True
pki_backup_password = XXXXXXXX
pki_profiles_in_ldap = True
pki_client_database_dir = /tmp/tmp-p7htha
pki_client_database_password = XXXXXXXX
pki_client_database_purge = False
pki_client_pkcs12_password = XXXXXXXX
pki_admin_name = admin
pki_admin_uid = admin
pki_admin_email = root@localhost
pki_admin_password = XXXXXXXX
pki_admin_nickname = ipa-ca-agent
pki_admin_subject_dn = cn=ipa-ca-agent,O=TESTRELM.TEST
pki_client_admin_cert_p12 = /root/ca-agent.p12
pki_ds_ldap_port = 389
pki_ds_password = XXXXXXXX
pki_ds_base_dn = o=ipaca
pki_ds_database = ipaca
pki_subsystem_subject_dn = cn=CA Subsystem,O=TESTRELM.TEST
pki_ocsp_signing_subject_dn = cn=OCSP Subsystem,O=TESTRELM.TEST
pki_ssl_server_subject_dn = cn=auto-hv-02-guest07.testrelm.test,O=TESTRELM.TEST
pki_audit_signing_subject_dn = cn=CA Audit,O=TESTRELM.TEST
pki_ca_signing_subject_dn = cn=Certificate Authority,O=TESTRELM.TEST
pki_subsystem_nickname = subsystemCert cert-pki-ca
pki_ocsp_signing_nickname = ocspSigningCert cert-pki-ca
pki_ssl_server_nickname = Server-Cert cert-pki-ca
pki_audit_signing_nickname = auditSigningCert cert-pki-ca
pki_ca_signing_nickname = caSigningCert cert-pki-ca
pki_ca_signing_key_algorithm = SHA256withRSA


2016-02-23T18:46:50Z DEBUG Starting external process
2016-02-23T18:46:50Z DEBUG args='/usr/sbin/pkispawn' '-s' 'CA' '-f' '/tmp/tmpeEIxcO'
2016-02-23T18:47:52Z DEBUG Process finished, return code=1

Comment 15 Matthew Harmsen 2016-03-08 00:22:40 UTC

Upstream ticket:
https://fedorahosted.org/pki/ticket/2232

Comment 16 Scott Poore 2016-03-10 15:55:01 UTC

Closing this as WORKSFORME because it was discovered that I was using an incorrect build of nss that is not to be released.  When I point to a standard yum repo with the proper builds I do not see this crash.