Bug 1374054

Summary: ipa-replica-install fails setting up certificate server
Product: Red Hat Enterprise Linux 7 Reporter: Scott Poore <spoore>
Component: pki-coreAssignee: RHCS Maintainers <rhcs-maint>
Status: CLOSED ERRATA QA Contact: Asha Akkiangady <aakkiang>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 7.3CC: edewata, frenaud, jcholast, jpazdziora, mharmsen, pvoborni, rcritten
Target Milestone: rcKeywords: Regression, TestBlocker
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: pki-core-10.3.3-10.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-09-10 06:31:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Scott Poore 2016-09-07 20:12:54 UTC
Description of problem:

I'm seeing ipa-replica-install fail when setting up certificate server:

Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes 30 seconds
  [1/25]: creating certificate server user
  [2/25]: creating certificate server db
  [3/25]: setting up initial replication
Starting replication, please wait until this has completed.

Update in progress, 1 seconds elapsed
Update in progress, 2 seconds elapsed
Update in progress, 3 seconds elapsed
Update in progress, 4 seconds elapsed
Update succeeded

  [4/25]: creating installation admin user
  [5/25]: setting up certificate server
ipa.ipaserver.install.cainstance.CAInstance: CRITICAL Failed to configure CA instance: Command '/usr/sbin/pkispawn -s CA -f /tmp/tmphC0h3E' returned non-zero exit status 1
ipa.ipaserver.install.cainstance.CAInstance: CRITICAL See the installation logs and the following files/directories for more information:
ipa.ipaserver.install.cainstance.CAInstance: CRITICAL   /var/log/pki/pki-tomcat
  [error] RuntimeError: CA configuration failed.
ipa.ipapython.install.cli.install_tool(Replica): ERROR    CA configuration failed.
ipa.ipapython.install.cli.install_tool(Replica): ERROR    The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

Version-Release number of selected component (if applicable):
ipa-server-4.4.0-9.el7.x86_64
pki-ca-10.3.3-9.el7.noarch


How reproducible:
unknown.  seeing failures in automated tests

Steps to Reproduce:
1.   install ipa server with dns configured
2.   point resolv.conf on replica to master
3.   ipa-replica-install with --setup-ca --setup-dns --forwarder=<forwarder> --ip-address=<hostip> --principal=admin --admin-password=<admin pass>


Actual results:
Failing as shown above

Expected results:
no failure

Additional info:

Comment 4 Petr Vobornik 2016-09-08 10:46:29 UTC
from PKI ca debug log:

[07/Sep/2016:13:57:59][http-bio-8443-exec-3]: Established LDAP connection using basic authentication to host ibm-x3250m4-02.testrelm.test port 389 as cn=Directory Manager
[07/Sep/2016:13:57:59][http-bio-8443-exec-3]: initializing with mininum 3 and maximum 15 connections to host ibm-x3250m4-02.testrelm.test port 389, secure connection, false, authentication type 1
[07/Sep/2016:13:57:59][http-bio-8443-exec-3]: increasing minimum connections by 3
[07/Sep/2016:13:57:59][http-bio-8443-exec-3]: new total available connections 3
[07/Sep/2016:13:57:59][http-bio-8443-exec-3]: new number of connections 3
[07/Sep/2016:13:57:59][http-bio-8443-exec-3]: SystemConfigService:processCerts(): san_server_cert not found for tag sslserver
org.mozilla.jss.NoSuchTokenException
	at org.mozilla.jss.CryptoManager.getTokenByName(CryptoManager.java:622)
	at com.netscape.cmsutil.crypto.CryptoUtil.getTokenByName(CryptoUtil.java:487)
	at com.netscape.cmsutil.crypto.CryptoUtil.generateRSAKeyPair(CryptoUtil.java:500)
	at com.netscape.cms.servlet.csadmin.ConfigurationUtils.createRSAKeyPair(ConfigurationUtils.java:2450)
	at org.dogtagpki.server.rest.SystemConfigService.processCert(SystemConfigService.java:467)
	at org.dogtagpki.server.rest.SystemConfigService.processCerts(SystemConfigService.java:387)
	at org.dogtagpki.server.rest.SystemConfigService.configure(SystemConfigService.java:187)
	at org.dogtagpki.server.rest.SystemConfigService.configure(SystemConfigService.java:121)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:280)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:234)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:221)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)
	at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
	at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
	at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
	at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:297)
	at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:55)
	at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:191)
	at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:187)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:186)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
	at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
	at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
	at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:260)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237)
	at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:55)
	at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:191)
	at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:187)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:186)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:745)
[07/Sep/2016:13:57:59][http-bio-8443-exec-3]: Error in setting certificate names and key sizes: org.mozilla.jss.NoSuchTokenException

Comment 5 Petr Vobornik 2016-09-08 10:52:03 UTC
Moving it to PKI component. Given that the exception is caught in "catch-all block" which indicates general, unexpected issue:

From SystemConfigService.java:409:
        } catch (Exception e) {
            CMS.debug(e);
            throw new PKIException("Error in setting certificate names and key sizes: " + e);
        }

Comment 7 Endi Sukma Dewata 2016-09-08 15:57:44 UTC
It looks like a recent bug fix uncovered an existing issue (i.e. token name not normalized) causing the NoSuchTokenException under certain cases. The basic CA installation worked just fine.

Comment 8 Endi Sukma Dewata 2016-09-08 19:23:06 UTC
The changes in bug #1372041 that causes the problem has now been reverted.