Bug 1650155

Summary: [DOC] IPA Replica installation fails as part of IPA-mIgration process
Product: Red Hat Enterprise Linux 7 Reporter: Nikhil Dehadrai <ndehadra>
Component: pki-coreAssignee: Marc Muehlfeld <mmuehlfe>
Status: CLOSED CURRENTRELEASE QA Contact: Asha Akkiangady <aakkiang>
Severity: unspecified Docs Contact: Marc Muehlfeld <mmuehlfe>
Priority: unspecified    
Version: 7.6CC: abokovoy, cfu, cpelland, edewata, ksiddiqu, mharmsen, mkosek, ndehadra, pvoborni, rcritten, tscherf
Target Milestone: pre-dev-freezeKeywords: Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
Installing an IdM replica on RHEL 7.6 fails if the IdM master runs on RHEL 6 With the update of the _pki-core_ package provided in Red Hat Enterprise Linux (RHEL) 7.6, certain ciphers that are not supported by hardware security modules (HSM) are no longer enabled by default in Certificate System. As a consequence, setting up Identity Management (IdM) on RHEL 7.6 as a replica with a master running on RHEL 6 fails with a "CRITICAL Failed to configure CA instance" error. To work around the problem, append the following entry to the end of the "NSSCipherSuite" parameter in the `/etc/httpd/conf.d/nss.conf` file: +ecdhe_rsa_aes_128_sha,+ecdhe_rsa_aes_256_sha As a result, the IdM installation on RHEL 7.6 no longer fails.
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-02-28 12:57:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1672180    

Description Nikhil Dehadrai 2018-11-15 13:14:19 UTC
Description of problem:
Replica installation fails against DL0 IPA master, causing the IPA-migration test suite to fail.

Version-Release number of selected component (if applicable):
RHEL 7.6up1

How reproducible:
Always

Steps to Reproduce:
1. Setup IPA master on RHEL 6.10
2. Setup IPA-REPLICA on RHEL 7.6up1
# prepare gpg file for replica on master
# copy gpg filr to replica
# run ipa-replica install comand using 
/usr/sbin/ipa-replica-install -U --setup-ca --setup-dns --forwarder=10.x.x.x -w Secret123 -p Secret123 /opt/rhqa_ipa/replica-info-intel-piketon-02.testrelm.test.gpg


Actual results:
Replica install fails

:: [ 02:35:41 ] :: [  BEGIN   ] :: Running ' /usr/sbin/ipa-replica-install -U --setup-ca --setup-dns --forwarder=10.x.x.x -w Secret123 -p Secret123 /opt/rhqa_ipa/replica-info-intel-piketon-02.testrelm.test.gpg'
..
..
..
..
..
Configuring ipa-custodia
  [1/5]: Making sure custodia container exists
  [2/5]: Generating ipa-custodia config file
  [3/5]: Generating ipa-custodia keys
  [4/5]: starting ipa-custodia 
  [5/5]: configuring ipa-custodia to start on boot
Done configuring ipa-custodia.
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/28]: configuring certificate server instance
ipaserver.install.dogtaginstance: CRITICAL Failed to configure CA instance: Command '/usr/sbin/pkispawn -s CA -f /tmp/tmpxQv9bc' returned non-zero exit status 1
ipaserver.install.dogtaginstance: CRITICAL See the installation logs and the following files/directories for more information:
ipaserver.install.dogtaginstance: CRITICAL   /var/log/pki/pki-tomcat
  [error] RuntimeError: CA configuration failed.
ipapython.admintool: ERROR    CA configuration failed.
ipapython.admintool: ERROR    The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.


Expected results:
Ipa replica installation against DL0 master should be successful, as a result ipa-migration should be successful

Comment 5 Alexander Bokovoy 2018-11-15 16:38:57 UTC
It looks like the failure is at pki side where a request to create a server certificate fails due to inability to write to SSL socket internally:

[15/Nov/2018:06:21:32][http-bio-8443-exec-3]: SystemConfigService: request: ConfigurationRequest [pin=XXXX, token=Internal Key Storage Token, tokenPassword=XXXX, securityDomainType=existingdomain, securityDomainUri=https://ibm-x3500m4-01.testrelm.test:443, securityDomainName=null, securityDomainUser=admin, securityDomainPassword=XXXX, securityDomainPostLoginSleepSeconds=null, isClone=true, cloneUri=https://ibm-x3500m4-01.testrelm.test:443, subsystemName=CA mgmt3.testrelm.test 8443, p12File=/tmp/ca.p12, p12Password=XXXX, hierarchy=root, dsHost=mgmt3.testrelm.test, dsPort=389, baseDN=o=ipaca, bindDN=cn=Directory Manager, bindpwd=XXXX, database=ipaca, secureConn=false, removeData=true, replicateSchema=false, masterReplicationPort=7389, cloneReplicationPort=389, replicationSecurity=TLS, systemCertsImported=false, systemCerts=[com.netscape.certsrv.system.SystemCertData@2977e3a5], issuingCA=https://ibm-x3500m4-01.testrelm.test:443, backupKeys=true, backupPassword=XXXX, backupFile=/etc/pki/pki-tomcat/alias/ca_backup_keys.p12, adminUID=null, adminPassword=XXXX, adminEmail=null, adminCertRequest=null, adminCertRequestType=null, adminSubjectDN=null, adminName=null, adminProfileID=null, adminCert=null, importAdminCert=false, generateServerCert=true, external=false, standAlone=false, authdbBaseDN=null, authdbHost=null, authdbPort=null, authdbSecureConn=null, caUri=null, kraUri=null, tksUri=null, enableServerSideKeyGen=null, importSharedSecret=null, generateSubsystemCert=null, sharedDB=false, sharedDBUserDN=null, createNewDB=true, setupReplication=True, subordinateSecurityDomainName=null, reindexData=False, startingCrlNumber=0, createSigningCertRecord=true, signingCertSerialNumber=1]
[15/Nov/2018:06:21:32][http-bio-8443-exec-3]: === Token Authentication ===
[15/Nov/2018:06:21:32][http-bio-8443-exec-3]: === Security Domain Configuration ===
[15/Nov/2018:06:21:32][http-bio-8443-exec-3]: Joining existing security domain
[15/Nov/2018:06:21:32][http-bio-8443-exec-3]: Resolving security domain URL https://ibm-x3500m4-01.testrelm.test:443
[15/Nov/2018:06:21:32][http-bio-8443-exec-3]: Getting security domain cert chain
[15/Nov/2018:06:21:32][http-bio-8443-exec-3]: ConfigurationUtils.importCertChain()
[15/Nov/2018:06:21:33][http-bio-8443-exec-3]: ConfigurationUtils: GET https://ibm-x3500m4-01.testrelm.test:443/ca/admin/ca/getCertChain
javax.ws.rs.ProcessingException: Unable to invoke request
	at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:287)
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:407)
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:442)
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocationBuilder.get(ClientInvocationBuilder.java:165)
	at com.netscape.certsrv.client.PKIConnection.get(PKIConnection.java:500)
	at com.netscape.cms.servlet.csadmin.ConfigurationUtils.get(ConfigurationUtils.java:259)
	at com.netscape.cms.servlet.csadmin.ConfigurationUtils.importCertChain(ConfigurationUtils.java:289)
	at org.dogtagpki.server.rest.SystemConfigService.logIntoSecurityDomain(SystemConfigService.java:918)
	at org.dogtagpki.server.rest.SystemConfigService.configureSecurityDomain(SystemConfigService.java:879)
	at org.dogtagpki.server.rest.SystemConfigService.configure(SystemConfigService.java:140)
	at org.dogtagpki.server.rest.SystemConfigService.configure(SystemConfigService.java:101)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:280)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:234)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:221)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)
	at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
	at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
	at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
	at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:297)
	at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:55)
	at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:191)
	at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:187)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:186)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
	at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
	at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
	at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:260)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237)
	at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:55)
	at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:191)
	at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:187)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:186)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: SocketException cannot write on socket
	at org.mozilla.jss.ssl.SSLSocket.write(SSLSocket.java:1503)
	at org.mozilla.jss.ssl.SSLOutputStream.write(SSLOutputStream.java:24)
	at org.apache.http.impl.io.AbstractSessionOutputBuffer.flushBuffer(AbstractSessionOutputBuffer.java:147)
	at org.apache.http.impl.io.AbstractSessionOutputBuffer.flush(AbstractSessionOutputBuffer.java:154)
	at org.apache.http.impl.AbstractHttpClientConnection.doFlush(AbstractHttpClientConnection.java:278)
	at org.apache.http.impl.AbstractHttpClientConnection.flush(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.flush(ManagedClientConnectionImpl.java:175)
	at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:260)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
	at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:283)
	... 72 more
[15/Nov/2018:06:21:33][http-bio-8443-exec-3]: SignedAuditLogger: event ACCESS_SESSION_TERMINATED

Comment 6 Endi Sukma Dewata 2018-11-15 19:19:17 UTC
This is a very generic SSL failure without any indication of what's causing it.

Has any other replication test been done using the same RHEL 7.6up1 build? Are they failing as well with the same error message?

Could you manually check if the proxy URL is accessible using the replica's NSS database (i.e. /etc/pki/pki-tomcat/alias)?
https://ibm-x3500m4-01.testrelm.test:443/ca/admin/ca/getCertChain

Could you perform the same test against the Tomcat URL directly?
https://ibm-x3500m4-01.testrelm.test:8443/ca/admin/ca/getCertChain

Please list the certs and keys in the master's NSS database and in the replica's NSS database. Thanks!

Comment 7 Endi Sukma Dewata 2018-11-15 19:23:19 UTC
Also, does the master has any logs (e.g. access log) associated with the failed operation? Does it show the reason for the SSL connection failure?

Comment 9 Endi Sukma Dewata 2018-11-15 21:46:48 UTC
Per cfu's request, could you also provide the cipher settings in the server.xml of both master and replica (i.e. strictCiphers, sslVersionRangeStream, sslVersionRangeDatagram, sslRangeCiphers)? Thanks.

Comment 22 Christina Fu 2018-11-19 18:24:53 UTC
first, fyi, the TLS_RSA_* ciphers were removed due to
https://bugzilla.redhat.com/show_bug.cgi?id=1578389
where after much investigation by both CS and NSS teams, it was determined that those ciphers are not supported by HSM in the FIPS environment (will cause BAD_RECORD_MAC error).
Also according to Bob Relyea, "We can basically say we don't support  RSA_ algorithms in FIPS mode with a HSM.
   - This may be fine because the RSA_ algorithms are deprecated in TLS 1.3 anyway (and currently not preferred in TLS 1.2)."
They are hence removed from the default list of ciphers.

As to the remedy for the cross platform issue, I'd prefer not making permanent changes to forward releases due to reason stated above. If temporary changes are acceptable, that'd be more ideal.

Comment 25 Rob Crittenden 2018-11-20 12:34:09 UTC
Based on the ciphers in #10 the following need to be added to the end of NSSCipherSuite line in /etc/httpd/conf.d/nss.conf:

+ecdhe_rsa_aes_128_sha,+ecdhe_rsa_aes_256_sha

This enables TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA and TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA which appear to be the only overlapping ciphers.

It worked in my test. Nikhil, can you confirm?

Comment 26 Matthew Harmsen 2018-11-26 17:44:02 UTC
As discussed in comments above, this bug is being re-assigned to the RHEL 6 ipa component for configuration changes to allow migration.

The test listed above still needs confirmation.

Comment 43 Endi Sukma Dewata 2019-03-11 23:36:23 UTC
*** Bug 1667434 has been marked as a duplicate of this bug. ***