Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1533497

Summary:	Replica installation using setup-ca option fails at domain-level 0 against upgraded ipa-server
Product:	Red Hat Enterprise Linux 7	Reporter:	Nikhil Dehadrai <ndehadra>
Component:	ipa	Assignee:	IPA Maintainers <ipa-maint>
Status:	CLOSED DUPLICATE	QA Contact:	ipa-qe <ipa-qe>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	7.5	CC:	amarecek, ndehadra, pasik, pvoborni, rcritten, tbordaz, tscherf
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-02-13 19:48:57 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Nikhil Dehadrai 2018-01-11 14:04:35 UTC

Description of problem:
Replica installation using setup-ca option fails at domain-level 0 against upgraded ipa-server

Version-Release number of selected component (if applicable):
ipa-server-4.5.4-8.el7

How reproducible:
Always

Steps to Reproduce:
1. Setup IPA-master upgraded from RHEL 70 to RHEL 75 (linear upgrade path: 70z > 71z >72z >73z > 740-day > 74z > 75).
2. Once the upgrade process is completed successfully, run 'ipa domainlevel-get' command.

[root@ipaqavmh ipa]# ipa domainlevel-get
-----------------------
Current domain level: 0
-----------------------


3. Configure replica gainst this upgraded IPA-master in step1.
# ipa-replica-install -U --setup-dns --forwarder <forwarder_ip> --allow-zone-overlap --setup-ca --admin-password <admin-password> --password <password> replica-system.gpg

Actual results:
After step3, On Replica noticed that replica installation fails at step: 

Done configuring ipa-custodia.
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/28]: configuring certificate server instance
ipa.ipaserver.install.cainstance.CAInstance: CRITICAL Failed to configure CA instance: Command '/usr/sbin/pkispawn -s CA -f /tmp/tmp0cfLnd' returned non-zero exit status 1
ipa.ipaserver.install.cainstance.CAInstance: CRITICAL See the installation logs and the following files/directories for more information:
ipa.ipaserver.install.cainstance.CAInstance: CRITICAL   /var/log/pki/pki-tomcat
  [error] RuntimeError: CA configuration failed.
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

ipa.ipapython.install.cli.install_tool(CompatServerReplicaInstall): ERROR    CA configuration failed.
ipa.ipapython.install.cli.install_tool(CompatServerReplicaInstall): ERROR    The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information


# tail output for /var/log/ipareplica-install.log 

2018-01-11T13:37:18Z CRITICAL Failed to configure CA instance: Command '/usr/sbin/pkispawn -s CA -f /tmp/tmp0cfLnd' returned non-zero exit status 1
2018-01-11T13:37:18Z CRITICAL See the installation logs and the following files/directories for more information:
2018-01-11T13:37:18Z CRITICAL   /var/log/pki/pki-tomcat
2018-01-11T13:37:18Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 504, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 494, in run_step
    method()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 615, in __spawn_instance
    self.tmp_agent_pwd)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/dogtaginstance.py", line 148, in spawn_instance
    self.handle_setup_error(e)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/dogtaginstance.py", line 386, in handle_setup_error
    raise RuntimeError("%s configuration failed." % self.subsystem)
RuntimeError: CA configuration failed.

2018-01-11T13:37:18Z DEBUG   [error] RuntimeError: CA configuration failed.
2018-01-11T13:37:18Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 333, in run
    cfgr.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 368, in run
    self.execute()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 392, in execute
    for _nothing in self._executor():
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 434, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 463, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 453, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 424, in __runner
    step()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 658, in _configure
    next(executor)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 434, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 463, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 521, in _handle_exception
    self.__parent._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 453, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 518, in _handle_exception
    super(ComponentBase, self)._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 453, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 424, in __runner
    step()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python2.7/site-packages/ipapython/install/common.py", line 63, in _install
    for _nothing in self._installer(self.parent):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/__init__.py", line 617, in main
    replica_install(self)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 386, in decorated
    func(installer)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 1458, in install
    ca.install(False, config, options)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ca.py", line 205, in install
    install_step_0(standalone, replica_config, options)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ca.py", line 284, in install_step_0
    use_ldaps=standalone)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 447, in configure_instance
    self.start_creation(runtime=runtime)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 504, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 494, in run_step
    method()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 615, in __spawn_instance
    self.tmp_agent_pwd)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/dogtaginstance.py", line 148, in spawn_instance
    self.handle_setup_error(e)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/dogtaginstance.py", line 386, in handle_setup_error
    raise RuntimeError("%s configuration failed." % self.subsystem)

2018-01-11T13:37:18Z DEBUG The ipa-replica-install command failed, exception: RuntimeError: CA configuration failed.
2018-01-11T13:37:18Z ERROR CA configuration failed.
2018-01-11T13:37:18Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information


Expected results:
Replica installation with setup-ca option should be successful.

Additional info:
When setup-ca option is not used, replica installation is successful

Comment 3 Rob Crittenden 2018-01-11 14:41:04 UTC

It failed in setting up replication:

setupReplication: java.io.IOException: consumer initialization failed. -11  - LDAP error: Connect error

From the dirsrv logs:

ERR - slapi_ldap_bind - Error: could not send startTLS request: error -11 (Connect error)

Are you sure there was connectivity between the hosts?

Comment 4 Nikhil Dehadrai 2018-01-12 06:19:22 UTC

(In reply to Rob Crittenden from comment #3)
> It failed in setting up replication:
> 
> setupReplication: java.io.IOException: consumer initialization failed. -11 
> - LDAP error: Connect error
> 
> From the dirsrv logs:
> 
> ERR - slapi_ldap_bind - Error: could not send startTLS request: error -11
> (Connect error)
> 
> Are you sure there was connectivity between the hosts?


Yes the machine (replica) seems to connect. 

[root@vm-idm-034 log]# cat ipareplica-conncheck.log | grep "Connection from"
Connection from replica to master is OK.
2018-01-11T13:32:56Z INFO Connection from master to replica is OK.

The bug was logged as separate issue identified during verification of bug bz#1500218

Comment 8 Rob Crittenden 2018-01-22 20:17:11 UTC

It feels like an SSL trust issue to me.

I confirmed connectivity is ok using ldapsearch but I had to pass in LDAPTLS_CACERT=/etc/ipa/ca.cert for it to work with -Z

Without it I get the same error seen in the dogtag install log

# ldapsearch -x -Z -H ldap://foo.bar.com -s base
ldap_start_tls: Connect error (-11)
        additional info: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed (self signed certificate in certificate chain)
ldap_result: Can't contact LDAP server (-1)

I noticed that even on the master there is no ipa-p11.kit in the global trust which I find strange, but I don't think it is related.

I ran ipa-certupdate on this failed install and subsequently ldapsearch works.

I'm not sure what is or could be missing between this vanilla 4.5 install when installing against an upgraded IPA master unless there is some assumption in the replica install that system-wide CA trust is already setup (ala ipa-client-install).