Bug 1500218

Summary: Replica installation at domain-level 0 fails against upgraded ipa-server
Product: Red Hat Enterprise Linux 7 Reporter: Nikhil Dehadrai <ndehadra>
Component: ipaAssignee: IPA Maintainers <ipa-maint>
Status: CLOSED ERRATA QA Contact: ipa-qe <ipa-qe>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.4CC: frenaud, nsoman, pasik, pvoborni, rcritten, tscherf
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ipa-4.5.4-4.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-10 16:48:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Nikhil Dehadrai 2017-10-10 08:34:43 UTC
Description of problem:
Replica installation at domain-level 0 fails against upgraded ipa-server from RHEL 70 to RHEL 7.4 update2

Version-Release number of selected component (if applicable):
ipa-server-4.5.0-21.el7_4.2.2.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Setup IPA server at RHEL 70
2. Upgrade this IPA master linearly to RHEL 7.4update2 
( In my case upgrade path is: RHEL70 > RHEL71z > RHEL72z > RHEl73z > RHEL74-0day > Rhel74up1 > RhEl74up2)
3. Configure new Replica against this upgraded server at domain-level 0.

Actual results:
After step3, Replica installation fails and keeps prompting for admin password

Run connection check to master
admin@<master>.testrelm.test's password: 



# ipa-replica-install.log
2017-10-10T08:01:57Z DEBUG Process finished, return code=2
2017-10-10T08:01:57Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 333, in run
    cfgr.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 366, in run
    self.validate()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 375, in validate
    for _nothing in self._validator():
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 434, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 458, in _handle_validate_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 453, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 424, in __runner
    step()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 636, in _configure
    next(validator)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 434, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 458, in _handle_validate_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 521, in _handle_exception
    self.__parent._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 453, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 518, in _handle_exception
    super(ComponentBase, self)._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 453, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 424, in __runner
    step()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python2.7/site-packages/ipapython/install/common.py", line 63, in _install
    for _nothing in self._installer(self.parent):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/__init__.py", line 615, in main
    replica_install_check(self)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 391, in decorated
    raise ScriptError()

2017-10-10T08:01:57Z DEBUG The ipa-replica-install command failed, exception: ScriptError: 
2017-10-10T08:01:57Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information


Workaround:
When the same ipa-master is set to domain-level 1 and then the same replica replica failing in above step3 is configured, then replica-installation is successful. 


Expected results:
Installation of new replica at domain-level 0 should be successful

Additional info:
The installation of new replica is successful when ipa-server is installed at RHEL 7.4up2 with domain-level 0.

Comment 8 Petr Vobornik 2017-10-24 10:39:59 UTC
Upstream ticket:
https://pagure.io/freeipa/issue/7221

Comment 11 Nikhil Dehadrai 2018-01-10 12:07:11 UTC
ipa-server-version:

Tested the bug with following observations:
1. Setup IPA-master upgraded from RHEL 70 to RHEL 75 (linear upgrade path: 70z > 71z >72z >73z > 740-day > 74z > 75).
2. Once the upgrade process is completed successfully, run 'ipa domainlevel-get' command.

[root@ipaqavmg ipa]# ipa domainlevel-get
-----------------------
Current domain level: 0
-----------------------

3. Configure replica gainst this upgraded IPA-master in step1.

On Replica
# ipa-replica-install -U --setup-dns --forwarder <forwarder_ip> --allow-zone-overlap --setup-ca --admin-password <admin-password> --password <password> replica-system.gpg

Noticed that replica installation fails at step: 
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/28]: configuring certificate server instance
ipa.ipaserver.install.cainstance.CAInstance: CRITICAL Failed to configure CA instance: Command '/usr/sbin/pkispawn -s CA -f /tmp/tmpNl6QNW' returned non-zero exit status 1
ipa.ipaserver.install.cainstance.CAInstance: CRITICAL See the installation logs and the following files/directories for more information:
ipa.ipaserver.install.cainstance.CAInstance: CRITICAL   /var/log/pki/pki-tomcat
  [error] RuntimeError: CA configuration failed.
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

ipa.ipapython.install.cli.install_tool(CompatServerReplicaInstall): ERROR    CA configuration failed.
ipa.ipapython.install.cli.install_tool(CompatServerReplicaInstall): ERROR    The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information


[root@spider ipa]# tail -100 /var/log/ipareplica-install.log 
subsystemCert cert-pki-ca                                    u,u,u
caSigningCert cert-pki-ca                                    CTu,Cu,Cu
Server-Cert cert-pki-ca                                      u,u,u
auditSigningCert cert-pki-ca                                 u,u,Pu

Installation failed:
com.netscape.certsrv.base.PKIException: Error in populating database: java.io.IOException: Failed to setup the replication for cloning.

Please check the CA logs in /var/log/pki/pki-tomcat/ca.

2018-01-10T11:57:48Z DEBUG stderr=
2018-01-10T11:57:48Z CRITICAL Failed to configure CA instance: Command '/usr/sbin/pkispawn -s CA -f /tmp/tmpNl6QNW' returned non-zero exit status 1
2018-01-10T11:57:48Z CRITICAL See the installation logs and the following files/directories for more information:
2018-01-10T11:57:48Z CRITICAL   /var/log/pki/pki-tomcat
2018-01-10T11:57:48Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 504, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 494, in run_step
    method()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 615, in __spawn_instance
    self.tmp_agent_pwd)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/dogtaginstance.py", line 148, in spawn_instance
    self.handle_setup_error(e)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/dogtaginstance.py", line 386, in handle_setup_error
    raise RuntimeError("%s configuration failed." % self.subsystem)
RuntimeError: CA configuration failed.

2018-01-10T11:57:48Z DEBUG   [error] RuntimeError: CA configuration failed.
2018-01-10T11:57:48Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 333, in run
    cfgr.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 368, in run
    self.execute()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 392, in execute
    for _nothing in self._executor():
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 434, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 463, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 453, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 424, in __runner
    step()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 658, in _configure
    next(executor)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 434, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 463, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 521, in _handle_exception
    self.__parent._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 453, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 518, in _handle_exception
    super(ComponentBase, self)._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 453, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 424, in __runner
    step()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python2.7/site-packages/ipapython/install/common.py", line 63, in _install
    for _nothing in self._installer(self.parent):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/__init__.py", line 617, in main
    replica_install(self)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 386, in decorated
    func(installer)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 1458, in install
    ca.install(False, config, options)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ca.py", line 205, in install
    install_step_0(standalone, replica_config, options)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ca.py", line 284, in install_step_0
    use_ldaps=standalone)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 447, in configure_instance
    self.start_creation(runtime=runtime)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 504, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 494, in run_step
    method()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 615, in __spawn_instance
    self.tmp_agent_pwd)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/dogtaginstance.py", line 148, in spawn_instance
    self.handle_setup_error(e)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/dogtaginstance.py", line 386, in handle_setup_error
    raise RuntimeError("%s configuration failed." % self.subsystem)

2018-01-10T11:57:48Z DEBUG The ipa-replica-install command failed, exception: RuntimeError: CA configuration failed.
2018-01-10T11:57:48Z ERROR CA configuration failed.
2018-01-10T11:57:48Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information


Thus on the basis of above observations, marking the status of bug to "ASSIGNED"

Comment 18 Florence Blanc-Renaud 2018-01-11 14:11:42 UTC
Moving back to ON_QA as the issue ipa-replica-conncheck has been fixed. A new bug has been opened 1533497 to track the issue seen on comment #11.

Comment 19 Nikhil Dehadrai 2018-01-11 14:15:03 UTC
ipa-server-version: 4.5.4.7

Verified the bug on the basis of following observations:

1. Verified that ipa-replica-conncheck related issue/failure is not observed when ipa-replica is setup against ipa-server which is linearly upgraded to latest version.
2. Noticed that ipa-replica installation still fails, thus a separate bug is logged BZ#1533497.

Thus on the basis of above observations and comments at comment#11 and comment#17, marking the status of this Bugzilla to "VERIFIED".

Comment 22 errata-xmlrpc 2018-04-10 16:48:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0918