Bug 1579190

Summary:	Improve Custodia client and key distribution handling [rhel-7.5.z]
Product:	Red Hat Enterprise Linux 7	Reporter:	Oneata Mircea Teodor <toneata>
Component:	ipa	Assignee:	IPA Maintainers <ipa-maint>
Status:	CLOSED ERRATA	QA Contact:	ipa-qe <ipa-qe>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	7.4	CC:	cheimes, fhanzelk, frenaud, ipa-maint, ksiddiqu, msauton, ndehadra, pvoborni, rcritten, tscherf
Target Milestone:	rc	Keywords:	ZStream
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:	ipa-4.5.4-10.el7_5.3	Doc Type:	Bug Fix
Doc Text:	When an Identity Management (IdM) host is promoted to a replica, IdM uses Custodia to transport secrets, for example Certificate Authority keys. Previously, Custodia uploaded the newly created public keys of the IdM host to the local LDAP server and then connected to one or two remote Custodia instances to check if the keys had been replicated there. Consequently, on a busy LDAP cluster the replica installation sometimes timed out. With this update, Custodia directly uploads its key to a remote IdM master and uses this remote IdM master to request all secrets. As a result, retrieval of secrets no longer depends on timely replication between local and remote LDAP servers.	Story Points:	---
Clone Of:	1577108	Environment:
Last Closed:	2018-06-26 16:49:05 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1577108
Bug Blocks:

Description Oneata Mircea Teodor 2018-05-17 06:56:04 UTC

This bug has been copied from bug #1577108 and has been proposed to be backported to 7.5 z-stream (EUS).

Comment 5 Nikhil Dehadrai 2018-05-28 11:18:51 UTC

Version: ipa-server-4.5.4-10.el7_5.2.x86_64

Tested the bug with following observations using the comment#4 as reference:
1. Setup IPA server with CA
2. Setup REPLICA-1 against this IPA in step1, without CA. The setup of REPLICA-1 is successful.
3. Now stop custodia service on REPLICA-1.
4. Setup REPLICA-2 with CA (--setup-ca), and using REPLICA-1 as source. The setup of REPLICA-2 FAILED.

Console:

Replica-1:
------------
[root@auto-hv-01-guest07 ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: STOPPED
ntpd Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful
[root@auto-hv-01-guest07 ~]# systemctl status ipa-custodia
● ipa-custodia.service - IPA Custodia Service
   Loaded: loaded (/usr/lib/systemd/system/ipa-custodia.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

May 28 05:14:11 auto-hv-01-guest07.testrelm.test systemd[1]: Starting IPA Custodia Service...
May 28 05:14:12 auto-hv-01-guest07.testrelm.test ipa-custodia[32580]: 2018-05-28 05:14:12 - server                ...ck
May 28 05:14:12 auto-hv-01-guest07.testrelm.test systemd[1]: Started IPA Custodia Service.
May 28 05:21:57 auto-hv-01-guest07.testrelm.test systemd[1]: Stopping IPA Custodia Service...
May 28 05:21:57 auto-hv-01-guest07.testrelm.test systemd[1]: Stopped IPA Custodia Service.
Hint: Some lines were ellipsized, use -l to show in full.
[root@auto-hv-01-guest07 ~]#



Replica-2:
-------------
[root@auto-hv-01-guest08 ~]# ipa-replica-install -U --setup-ca --setup-dns --forwarder=x.x.x.x --ip-address=x.x.x.x -P admin -w Secret123
WARNING: conflicting time&date synchronization service 'chronyd' will
be disabled in favor of ntpd

Configuring client side components
Discovery was successful!
Client hostname: auto-hv-01-guest08.testrelm.test
Realm: TESTRELM.TEST
DNS Domain: testrelm.test
IPA Server: auto-hv-01-guest07.testrelm.test
BaseDN: dc=testrelm,dc=test

continued....

  [27/27]: configuring certmonger renewal for lightweight CAs
Done configuring certificate server (pki-tomcatd).
Configuring Kerberos KDC (krb5kdc)
  [1/1]: installing X509 Certificate for PKINIT
Done configuring Kerberos KDC (krb5kdc).
Applying LDAP updates
Upgrading IPA:. Estimated time: 1 minute 30 seconds
  [1/9]: stopping directory server
  [2/9]: saving configuration
  [3/9]: disabling listeners
  [4/9]: enabling DS global lock
  [5/9]: starting directory server
  [6/9]: upgrading server
  [7/9]: stopping directory server
  [8/9]: restoring configuration
  [9/9]: starting directory server
Done.
Restarting the KDC
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

ipa.ipapython.install.cli.install_tool(CompatServerReplicaInstall): ERROR    503 Server Error: Service Unavailable
ipa.ipapython.install.cli.install_tool(CompatServerReplicaInstall): ERROR    The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
[root@auto-hv-01-guest08 ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa: INFO: The ipactl command was successful



ipa-replica-install-log ( REPLICA-2)
---------------------------------------
2018-05-28T09:30:49Z DEBUG   [9/9]: starting directory server
2018-05-28T09:30:49Z DEBUG Starting external process
2018-05-28T09:30:49Z DEBUG args=/bin/systemctl start dirsrv
2018-05-28T09:30:53Z DEBUG Process finished, return code=0
2018-05-28T09:30:53Z DEBUG stdout=
2018-05-28T09:30:53Z DEBUG stderr=
2018-05-28T09:30:53Z DEBUG Created connection context.ldap2_139880832010256
2018-05-28T09:30:53Z DEBUG   duration: 4 seconds
2018-05-28T09:30:53Z DEBUG Done.
2018-05-28T09:30:53Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state'
2018-05-28T09:30:53Z DEBUG Saving StateFile to '/var/lib/ipa/sysupgrade/sysupgrade.state'
2018-05-28T09:30:53Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state'
2018-05-28T09:30:53Z DEBUG Saving StateFile to '/var/lib/ipa/sysupgrade/sysupgrade.state'
2018-05-28T09:30:53Z DEBUG Restarting the KDC
2018-05-28T09:30:53Z DEBUG Starting external process
2018-05-28T09:30:53Z DEBUG args=/bin/systemctl restart krb5kdc.service
2018-05-28T09:30:53Z DEBUG Process finished, return code=0
2018-05-28T09:30:53Z DEBUG stdout=
2018-05-28T09:30:53Z DEBUG stderr=
2018-05-28T09:30:53Z DEBUG Starting external process
2018-05-28T09:30:53Z DEBUG args=/bin/systemctl is-active krb5kdc.service
2018-05-28T09:30:54Z DEBUG Process finished, return code=0
2018-05-28T09:30:54Z DEBUG stdout=active

2018-05-28T09:30:54Z DEBUG stderr=
2018-05-28T09:30:54Z INFO Waiting up to 300 seconds to see our keys appear on host: auto-hv-01-guest07.testrelm.test
2018-05-28T09:30:54Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 333, in run
    cfgr.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 368, in run
    self.execute()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 392, in execute
    for _nothing in self._executor():
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 434, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 463, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 453, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 424, in __runner
    step()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 658, in _configure
    next(executor)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 434, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 463, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 521, in _handle_exception
    self.__parent._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 453, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 518, in _handle_exception
    super(ComponentBase, self)._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 453, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 424, in __runner
    step()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python2.7/site-packages/ipapython/install/common.py", line 63, in _install
    for _nothing in self._installer(self.parent):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/__init__.py", line 617, in main
    replica_install(self)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 399, in decorated
    func(installer)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 1515, in install
    custodia.import_dm_password(config.master_host_name)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/custodiainstance.py", line 187, in import_dm_password
    cli.fetch_key('dm/DMHash')
  File "/usr/lib/python2.7/site-packages/ipaserver/secrets/client.py", line 101, in fetch_key
    r.raise_for_status()
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 834, in raise_for_status
    raise HTTPError(http_error_msg, response=self)

2018-05-28T09:30:54Z DEBUG The ipa-replica-install command failed, exception: HTTPError: 503 Server Error: Service Unavailable
2018-05-28T09:30:54Z ERROR 503 Server Error: Service Unavailable
2018-05-28T09:30:54Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information


Thus marking the status of bug to 'ASSIGNED'

Comment 6 Rob Crittenden 2018-06-05 17:31:02 UTC

We would need to see the Apache error log to see why it threw a 503.

Comment 8 Rob Crittenden 2018-06-06 15:19:21 UTC

Hmm, can you look in the access logs on both the master and replica for the 503 and see if there is anything in the Apache error or dogtag debug at the same time?

Comment 14 Rob Crittenden 2018-06-11 14:45:15 UTC

ipa-4-5:

    d3c09a6 Use one Custodia peer to retrieve all secrets

Comment 18 Nikhil Dehadrai 2018-06-12 15:20:38 UTC

Version: ipa-server-4.5.4-10.el7_5.3.x86_64

Tested the bug with following observations using the comment#4 as reference:
1. Setup IPA server with CA
2. Setup REPLICA-1 against this IPA in step1, without CA. The setup of REPLICA-1 is successful.
3. Now stop custodia service on REPLICA-1.
4. Setup REPLICA-2 with CA (--setup-ca), and using REPLICA-1 as source. The setup of REPLICA-2 is successful.

Thus the issue mentioned in above comment#5 is no more observed.

Setup: (Line Topology)
-------------------------

MASTER ------> REPLICA-1 ----------> Replica-2


Console-log:
------------------

MASTER:
-------------
[root@ipaqavmb ~]# rpm -q ipa-server
ipa-server-4.5.4-10.el7_5.3.x86_64
[root@ipaqavmb ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful
[root@ipaqavmb ~]# tail -1 /var/log/ipaserver-install.log 
2018-06-12T13:13:13Z INFO The ipa-server-install command was successful
[root@ipaqavmb ~]# kinit admin
Password for admin: 
[root@ipaqavmb ~]#


REPLICA-1:
-------------
[root@bkr-hv01-guest17 ~]# ipa-replica-install -U --setup-dns --forwarder=x.x.x.x --ip-address=x.x.x.x -P admin -w Secret123

Configuring client side components
Discovery was successful!
Client hostname: bkr-hv01-guest17.testrelm.test
Realm: TESTRELM.TEST
DNS Domain: testrelm.test
IPA Server: ipaqavmb.testrelm.test
BaseDN: dc=testrelm,dc=test

[root@bkr-hv01-guest17 ~]# tail -1 /var/log/ipareplica-install.log 
2018-06-12T14:21:31Z INFO The ipa-replica-install command was successful
[root@bkr-hv01-guest17 ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: STOPPED
ntpd Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful
[root@bkr-hv01-guest17 ~]# kinit admin
Password for admin: 
[root@bkr-hv01-guest17 ~]# rpm -q ipa-server
ipa-server-4.5.4-10.el7_5.3.x86_64
[root@bkr-hv01-guest17 ~]#



REPLICA-2:
---------------------
[root@ipaqavmc ~]# ipa-replica-install -U --setup-ca --setup-dns --forwarder=x.x.x.x --ip-address=x.x.x.x -P admin -w Secret123
Configuring client side components
Discovery was successful!
Client hostname: ipaqavmc.testrelm.test
Realm: TESTRELM.TEST
DNS Domain: testrelm.test
IPA Server: bkr-hv01-guest17.testrelm.test
BaseDN: dc=testrelm,dc=test



[root@ipaqavmc ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful
[root@ipaqavmc ~]# kinit admin
Password for admin: 
[root@ipaqavmc ~]# ipa-replica-manage list
ipaqavmc.testrelm.test: master
ipaqavmb.testrelm.test: master
bkr-hv01-guest17.testrelm.test: master
[root@ipaqavmc ~]# tail -1 /var/log/ipareplica-install.log 
2018-06-12T14:57:20Z INFO The ipa-replica-install command was successful
[root@ipaqavmc ~]# 


Thus on the basis of above observations and comment#17, marking the status of bug to 'VERIFIED'.

Comment 24 errata-xmlrpc 2018-06-26 16:49:05 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1985