Bug 1437879

Cloned from upstream: https://pagure.io/freeipa/issue/6743

Replica install failing with copr build 

Console output/conncheck log on Replica:
=========================================
```
[root@qe-blade-10 ~]# /usr/sbin/ipa-replica-install -U --setup-ca --setup-dns --forwarder=10.16.36.29 --ip-address=10.19.34.80 -P admin -w xxxxxxxx
WARNING: conflicting time&date synchronization service 'chronyd' will
be disabled in favor of ntpd

Configuring client side components
Discovery was successful!
Client hostname: qe-blade-10.testrelm.test
Realm: TESTRELM.TEST
DNS Domain: testrelm.test
IPA Server: qe-blade-08.testrelm.test
BaseDN: dc=testrelm,dc=test

Skipping synchronizing time with NTP server.
Successfully retrieved CA cert
    Subject:     CN=Certificate Authority,O=TESTRELM.TEST
    Issuer:      CN=Certificate Authority,O=TESTRELM.TEST
    Valid From:  2017-03-09 09:42:13
    Valid Until: 2037-03-09 08:42:13

Enrolled in IPA realm TESTRELM.TEST
Created /etc/ipa/default.conf
New SSSD config will be created
Configured sudoers in /etc/nsswitch.conf
Configured /etc/sssd/sssd.conf
Configured /etc/krb5.conf for IPA realm TESTRELM.TEST
trying https://qe-blade-08.testrelm.test/ipa/json
Forwarding 'ping' to json server 'https://qe-blade-08.testrelm.test/ipa/json'
Forwarding 'ca_is_enabled' to json server 'https://qe-blade-08.testrelm.test/ipa/json'
Systemwide CA database updated.
Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_ecdsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_ed25519_key.pub
Forwarding 'host_mod' to json server 'https://qe-blade-08.testrelm.test/ipa/json'
SSSD enabled
Configured /etc/openldap/ldap.conf
Configured /etc/ssh/ssh_config
Configured /etc/ssh/sshd_config
Configuring testrelm.test as NIS domain.
Client configuration complete.
The ipa-client-install command was successful

ipa         : ERROR    Reverse DNS resolution of address 10.19.34.80 (qe-blade-10.testrelm.test) failed. Clients may not function properly. Please check your DNS setup. (Note that this check queries IPA DNS directly and ignores /etc/hosts.)
ipa         : ERROR    Reverse DNS resolution of address 2620:52:0:1322:221:5eff:fe20:2f4e (qe-blade-10.testrelm.test) failed. Clients may not function properly. Please check your DNS setup. (Note that this check queries IPA DNS directly and ignores /etc/hosts.)
Checking DNS forwarders, please wait ...
Run connection check to master
Removing client side components
Unenrolling client from IPA server
Removing Kerberos service principals from /etc/krb5.keytab
Disabling client Kerberos and LDAP configurations
Redundant SSSD configuration file /etc/sssd/sssd.conf was moved to /etc/sssd/sssd.conf.deleted
Restoring client configuration files
Unconfiguring the NIS domain.
nscd daemon is not installed, skip configuration
nslcd daemon is not installed, skip configuration
Systemwide CA database updated.
Client uninstall complete.
The ipa-client-install command was successful

Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

ipa.ipapython.install.cli.install_tool(CompatServerReplicaInstall): ERROR    Connection check failed!
See /var/log/ipareplica-conncheck.log for more information.
If the check results are not valid it can be skipped with --skip-conncheck parameter.
ipa.ipapython.install.cli.install_tool(CompatServerReplicaInstall): ERROR    The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
[root@qe-blade-10 ~]# tail -n 20 /var/log/ipareplica-conncheck.log
2017-03-09T09:51:22Z DEBUG stdout=184556

2017-03-09T09:51:22Z DEBUG stderr=
2017-03-09T09:51:22Z DEBUG Starting external process
2017-03-09T09:51:22Z DEBUG args=keyctl pupdate 184556
2017-03-09T09:51:22Z DEBUG Process finished, return code=0
2017-03-09T09:51:22Z DEBUG stdout=
2017-03-09T09:51:22Z DEBUG stderr=
2017-03-09T09:51:22Z DEBUG Destroyed connection context.rpcclient_102771536
2017-03-09T09:51:22Z ERROR ERROR: Remote master check failed with following error message(s):
an internal error has occurred
2017-03-09T09:51:22Z DEBUG Stopping listening thread.
2017-03-09T09:51:22Z DEBUG 389 tcp: Stopped listening
2017-03-09T09:51:22Z DEBUG 636 tcp: Stopped listening
2017-03-09T09:51:22Z DEBUG 88 tcp: Stopped listening
2017-03-09T09:51:22Z DEBUG 88 udp: Stopped listening
2017-03-09T09:51:22Z DEBUG 464 tcp: Stopped listening
2017-03-09T09:51:22Z DEBUG 464 udp: Stopped listening
2017-03-09T09:51:22Z DEBUG 80 tcp: Stopped listening
2017-03-09T09:51:22Z DEBUG 443 tcp: Stopped listening
[root@qe-blade-10 ~]#

[root@qe-blade-10 ~]# rpm -q ipa-server
ipa-server-4.4.90-201703072305.el7.x86_64
[root@qe-blade-10 ~]# 


On Maser:
=========
[root@qe-blade-08 ~]# tail -n 25 /var/log/httpd/error_log 
[Thu Mar 09 04:51:06.026021 2017] [:error] [pid 16945] ipa: INFO: [jsonserver_kerb] host/qe-blade-10.testrelm.test: host_mod(u'qe-blade-10.testrelm.test', ipasshpubkey=(u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDh6qLjI5QlNIF4haTzYFgBsf5bwT/3uqgtJwJ4vrl5oauvzT4gxULJjaQN6M0K6VHO/8MEDmcuSAjDLiOREtRDYF+RyN9oxjd1l7akn/iV5vXPCeL5csn3OvhZla1EHS9ZCXqjsmB+TlfYVQwlI0ixebylM8CGtEGeVnQyLPxv3BkeFdlt5GpuWAFBws2AQPUe1DRF4OA9C9OoO+WssZQlMs+Eb+1vaVPEIvAuXjcjSQZcddpV0tzmIuPWf5w3iXvpJaZsKGvlzY5iR30vYpS/UBG+O6rkGgHtDvfD95AZFDGsQ17/gdtA1ZaZvHA2Dok3SotE+57mPB0NhlNlmWth', u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNMCdALplMvbVKyW6ZgDmaC0tO/KmefA7O6nuOFL4S1yShKsx1cKLUOrvGShwqvVHh9jnT/wgNZlcwWjief2v0M=', u'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE2NsafAr0yT7uf11hbVieb2d6P/zM7ofyVB6AzXNoj+'), updatedns=False, version=u'2.26'): EmptyModlist
[Thu Mar 09 04:51:13.562701 2017] [:error] [pid 16944] ipa: INFO: [jsonserver_session] host/qe-blade-10.testrelm.test: env((u'version',)): SUCCESS
[Thu Mar 09 04:51:13.664681 2017] [:error] [pid 16945] ipa: INFO: [jsonserver_session] host/qe-blade-10.testrelm.test: env((u'fips_mode',)): SUCCESS
[Thu Mar 09 04:51:21.987427 2017] [:error] [pid 16944] ipa: INFO: [jsonserver_session] admin: ping/1(version=u'2.219'): SUCCESS
[Thu Mar 09 04:51:22.090253 2017] [:error] [pid 16945] ipa: ERROR: non-public: DBusException: org.freedesktop.DBus.Error.AccessDenied: Rejected send message, 1 matched rules; type="method_call", sender=":1.280" (uid=388 pid=16945 comm="(wsgi:ipa)      -DFOREGROUND ") interface="org.freeipa.server" member="conncheck" error name="(unset)" requested_reply="0" destination="org.freeipa.server" (uid=0 pid=16673 comm="/usr/sbin/oddjobd -n -p /var/run/oddjobd.pid -t 30")
[Thu Mar 09 04:51:22.090297 2017] [:error] [pid 16945] Traceback (most recent call last):
[Thu Mar 09 04:51:22.090300 2017] [:error] [pid 16945]   File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 368, in wsgi_execute
[Thu Mar 09 04:51:22.090303 2017] [:error] [pid 16945]     result = command(*args, **options)
[Thu Mar 09 04:51:22.090305 2017] [:error] [pid 16945]   File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 447, in __call__
[Thu Mar 09 04:51:22.090313 2017] [:error] [pid 16945]     return self.__do_call(*args, **options)
[Thu Mar 09 04:51:22.090316 2017] [:error] [pid 16945]   File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 475, in __do_call
[Thu Mar 09 04:51:22.090318 2017] [:error] [pid 16945]     ret = self.run(*args, **options)
[Thu Mar 09 04:51:22.090325 2017] [:error] [pid 16945]   File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 797, in run
[Thu Mar 09 04:51:22.090327 2017] [:error] [pid 16945]     return self.execute(*args, **options)
[Thu Mar 09 04:51:22.090330 2017] [:error] [pid 16945]   File "/usr/lib/python2.7/site-packages/ipaserver/plugins/server.py", line 892, in execute
[Thu Mar 09 04:51:22.090332 2017] [:error] [pid 16945]     ret, stdout, _stderr = server.conncheck(keys[-1])
[Thu Mar 09 04:51:22.090335 2017] [:error] [pid 16945]   File "/usr/lib64/python2.7/site-packages/dbus/proxies.py", line 70, in __call__
[Thu Mar 09 04:51:22.090337 2017] [:error] [pid 16945]     return self._proxy_method(*args, **keywords)
[Thu Mar 09 04:51:22.090339 2017] [:error] [pid 16945]   File "/usr/lib64/python2.7/site-packages/dbus/proxies.py", line 145, in __call__
[Thu Mar 09 04:51:22.090357 2017] [:error] [pid 16945]     **keywords)
[Thu Mar 09 04:51:22.090360 2017] [:error] [pid 16945]   File "/usr/lib64/python2.7/site-packages/dbus/connection.py", line 651, in call_blocking
[Thu Mar 09 04:51:22.090362 2017] [:error] [pid 16945]     message, timeout)
[Thu Mar 09 04:51:22.090365 2017] [:error] [pid 16945] DBusException: org.freedesktop.DBus.Error.AccessDenied: Rejected send message, 1 matched rules; type="method_call", sender=":1.280" (uid=388 pid=16945 comm="(wsgi:ipa)      -DFOREGROUND ") interface="org.freeipa.server" member="conncheck" error name="(unset)" requested_reply="0" destination="org.freeipa.server" (uid=0 pid=16673 comm="/usr/sbin/oddjobd -n -p /var/run/oddjobd.pid -t 30")
[Thu Mar 09 04:51:22.090739 2017] [:error] [pid 16945] ipa: INFO: [jsonserver_session] admin: server_conncheck(u'qe-blade-08.testrelm.test', u'qe-blade-10.testrelm.test', version=u'2.162'): InternalError
[Thu Mar 09 04:51:23.888203 2017] [:error] [pid 16944] ipa: INFO: [xmlserver] host/qe-blade-10.testrelm.test: host_disable(u'qe-blade-10.testrelm.test', version=u'2.51'): SUCCESS
[root@qe-blade-08 ~]#

[root@qe-blade-08 ~]# rpm -q ipa-server
ipa-server-4.4.90-201703072305.el7.x86_64
[root@qe-blade-08 ~]#
```