Bug 1155654

We need some logs to diagnose this.

debug log (from master and replica)
journal
ipa-replica-log

Petr, can you please provide the logs given that Jan is not available at the moment?

Created attachment 952237 [details]
journalctl

Created attachment 952246 [details]
pki-tomcat/localhost-2014-10-30.log

Attached logs are from installation of master, where it failed for me as well.

pki-*-10.1.2-4.el7.noarch
389-ds-base-1.3.3.1-7.el7.x86_64

the interesting part is:

 server[26343]: SSLAuthenticatorWithFallback: Creating SSL authenticator with fallback
 server[26343]: SSLAuthenticatorWithFallback: Setting container
 server[26343]: SSLAuthenticatorWithFallback: Initializing authenticators
 server[26343]: SSLAuthenticatorWithFallback: Starting authenticators
 server[26343]: Internal Database Error encountered: Could not connect to LDAP server host <hostname> port 636 Error netscape.ldap.LDAPException: IO Error creating JSS SSL Socket (-1)

Thierry already did some investigation in https://fedorahosted.org/freeipa/ticket/4666

ipa-server-install failed with the error in attachment 952246 [details]

Sorry, comment 6 is not related to this issue. It's bug 1158410.

Created attachment 953201 [details]
replica ca debug log

Created attachment 953202 [details]
replica ca system log

Created attachment 953203 [details]
master ca debug log

Created attachment 953204 [details]
replica ca spawn log

Created attachment 953205 [details]
replica pki-tomcatd journal

Created attachment 953216 [details]
ipa-replicainstall.log

proper logs attached

Petr,

I do not know if this will help or not, however, I logged onto both of the machines mentioned in the logs:

    * vm-107.idm.lab.eng.brq.redhat.com (contains master IPA)
    * vm-108.idm.lab.eng.brq.redhat.com (contained replica IPA)

Although the replica IPA had been removed, and you stated earlier that this problem did not have to do with bug 1158410, I did notice the following:

    * vm-107.idm.lab.eng.brq.redhat.com (master) contained the following:
      * jss-4.2.6-35.el7.x86_64
      * tomcatjss-7.1.0-5.el7.noarch
      both of which include the TLS v1.1 + v1.2 changes and
      therefore '/etc/pki/pki-tomcat/server.xml' contained the following:
          * sslVersionRangeStream="tls1_0:tls1_2"
          * sslVersionRangeDatagram="tls1_1:tls1_2"

whereas:

    * vm-108.idm.lab.eng.brq.redhat.com (replica) contained the following:
      * jss-4.2.6-33.el7.x86_64
      * tomcatjss-7.1.0-4.el7.noarch
    which do not contain the TLS v1.1 + v1.2 changes

The logs appeared to indicate that the replica "Failed to obtain configuration entries from the master for cloning", and although it would seem that the replica could communicate with the master via TLS 1.0 over Streams, as a simple test, could you upgrade the JSS and TOMCATJSS on the IPA replica machine, and retry to see if this was possibly the issue?

Thanks,
-- Matt

Matt,

Retried, same issue. Server are in a state right after the fail, if you want to investigate.

Both servers have:
pki-base-10.1.2-4.el7.noarch
tomcatjss-7.1.0-5.el7.noarch
389-ds-base-1.3.3.1-6.el7.x86_64
ipa-server-4.1.0-3.el7.x86_64
jss-4.2.6-35.el7.x86_64

I took a look at the logs on the master and noticed a few interesting things:

1) The  original error reported in this bug occurs from a failure of the master to send the requested data back to the clone.

We see the following in the debug log:

[06/Nov/2014:17:14:53][ajp-bio-127.0.0.1-8009-exec-7]: Failed to send the XML output
[06/Nov/2014:17:14:53][ajp-bio-127.0.0.1-8009-exec-7]: CMSServlet: curDate=Thu Nov 06 17:14:53 CET 2014 id=caGetConfigEntries time=934

This corresponds to a log in the journal which just (unhelpfully) says "ERROR"

2) There appear to be three subsequent attempts to call the getConfigEntries servlet, but perhaps these were done from a browser?  These fail because they do not contain the correct token obtained from logging into the security domain.  In fact, I see no logins to the security domain.

I made the following suggestion to Matt and Ade earlier in email, I am curious of what the result was or whether it was done at all:
"
...The first question I'd ask is, does cloning from Dogtag works in the same environment?
If it does, the next is what's their dogtag related config's? 
"

Verified

pki version:
============
[root@dhcp207-133 ~]# rpm -q ipa-server pki-ca
ipa-server-4.1.0-13.el7.x86_64
pki-ca-10.1.2-7.el7.noarch
[root@dhcp207-133 ~]# 

snip from automation log:
========================
:: [  BEGIN   ] :: Running ' /usr/sbin/ipa-replica-install -U --setup-ca --setup-dns --forwarder=10.65.201.89 -w xxxxxxxx -p xxxxxxxx /opt/rhqa_ipa/replica-info-dhcp207-133.testrelm.test.gpg'
Checking forwarders, please wait ...
WARNING: DNS forwarder 10.65.201.89 does not return DNSSEC signatures in answers
Please fix forwarder configuration to enable DNSSEC support.
(For BIND 9 add directive "dnssec-enable yes;" to "options {}")
WARNING: DNSSEC validation will be disabled
Run connection check to master
Check connection from replica to remote master 'master.testrelm.test':
   Directory Service: Unsecure port (389): OK
   Directory Service: Secure port (636): OK
   Kerberos KDC: TCP (88): OK
   Kerberos Kpasswd: TCP (464): OK
   HTTP Server: Unsecure port (80): OK
   HTTP Server: Secure port (443): OK

The following list of ports use UDP protocol and would need to be
checked manually:
   Kerberos KDC: UDP (88): SKIPPED
   Kerberos Kpasswd: UDP (464): SKIPPED

Connection from replica to master is OK.
Start listening on required ports for remote master check
Get credentials to log in to remote master
Check SSH connection to remote master
Execute check on remote master
Check connection from master to remote replica 'dhcp207-133.testrelm.test':
   Directory Service: Unsecure port (389): OK
   Directory Service: Secure port (636): OK
   Kerberos KDC: TCP (88): OK
   Kerberos KDC: UDP (88): OK
   Kerberos Kpasswd: TCP (464): OK
   Kerberos Kpasswd: UDP (464): OK
   HTTP Server: Unsecure port (80): OK
   HTTP Server: Secure port (443): OK

Connection from master to replica is OK.

Connection check OK
Using reverse zone(s) 207.65.10.in-addr.arpa.
Configuring NTP daemon (ntpd)
  [1/4]: stopping ntpd
  [2/4]: writing configuration
  [3/4]: configuring ntpd to start on boot
  [4/4]: starting ntpd
Done configuring NTP daemon (ntpd).
Configuring directory server (dirsrv): Estimated time 1 minute
  [1/35]: creating directory server user
  [2/35]: creating directory server instance
  [3/35]: adding default schema
  [4/35]: enabling memberof plugin
  [5/35]: enabling winsync plugin
  [6/35]: configuring replication version plugin
  [7/35]: enabling IPA enrollment plugin
  [8/35]: enabling ldapi
  [9/35]: configuring uniqueness plugin
  [10/35]: configuring uuid plugin
  [11/35]: configuring modrdn plugin
  [12/35]: configuring DNS plugin
  [13/35]: enabling entryUSN plugin
  [14/35]: configuring lockout plugin
  [15/35]: creating indices
  [16/35]: enabling referential integrity plugin
  [17/35]: configuring ssl for ds instance
  [18/35]: configuring certmap.conf
  [19/35]: configure autobind for root
  [20/35]: configure new location for managed entries
  [21/35]: configure dirsrv ccache
  [22/35]: enable SASL mapping fallback
  [23/35]: restarting directory server
  [24/35]: setting up initial replication
Starting replication, please wait until this has completed.
Update in progress, 5 seconds elapsed
Update succeeded

  [25/35]: updating schema
  [26/35]: setting Auto Member configuration
  [27/35]: enabling S4U2Proxy delegation
  [28/35]: importing CA certificates from LDAP
  [29/35]: initializing group membership
  [30/35]: adding master entry
  [31/35]: configuring Posix uid/gid generation
  [32/35]: adding replication acis
  [33/35]: enabling compatibility plugin
  [34/35]: tuning directory server
  [35/35]: configuring directory to start on boot
Done configuring directory server (dirsrv).
Configuring certificate server (pki-tomcatd): Estimated time 3 minutes 30 seconds
  [1/22]: creating certificate server user
  [2/22]: configuring certificate server instance
  [3/22]: stopping certificate server instance to update CS.cfg
  [4/22]: backing up CS.cfg
  [5/22]: disabling nonces
  [6/22]: set up CRL publishing
  [7/22]: enable PKIX certificate path discovery and validation
  [8/22]: starting certificate server instance
  [9/22]: creating RA agent certificate database
  [10/22]: importing CA chain to RA certificate database
  [11/22]: fixing RA database permissions
  [12/22]: setting up signing cert profile
  [13/22]: set certificate subject base
  [14/22]: enabling Subject Key Identifier
  [15/22]: enabling Subject Alternative Name
  [16/22]: enabling CRL and OCSP extensions for certificates
  [17/22]: setting audit signing renewal to 2 years
  [18/22]: configuring certificate server to start on boot
  [19/22]: configure certmonger for renewals
  [20/22]: configure certificate renewals
  [21/22]: configure Server-Cert certificate renewal
  [22/22]: Configure HTTP to proxy connections
Done configuring certificate server (pki-tomcatd).
Restarting the directory and certificate servers
Configuring Kerberos KDC (krb5kdc): Estimated time 30 seconds
  [1/9]: adding sasl mappings to the directory
  [2/9]: writing stash file from DS
  [3/9]: configuring KDC
  [4/9]: creating a keytab for the directory
  [5/9]: creating a keytab for the machine
  [6/9]: adding the password extension to the directory
  [7/9]: enable GSSAPI for replication
  [8/9]: starting the KDC
  [9/9]: configuring KDC to start on boot
Done configuring Kerberos KDC (krb5kdc).
Configuring kadmin
  [1/2]: starting kadmin 
  [2/2]: configuring kadmin to start on boot
Done configuring kadmin.
Configuring ipa_memcached
  [1/2]: starting ipa_memcached 
  [2/2]: configuring ipa_memcached to start on boot
Done configuring ipa_memcached.
Configuring the web interface (httpd): Estimated time 1 minute
  [1/14]: setting mod_nss port to 443
  [2/14]: setting mod_nss protocol list to TLSv1.0 - TLSv1.2
  [3/14]: setting mod_nss password file
  [4/14]: enabling mod_nss renegotiate
  [5/14]: adding URL rewriting rules
  [6/14]: configuring httpd
  [7/14]: setting up ssl
  [8/14]: importing CA certificates from LDAP
  [9/14]: publish CA cert
  [10/14]: creating a keytab for httpd
  [11/14]: clean up any existing httpd ccache
  [12/14]: configuring SELinux for httpd
  [13/14]: restarting httpd
  [14/14]: configuring httpd to start on boot
Done configuring the web interface (httpd).
Configuring ipa-otpd
  [1/2]: starting ipa-otpd 
  [2/2]: configuring ipa-otpd to start on boot
Done configuring ipa-otpd.
Applying LDAP updates
Restarting Directory server to apply updates
  [1/2]: stopping directory server
  [2/2]: starting directory server
Done.
Restarting the directory server
Restarting the KDC
Restarting the certificate server
Configuring DNS (named)
  [1/9]: generating rndc key file
  [2/9]: setting up reverse zone
  [3/9]: setting up our own record
  [4/9]: adding NS record to the zones
  [5/9]: setting up CA record
  [6/9]: setting up kerberos principal
  [7/9]: setting up named.conf
  [8/9]: configuring named to start on boot
  [9/9]: changing resolv.conf to point to ourselves
Done configuring DNS (named).

Restarting named

Global DNS configuration in LDAP server is empty
You can use 'dnsconfig-mod' command to set global DNS options that
would override settings in local named.conf files

Restarting the web server
:: [   PASS   ] :: Command ' /usr/sbin/ipa-replica-install -U --setup-ca --setup-dns --forwarder=10.65.201.89 -w xxxxxxxx -p xxxxxxxx /opt/rhqa_ipa/replica-info-dhcp207-133.testrelm.test.gpg' (Expected 0, got 0)

Changing the component to pki-core, as this is a pki-core bug.