Bug 1398600

Summary: IPA replica install fails with dirsrv errors.
Product: Red Hat Enterprise Linux 7 Reporter: Manu Augustine <maugusti>
Component: ipaAssignee: Pavel Picka <ppicka>
Status: CLOSED ERRATA QA Contact: Pavel Picka <ppicka>
Severity: high Docs Contact:
Priority: high    
Version: 7.3CC: afarley, aheverle, apitanga, edewata, ekeck, frenaud, ftweedal, gparente, hkhot, ipa-qe, jcholast, jreznik, maugusti, mbabinsk, mbasti, nsoman, olivier.raggi, pvoborni, rcritten, sauchter, tkrizek, tscherf
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: ipa-4.4.0-14.el7.5 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1416481 (view as bug list) Environment:
Last Closed: 2017-08-01 09:42:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1413136    
Bug Blocks: 1416481    
Attachments:
Description Flags
IPA patch
none
output none

Description Manu Augustine 2016-11-25 11:49:34 UTC
Description of problem:

Fresh install of IPA server, when trying to make a replia of it, instllation fails at stage retrieving DS certificates from master server


Version-Release number of selected component (if applicable):
Rhel 7.3, IPA 4.4

How reproducible:


Steps to Reproduce:
 On server01 IdM Server installed ok (master) cmdline:
====
ipa-server-install --mkhomedir --no-ntp --no_hbac_allow --no-ssh --no-sshd --hostname=server01.example.com --domain=example.com --realm=EXAMPLE.COM --ds-password="XXXXX" --admin-password="YYYYY" -U

Created and defined user -hostregister- to enroll hosts
ipa domainlevel-get command gets 1

2. On client02 IdM Client installed ok, cmdline:

====
ipa-client-install --mkhomedir --no-krb5-offline-passwords --no-nisdomain --no-dns-sshfp --no-ssh --no-sshd --no-ntp --domain=example --server=client01.example.com --realm=EXAMPLE.COM --principal=hostregister --password=hostregister -U --force-join

3. On client02 Promote installed IdM Client to Replica server, cmdline:
====
kinit admin
ipa-replica-install --mkhomedir --no-ssh --no-sshd --no-ntp

4. Error on step [29/44] due to [28/44], ipa-replica-install logs:
====
[root@ulpldp02 ~]# kinit admin
Password for admin.EUS:
[root@ulpldp02 ~]# ipa-replica-install --mkhomedir --no-ssh --no-sshd --no-ntp
Run connection check to master
Connection check OK
Configuring directory server (dirsrv). Estimated time: 1 minute
  [1/44]: creating directory server user
  [2/44]: creating directory server instance
  [3/44]: updating configuration in dse.ldif
  [4/44]: restarting directory server
  [5/44]: adding default schema
  [6/44]: enabling memberof plugin
  [7/44]: enabling winsync plugin
  [8/44]: configuring replication version plugin
  [9/44]: enabling IPA enrollment plugin
  [10/44]: enabling ldapi
  [11/44]: configuring uniqueness plugin
  [12/44]: configuring uuid plugin
  [13/44]: configuring modrdn plugin
  [14/44]: configuring DNS plugin
  [15/44]: enabling entryUSN plugin
  [16/44]: configuring lockout plugin
  [17/44]: configuring topology plugin
  [18/44]: creating indices
  [19/44]: enabling referential integrity plugin
  [20/44]: configuring certmap.conf
  [21/44]: configure autobind for root
  [22/44]: configure new location for managed entries
  [23/44]: configure dirsrv ccache
  [24/44]: enabling SASL mapping fallback
  [25/44]: restarting directory server
  [26/44]: creating DS keytab
  [27/44]: retrieving DS Certificate
  [28/44]: restarting directory server
ipa         : CRITICAL Failed to restart the directory server (Command '/bin/systemctl restart dirsrv' returned non-zero exit status 1). See the installation log for details.
  [29/44]: setting up initial replication
  [error] error: [Errno 111] Connection refused
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

ipa.ipapython.install.cli.install_tool(Replica): ERROR    [Errno 111] Connection refused
ipa.ipapython.install.cli.install_tool(Replica): ERROR    The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information


Actual results:
Replica install incomplete

Expected results:
Replica install complete

Additional info:

Comment 2 Martin Bašti 2016-11-25 11:56:04 UTC
Hello can we get directory server error log and journalctl -xe why restart failed?

Comment 6 Martin Bašti 2016-11-25 12:41:20 UTC
Upstream ticket:
https://fedorahosted.org/freeipa/ticket/6514

Comment 7 Martin Bašti 2016-11-25 12:53:25 UTC
JFTR after IRC discussion: it looks that certmonger received CA_UNREACHABLE, ticket above was opened to improve error reporting.

Waiting for more logs.

Comment 18 Martin Bašti 2016-11-30 15:35:27 UTC
Better error reporting has been fixed upstream, this doesn't fix the root cause
master:
https://fedorahosted.org/freeipa/changeset/dbb98765d73519289ee22f3de1a5ccde140f6f5d

Comment 29 Fraser Tweedale 2016-12-14 02:29:00 UTC
pki-tomcatd should be listening on port ::1:8009.
The /etc/pki/pki-tomcat/server.xml configuration seems correct.
Use netstat/netcat to confirm that the port is open when Dogtag is running.

You can also restart Dogtag and look in /var/log/pki/pki-tomcat/catalina.<date>.log for a line like:

   14-Dec-2016 12:26:30.618 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-0:0:0:0:0:0:0:1-8009"]

I'm interested if Flo's comments in comment #27 are applicable (IPv6
disabled).

Comment 36 Petr Vobornik 2016-12-15 14:46:11 UTC
Does it make sense to check for presence of:

/etc/hosts
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

Before replica install?

Comment 39 Florence Blanc-Renaud 2016-12-16 15:00:41 UTC
In reply to comment #36 :
the issue happens on the replica when IPv6 is disabled on the *master*, not sure whether we can check this remotely.

Comment 40 Petr Vobornik 2016-12-16 15:53:11 UTC
True, sounds like another use case for http://www.freeipa.org/page/V4/Diagnostics_Tool

Comment 42 Petr Vobornik 2016-12-20 11:51:43 UTC
Upstream ticket:
https://fedorahosted.org/freeipa/ticket/6575

Comment 43 Petr Vobornik 2017-01-02 16:42:29 UTC
I'm not sure if this is pure misconfiguration from admin side or a more issue in some component. Either way it seems quite common. And therefore some fix would be good.

IPA shouldn't probably touch the file on update but PKI can at least report that it in more usable manner. 

Or ideally a component which "owns" this part of /etc/hosts should update it. I don't know which on it is.

Moving to pki-core to consider reporting e.g. in self-test.

Comment 44 Endi Sukma Dewata 2017-01-03 17:21:55 UTC
Petr,

As discussed over IRC, it might be better if we could use a generic "localhost" address instead of "127.0.0.1" or "::1" so we don't have to change the /etc/hosts. We need someone to confirm that "localhost" would work in both IPv4 and IPv6 environments.

Suppose that works, we can fix the default AJP hostname configuration in PKI with this ticket: https://fedorahosted.org/pki/ticket/2570. Once that is fixed, IPA probably should revert the change made in bug #1081561. However, these changes will fix new installations only. Existing installations will still have the old "127.0.0.1" or "::1". Is that OK, or do we need to replace them with "localhost" automatically? If so, should that be done by PKI or IPA?

Comment 46 Petr Vobornik 2017-01-10 16:24:08 UTC
Endi, Flo tested setting 'localhost' and it solves the issue for new installs.

Flo will create IPA patch which changes for new installs (similar as in https://git.fedorahosted.org/cgit/freeipa.git/commit/?id=fa3b3193fabcaa37c2ba9865089fcfc06939c77f )

Whether to change the value also on upgrade is a question. If we would decide to change it on upgrade and given that the configuration value comes from IPA, it would make sense to request the change on IPA update.

Endi, is there a "nice" way which IPA could you to amend PKI config?

Comment 48 Fraser Tweedale 2017-01-11 01:19:45 UTC
Petr, Flo:

There is no "nice" way for IPA to amend the PKI config.  You'll have to search for the `address="::1"` in /etc/pki/pki-tomcat/server.xml and replace with `address="localhost"`.

It could be done via a proper XML lib, but simple string search/replace will probably suffice.

Comment 49 Endi Sukma Dewata 2017-01-13 16:20:13 UTC
Here's a sample script to replace an attribute in server.xml:
https://git.fedorahosted.org/cgit/pki.git/tree/base/server/upgrade/10.2.2/02-EnableWebApplicationAutoDeploy

Comment 51 Endi Sukma Dewata 2017-01-13 18:01:23 UTC
Petr,

I think there are 2 options for IPA:

1. Override pki_ajp_host with "localhost" instead of "::1".
2. Wait until PKI ticket #2570 (bug #1413136) is fixed, then remove the pki_ajp_host override.

With option #1 IPA will be responsible to fix existing server.xml during upgrade. With option #2 it will be PKI's responsibility. Alternatively, we can leave server.xml unchanged and let the admin fix it manually if needed.

Which one would you prefer?

I'm returning this bug to IPA since the fix requires changes in IPA either way.

Comment 52 Petr Vobornik 2017-01-16 11:04:53 UTC
I'd prefer fixing on PKI side and removing this very specific config from IPA. 

But it depends on timing. Is bug #1413136 something to be fixed in 7.3 batch update? If yes, then fix it on PKI side including updates, remove custom config from IPA. If not, then we might go with the IPA fix for 7.3, including update. And then for 7.4 we can remove the config from IPA installer and let it be completely a responsibility of PKI.

Comment 53 Endi Sukma Dewata 2017-01-20 14:33:32 UTC
Bug #1413136 is already fixed for 7.4 and proposed for 7.3. The included upgrade script will automatically replace "127.0.0.1" and "::1" to "localhost" in existing servers, so all IPA needs to do is remove the pki_ajp_host override.

Comment 54 Tomas Krizek 2017-01-25 10:12:06 UTC
Created attachment 1244203 [details]
IPA patch

The submitted patch fixes the IPA-side configuration. The patch will be accepted upstream at a later point, when the new version of PKI is released.

Comment 55 Petr Vobornik 2017-01-25 14:13:00 UTC
Moving to POST According to comment 54

Comment 58 Martin Bašti 2017-02-17 14:00:26 UTC
JFTR upstream fix
master:
https://fedorahosted.org/freeipa/changeset/eaa87c75b9f57500265b2dc9480b996b2b92e1e3

Comment 59 Martin Babinsky 2017-02-20 09:27:16 UTC
Fix pushed to ipa-4-4 branch
ipa-4-4:
https://fedorahosted.org/freeipa/changeset/4a30e9d53475d60fb76242a098f1d969d6b19f75

Comment 65 Pavel Picka 2017-05-18 10:38:24 UTC
Created attachment 1279956 [details]
output

Verified on ipa-server-4.5.0-9.el7.x86_64

Comment 66 errata-xmlrpc 2017-08-01 09:42:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2304