Description of problem:
Fresh install of IPA server, when trying to make a replia of it, instllation fails at stage retrieving DS certificates from master server
Version-Release number of selected component (if applicable):
Rhel 7.3, IPA 4.4
Steps to Reproduce:
On server01 IdM Server installed ok (master) cmdline:
ipa-server-install --mkhomedir --no-ntp --no_hbac_allow --no-ssh --no-sshd --hostname=server01.example.com --domain=example.com --realm=EXAMPLE.COM --ds-password="XXXXX" --admin-password="YYYYY" -U
Created and defined user -hostregister- to enroll hosts
ipa domainlevel-get command gets 1
2. On client02 IdM Client installed ok, cmdline:
ipa-client-install --mkhomedir --no-krb5-offline-passwords --no-nisdomain --no-dns-sshfp --no-ssh --no-sshd --no-ntp --domain=example --server=client01.example.com --realm=EXAMPLE.COM --principal=hostregister --password=hostregister -U --force-join
3. On client02 Promote installed IdM Client to Replica server, cmdline:
ipa-replica-install --mkhomedir --no-ssh --no-sshd --no-ntp
4. Error on step [29/44] due to [28/44], ipa-replica-install logs:
[root@ulpldp02 ~]# kinit admin
Password for admin@APP10.OSAKIDETZA.EUS:
[root@ulpldp02 ~]# ipa-replica-install --mkhomedir --no-ssh --no-sshd --no-ntp
Run connection check to master
Connection check OK
Configuring directory server (dirsrv). Estimated time: 1 minute
[1/44]: creating directory server user
[2/44]: creating directory server instance
[3/44]: updating configuration in dse.ldif
[4/44]: restarting directory server
[5/44]: adding default schema
[6/44]: enabling memberof plugin
[7/44]: enabling winsync plugin
[8/44]: configuring replication version plugin
[9/44]: enabling IPA enrollment plugin
[10/44]: enabling ldapi
[11/44]: configuring uniqueness plugin
[12/44]: configuring uuid plugin
[13/44]: configuring modrdn plugin
[14/44]: configuring DNS plugin
[15/44]: enabling entryUSN plugin
[16/44]: configuring lockout plugin
[17/44]: configuring topology plugin
[18/44]: creating indices
[19/44]: enabling referential integrity plugin
[20/44]: configuring certmap.conf
[21/44]: configure autobind for root
[22/44]: configure new location for managed entries
[23/44]: configure dirsrv ccache
[24/44]: enabling SASL mapping fallback
[25/44]: restarting directory server
[26/44]: creating DS keytab
[27/44]: retrieving DS Certificate
[28/44]: restarting directory server
ipa : CRITICAL Failed to restart the directory server (Command '/bin/systemctl restart dirsrv@EXAMPLE.service' returned non-zero exit status 1). See the installation log for details.
[29/44]: setting up initial replication
[error] error: [Errno 111] Connection refused
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.
ipa.ipapython.install.cli.install_tool(Replica): ERROR [Errno 111] Connection refused
ipa.ipapython.install.cli.install_tool(Replica): ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
Replica install incomplete
Replica install complete
Hello can we get directory server error log and journalctl -xe why restart failed?
JFTR after IRC discussion: it looks that certmonger received CA_UNREACHABLE, ticket above was opened to improve error reporting.
Waiting for more logs.
Better error reporting has been fixed upstream, this doesn't fix the root cause
email@example.com should be listening on port ::1:8009.
The /etc/pki/pki-tomcat/server.xml configuration seems correct.
Use netstat/netcat to confirm that the port is open when Dogtag is running.
You can also restart Dogtag and look in /var/log/pki/pki-tomcat/catalina.<date>.log for a line like:
14-Dec-2016 12:26:30.618 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-0:0:0:0:0:0:0:1-8009"]
I'm interested if Flo's comments in comment #27 are applicable (IPv6
Does it make sense to check for presence of:
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
Before replica install?
In reply to comment #36 :
the issue happens on the replica when IPv6 is disabled on the *master*, not sure whether we can check this remotely.
True, sounds like another use case for http://www.freeipa.org/page/V4/Diagnostics_Tool
I'm not sure if this is pure misconfiguration from admin side or a more issue in some component. Either way it seems quite common. And therefore some fix would be good.
IPA shouldn't probably touch the file on update but PKI can at least report that it in more usable manner.
Or ideally a component which "owns" this part of /etc/hosts should update it. I don't know which on it is.
Moving to pki-core to consider reporting e.g. in self-test.
As discussed over IRC, it might be better if we could use a generic "localhost" address instead of "127.0.0.1" or "::1" so we don't have to change the /etc/hosts. We need someone to confirm that "localhost" would work in both IPv4 and IPv6 environments.
Suppose that works, we can fix the default AJP hostname configuration in PKI with this ticket: https://fedorahosted.org/pki/ticket/2570. Once that is fixed, IPA probably should revert the change made in bug #1081561. However, these changes will fix new installations only. Existing installations will still have the old "127.0.0.1" or "::1". Is that OK, or do we need to replace them with "localhost" automatically? If so, should that be done by PKI or IPA?
Endi, Flo tested setting 'localhost' and it solves the issue for new installs.
Flo will create IPA patch which changes for new installs (similar as in https://git.fedorahosted.org/cgit/freeipa.git/commit/?id=fa3b3193fabcaa37c2ba9865089fcfc06939c77f )
Whether to change the value also on upgrade is a question. If we would decide to change it on upgrade and given that the configuration value comes from IPA, it would make sense to request the change on IPA update.
Endi, is there a "nice" way which IPA could you to amend PKI config?
There is no "nice" way for IPA to amend the PKI config. You'll have to search for the `address="::1"` in /etc/pki/pki-tomcat/server.xml and replace with `address="localhost"`.
It could be done via a proper XML lib, but simple string search/replace will probably suffice.
Here's a sample script to replace an attribute in server.xml:
I think there are 2 options for IPA:
1. Override pki_ajp_host with "localhost" instead of "::1".
2. Wait until PKI ticket #2570 (bug #1413136) is fixed, then remove the pki_ajp_host override.
With option #1 IPA will be responsible to fix existing server.xml during upgrade. With option #2 it will be PKI's responsibility. Alternatively, we can leave server.xml unchanged and let the admin fix it manually if needed.
Which one would you prefer?
I'm returning this bug to IPA since the fix requires changes in IPA either way.
I'd prefer fixing on PKI side and removing this very specific config from IPA.
But it depends on timing. Is bug #1413136 something to be fixed in 7.3 batch update? If yes, then fix it on PKI side including updates, remove custom config from IPA. If not, then we might go with the IPA fix for 7.3, including update. And then for 7.4 we can remove the config from IPA installer and let it be completely a responsibility of PKI.
Bug #1413136 is already fixed for 7.4 and proposed for 7.3. The included upgrade script will automatically replace "127.0.0.1" and "::1" to "localhost" in existing servers, so all IPA needs to do is remove the pki_ajp_host override.
Created attachment 1244203 [details]
The submitted patch fixes the IPA-side configuration. The patch will be accepted upstream at a later point, when the new version of PKI is released.
Moving to POST According to comment 54
JFTR upstream fix
Fix pushed to ipa-4-4 branch
Created attachment 1279956 [details]
Verified on ipa-server-4.5.0-9.el7.x86_64
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.