Bug 688934
| Summary: | Wait for Directory Service ports to open | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Rob Crittenden <rcritten> |
| Component: | ipa | Assignee: | Rob Crittenden <rcritten> |
| Status: | CLOSED ERRATA | QA Contact: | Chandrasekar Kannan <ckannan> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 6.1 | CC: | benl, ddumas, dpal, jgalipea, mkosek, nsoman, syeghiay |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | ipa-2.0.0-17.el6 | Doc Type: | Bug Fix |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2011-05-19 13:44:44 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
master: 18542cd165d047cba69ed2b3ac12e59993bf2fb0 Please add steps to reproduce this issue. Thanks! This was an intermittent problem seen while trying to install IPA. It was more common on busy and/or low-powered machines. So you can try limiting memory to cause swapping in a VM. This issue is difficult to reproduce, but I suggest using this approach:
1. Edit /etc/init.d/dirsrv and temporarily block port 389 used by ldapmodify to access the Directory Server:
...
restart() {
stop
(iptables -I INPUT -p tcp --dport 389 -j REJECT; sleep 10; iptables -F) &
start
}
...
2. Start the standard IPA installation
Unpatched version should fail with a lot of ldapmodify error messages like the one above. Pathed version should wait until the port 389 is open (10 seconds) and then continue with the installation.
Having trouble reproducing the issue on an older version of ipa install.
Using: ipa-server-2.0.0-16.el6.x86_64
First edited /etc/init.d/dirsrv as suggested above.
Installed successfully, no errors in install log.
Next tried few attempts changing sleep to 60, 100, 300.
Also tried with --selfsign
Still was able to successfully install ipa server.
Was in touch with mkosek to get steps to see the problem.
He suggested editing it - so that now my section looks like below:
restart() {
stop
iptables -I INPUT -p tcp --dport 389 -j REJECT
start
}
And then when the install hangs, to run iptables -F
Saw the install fail, with no hangs and the errors when using ipa-server-2.0.0-16.el6.x86_64. With no hangs - didn't run iptables -F
Assuming - this reproduced the issue...moved to another machine using ipa-server-2.0.0-21.el6.x86_64, where this is fixed.
Tried this thrice with the new install, but install never hung. So didn't see when would be the right time to run iptables -F to unblock the port.
There are issues with the approach I posted above. Additionally, the patched IPA installer does not wait for open ports after *every* dirsrv restart, but only in few cases (including the reported case). This shall be fixed in the IPA upstream.
You can use the following updated /etc/init.d/dirsrv to simulate the reported issue:
restart() {
stop
# BZ 688934 reproducer START
IPA_PRE_TUNING_PHASE=`grep limits.conf /var/lib/ipa/sysrestore/sysrestore.index`
IPA_POST_TUNING_PHASE=`grep ldappwd /var/lib/ipa/sysrestore/sysrestore.index`
# IPA installator is now in step [31/32]: tuning directory server
if [ -n "$IPA_PRE_TUNING_PHASE" -a -z "$IPA_POST_TUNING_PHASE" ]; then
# block DS port
iptables -I INPUT -p tcp --dport 389 -j REJECT
fi
# BZ 688934 reproducer END
start
}
This will block the DS port only for the reported case, i.e. step "[31/32]: tuning directory server" in the IPA installation procedure.
Reproducing steps:
1) Update /etc/init.d/dirsrv as written above
2) Start IPA installation (--selfsign option may be used for faster test)
3) When the installer reaches step "[31/32]: tuning directory server", the IPA should:
a) For unpatched version: print the reported error message, installation fails
b) Patched version should get into a waiting loop until the port is unblocked. Use "iptables -F" to unblock the port before 300 seconds timeout. Installation should succeed then.
Followed the steps above, and saw the install fail in old server. Then did the same - saw the install hang with new server, and after unblocking ports, install continued. Also, tried install...leaving tghe port blocked, and install failed as expected. But this fix addresses the port not being available at step [31/32]. When the port is not available at other times of install, it still doesn't wait for the port to open, and install fails. Opened bug 697878 to address the other cases when the installs fails, and doesn't wait for port to open. Marking this verified, since the step this was reported for, has been fixed. An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHEA-2011-0631.html |
Description of problem: Sometimes there is a timing issue in IPA installation. When Directory Server instance is restarted and then there are LDAP operations started, the LDAP listening ports may not be open yet. This leads to errors like the following one: $ sudo ipa-server-install ... [30/32]: enabling compatibility plugin [31/32]: tuning directory server root : CRITICAL Failed to load ds-nfiles.ldif: Command '/usr/bin/ldapmodify -h vm-077.idm.lab.bos.redhat.com -v -f /tmp/tmpyhvNAE -x -D cn=Directory Manager -y /tmp/tmpgRvDSm' returned non-zero exit status 255 [32/32]: configuring directory to start on boot done configuring dirsrv. Unexpected error - see ipaserver-install.log for details: {'desc': "Can't contact LDAP server"} This could be solved by implementing a wait for open listening ports to function DsInstance::__restart_instance (like it is in CaInstance::__restart_instance). A timeout would be convenient. It may be set to some safe value, for example 5 mins. A common function implementing the open-port wait used in both CaInstance and DsInstance would be convenient.