Upstream ticket:
https://fedorahosted.org/freeipa/ticket/2965

Scott, does /var/run/httpd/httpd.pid exist?

I'm assuming that service httpd stop also fails?

Can you get the following:
- rpm -V httpd
- ls -l /etc/httpd
- ls -ldZ /var/run/httpd
- ls -ld /var/run/httpd
- grep -e ^ServerRoot -e ^PidFile /etc/httpd/conf/httpd.conf

I'm assuming that once a server gets into this state you can never get IPA installed on it?

httpd.pid's gone by the time I see this and can check but, I think it was used to kill the daemon/controller process.   That's what I'm guessing from all the orphans with PPID of 1 anyway.  Does that not make sense?


[root@ipaqavmd ~]# rpm -V httpd
[root@ipaqavmd ~]# echo $?
0
[root@ipaqavmd ~]# ls -l /etc/httpd
total 12
drwxr-xr-x. 2 root root 4096 Aug 21 09:49 alias
drwxr-xr-x. 2 root root 4096 Aug 21 09:49 conf
drwxr-xr-x. 2 root root 4096 Aug 21 09:49 conf.d
lrwxrwxrwx. 1 root root   19 Aug 21 09:40 logs -> ../../var/log/httpd
lrwxrwxrwx. 1 root root   27 Aug 21 09:40 modules -> ../../usr/lib/httpd/modules
lrwxrwxrwx. 1 root root   19 Aug 21 09:40 run -> ../../var/run/httpd
[root@ipaqavmd ~]# 
[root@ipaqavmd ~]# ls -ldZ /var/run/httpd
drwx--x---. root apache system_u:object_r:httpd_var_run_t:s0 /var/run/httpd
[root@ipaqavmd ~]# ls -ld /var/run/httpd
drwx--x---. 2 root apache 4096 Aug 21 09:50 /var/run/httpd
[root@ipaqavmd ~]# ls -ld /var/run/httpd/httpd.pid
ls: cannot access /var/run/httpd/httpd.pid: No such file or directory
[root@ipaqavmd ~]# grep -e ^ServerRoot -e ^PidFile /etc/httpd/conf/httpd.conf
ServerRoot "/etc/httpd"
PidFile run/httpd.pid

Also, it does look like I can uninstall/reinstall and it works.

on replica: ipa-server-install --uninstall -U
on master: ipa-replica-manage del $REPLICA --force
on master: ipa-replica-prepare -p $ADMINPW --ip-address=$SLAVE_IP $SLAVE_S.$DOMAIN
on replica: sftp $MASTER:/var/lib/ipa/replica-info-$SLAVEFQDN.gpg /dev/shm
on replica: ipa-replica-install -U --setup-dns --forwarder=$DNSFORWARD -w $ADMINPW -p $ADMINPW /dev/shm/replica-info-$SLAVEFQDN.gpg

Interestingly enough, I just saw this again on RHEL6.4 ipa-server-install (not replica)

Configuring the web interface: Estimated time 1 minute
  [1/14]: disabling mod_ssl in httpd
  [2/14]: setting mod_nss port to 443
  [3/14]: setting mod_nss password file
  [4/14]: enabling mod_nss renegotiate
  [5/14]: adding URL rewriting rules
  [6/14]: configuring httpd
  [7/14]: setting up ssl
  [8/14]: setting up browser autoconfig
  [9/14]: publish CA cert
  [10/14]: creating a keytab for httpd
  [11/14]: clean up any existing httpd ccache
  [12/14]: configuring SELinux for httpd
  [13/14]: restarting httpd
Unexpected error - see /var/log/ipaserver-install.log for details:
CalledProcessError: Command '/sbin/service httpd restart ' returned non-zero exit status 1
2012-09-25T22:09:23Z DEBUG   [13/14]: restarting httpd
2012-09-25T22:09:24Z DEBUG args=/sbin/service httpd status 
2012-09-25T22:09:24Z DEBUG stdout=httpd is stopped

2012-09-25T22:09:24Z DEBUG stderr=
2012-09-25T22:09:24Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2012-09-25T22:09:37Z DEBUG args=/sbin/service httpd restart 
2012-09-25T22:09:37Z DEBUG stdout=Stopping httpd:          [FAILED]
Starting httpd:                                            [FAILED]

2012-09-25T22:09:37Z DEBUG stderr=[Tue Sep 25 17:09:36 2012] [warn] worker ajp://localhost:9447 already used by another worker
[Tue Sep 25 17:09:36 2012] [warn] worker ajp://localhost:9447 already used by another worker

2012-09-25T22:09:37Z INFO   File "/usr/lib/python2.6/site-packages/ipaserver/install/installutils.py", line 614, in run_script
    return_value = main_function()

  File "/usr/sbin/ipa-server-install", line 1033, in main
    http.create_instance(realm_name, host_name, domain_name, dm_password, autoconfig=True, self_signed_ca=options.selfsign, subject_base=options.subject, auto_redirect=options.ui_redirect)

  File "/usr/lib/python2.6/site-packages/ipaserver/install/httpinstance.py", line 102, in create_instance
    self.start_creation("Configuring the web interface", 60)

  File "/usr/lib/python2.6/site-packages/ipaserver/install/service.py", line 321, in start_creation
    method()

  File "/usr/lib/python2.6/site-packages/ipaserver/install/httpinstance.py", line 106, in __start
    self.restart()

  File "/usr/lib/python2.6/site-packages/ipaserver/install/service.py", line 267, in restart
    self.service.restart(instance_name, capture_output=capture_output, wait=wait)

  File "/usr/lib/python2.6/site-packages/ipapython/platform/redhat.py", line 74, in restart
    ipautil.run(["/sbin/service", self.service_name, "restart", instance_name], capture_output=capture_output)

  File "/usr/lib/python2.6/site-packages/ipapython/ipautil.py", line 307, in run
    raise CalledProcessError(p.returncode, args)

2012-09-25T22:09:37Z INFO The ipa-server-install command failed, exception: CalledProcessError: Command '/sbin/service httpd restart ' returned non-zero exit status 1

I filed httpd Bug for RHEL 6.4 so we can get some response from httpd maintenance team: Bug 867467.

Bug 867467 has been closed as NEXTRELEASE. You can check Joe Orton's response with reasoning there. Bottomline is that with sysV scripts cannot guarantee that httpd will always restart successfully when restarted in such a short notice. This should be fixed with systemd.

As for 6.4, I think this will instead lead to a patch to enforce a wait+fallback in case httpd restart fails.

Went with a wait+fallback in case httpd restart fails.

Fixed upstream.

master: 9126b18d8bbf7d8f0f8981deb84f80f2eaa27bfe

ipa-3-0: 5f95fbfad2729e59d67730d2d476befa4e0a2e02

Verified. 

Version ::

ipa-server-3.0.0-10.el6.i686
ipa-client-3.0.0-10.el6.i686

Manual Test Results ::

I have not seen this re-occur on RHEL6.4 tests on x86_64 servers.  Testing here on i686:

[root@rhel6-5 ~]# ipa-replica-install -U --setup-dns --forwarder=$DNSFORWARD -w $ADMINPW -p $ADMINPW /dev/shm/replica-info-rhel6-5.testrelm.com.gpg 
Run connection check to master
Check connection from replica to remote master 'rhel6-1.testrelm.com':
   Directory Service: Unsecure port (389): OK
   Directory Service: Secure port (636): OK
   Kerberos KDC: TCP (88): OK
   Kerberos Kpasswd: TCP (464): OK
   HTTP Server: Unsecure port (80): OK
   HTTP Server: Secure port (443): OK

The following list of ports use UDP protocol and would need to be
checked manually:
   Kerberos KDC: UDP (88): SKIPPED
   Kerberos Kpasswd: UDP (464): SKIPPED

Connection from replica to master is OK.
Start listening on required ports for remote master check
Get credentials to log in to remote master
Execute check on remote master
Check connection from master to remote replica 'rhel6-5.testrelm.com':
   Directory Service: Unsecure port (389): OK
   Directory Service: Secure port (636): OK
   Kerberos KDC: TCP (88): OK
   Kerberos KDC: UDP (88): OK
   Kerberos Kpasswd: TCP (464): OK
   Kerberos Kpasswd: UDP (464): OK
   HTTP Server: Unsecure port (80): OK
   HTTP Server: Secure port (443): OK

Connection from master to replica is OK.

Connection check OK
Configuring NTP daemon (ntpd)
  [1/4]: stopping ntpd
  [2/4]: writing configuration
  [3/4]: configuring ntpd to start on boot
  [4/4]: starting ntpd
Done configuring NTP daemon (ntpd).
Configuring directory server (dirsrv): Estimated time 1 minute
  [1/31]: creating directory server user
  [2/31]: creating directory server instance
  [3/31]: adding default schema
  [4/31]: enabling memberof plugin
  [5/31]: enabling winsync plugin
  [6/31]: configuring replication version plugin
  [7/31]: enabling IPA enrollment plugin
  [8/31]: enabling ldapi
  [9/31]: disabling betxn plugins
  [10/31]: configuring uniqueness plugin
  [11/31]: configuring uuid plugin
  [12/31]: configuring modrdn plugin
  [13/31]: enabling entryUSN plugin
  [14/31]: configuring lockout plugin
  [15/31]: creating indices
  [16/31]: enabling referential integrity plugin
  [17/31]: configuring ssl for ds instance
  [18/31]: configuring certmap.conf
  [19/31]: configure autobind for root
  [20/31]: configure new location for managed entries
  [21/31]: restarting directory server
  [22/31]: setting up initial replication
Starting replication, please wait until this has completed.
Update in progress
Update in progress
Update in progress
Update in progress
Update succeeded
  [23/31]: adding replication acis
  [24/31]: setting Auto Member configuration
  [25/31]: enabling S4U2Proxy delegation
  [26/31]: initializing group membership
  [27/31]: adding master entry
  [28/31]: configuring Posix uid/gid generation
  [29/31]: enabling compatibility plugin
  [30/31]: tuning directory server
  [31/31]: configuring directory to start on boot
Done configuring directory server (dirsrv).
Configuring Kerberos KDC (krb5kdc): Estimated time 30 seconds
  [1/9]: adding sasl mappings to the directory
  [2/9]: writing stash file from DS
  [3/9]: configuring KDC
  [4/9]: creating a keytab for the directory
  [5/9]: creating a keytab for the machine
  [6/9]: adding the password extension to the directory
  [7/9]: enable GSSAPI for replication
  [8/9]: starting the KDC
  [9/9]: configuring KDC to start on boot
Done configuring Kerberos KDC (krb5kdc).
Configuring kadmin
  [1/2]: starting kadmin 
  [2/2]: configuring kadmin to start on boot
Done configuring kadmin.
Configuring ipa_memcached
  [1/2]: starting ipa_memcached 
  [2/2]: configuring ipa_memcached to start on boot
Done configuring ipa_memcached.
Configuring the web interface (httpd): Estimated time 1 minute
  [1/13]: disabling mod_ssl in httpd
  [2/13]: setting mod_nss port to 443
  [3/13]: setting mod_nss password file
  [4/13]: enabling mod_nss renegotiate
  [5/13]: adding URL rewriting rules
  [6/13]: configuring httpd
  [7/13]: setting up ssl
  [8/13]: publish CA cert
  [9/13]: creating a keytab for httpd
  [10/13]: clean up any existing httpd ccache
  [11/13]: configuring SELinux for httpd
  [12/13]: restarting httpd
  [13/13]: configuring httpd to start on boot
Done configuring the web interface (httpd).
Applying LDAP updates
Restarting the directory server
Restarting the KDC
Using reverse zone 122.168.192.in-addr.arpa.
Configuring DNS (named)
  [1/8]: adding NS record to the zone
  [2/8]: setting up reverse zone
  [3/8]: setting up our own record
  [4/8]: setting up kerberos principal
  [5/8]: setting up named.conf
  [6/8]: restarting named
  [7/8]: configuring named to start on boot
  [8/8]: changing resolv.conf to point to ourselves
Done configuring DNS (named).

Global DNS configuration in LDAP server is empty
You can use 'dnsconfig-mod' command to set global DNS options that
would override settings in local named.conf files

Restarting the web server

[root@rhel6-5 ~]# grep "Command '/sbin/service httpd restart ' returned non-zero exit status 1" /var/log/ipareplica-install.log

[root@rhel6-5 ~]#

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0528.html