Bug 698796

Summary: Race conditions during IPA installation
Product: Red Hat Enterprise Linux 6 Reporter: Dmitri Pal <dpal>
Component: pki-coreAssignee: Ade Lee <alee>
Status: CLOSED ERRATA QA Contact: Chandrasekar Kannan <ckannan>
Severity: high Docs Contact:
Priority: high    
Version: 6.1CC: awnuk, benl, jgalipea, jmagne, kchamart, mkosek
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 698885 (view as bug list) Environment:
Last Closed: 2011-12-06 16:29:01 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 698885    
Attachments:
Description Flags
patch to fix jmagne: review+

Description Dmitri Pal 2011-04-21 20:00:55 UTC
This issue is known to be observed in F15.

# ipa-server-install -p secret123 -a secret123
...
Configuring certificate server: Estimated time 6 minutes
  [1/17]: creating certificate server user
  [2/17]: creating pki-ca instance
  [3/17]: restarting certificate server
  [4/17]: configuring certificate server instance
  [5/17]: restarting certificate server
  [6/17]: creating CA agent PKCS#12 file in /root
  [7/17]: creating RA agent certificate database
  [8/17]: importing CA chain to RA certificate database
Unexpected error - see ipaserver-install.log for details:
 Unable to retrieve CA chain: request failed with HTTP status 500

From Ade:

Basically, one of the things that the CA does on startup is read some
configuration data in /etc/pki-ca/registry.cfg.  The way the code is
currently written, it reads in this data and then promptly overwrites
this file the data it has just read.  It actually does this multiple
times - once for each plugin it reads in.

During the IPA installation process, we end up starting up and shutting
down the CA multiple times.  In fact, one of these restarts is issued
while the server is coming up - and in particular - while this file is
being overwritten. The way tomcat code works, if the server is shut down
while  not completely up (ie. before the ports are available), then the
server will call System.exit() immediately.  This means that the
registry.cfg file is left in an incomplete, corrupted state - and any
subsequent restarts are incomplete.

The simple code fix is simply not to rewrite this config file on
startup.  There is no reason to do so - as we are just rewriting the
data we just read in.

I have coded a fix that addresses this - and was able to configure IPA
multiple times with no issue.

Comment 1 Dmitri Pal 2011-04-21 20:01:11 UTC
https://fedorahosted.org/freeipa/ticket/1186

Comment 2 Dmitri Pal 2011-04-21 20:01:47 UTC
https://fedorahosted.org/freeipa/ticket/1186

Comment 3 Ade Lee 2011-04-21 20:26:26 UTC
Created attachment 493983 [details]
patch to fix

Comment 4 Jack Magne 2011-04-21 21:00:48 UTC
Comment on attachment 493983 [details]
patch to fix

This fix looks pretty straightforward. It would be good to have awnuk to have a quick look in case he knows of any gotchas.

Comment 5 Ade Lee 2011-04-25 19:03:01 UTC
patch applied in tip in https://bugzilla.redhat.com/show_bug.cgi?id=698885

To be applied at the appropriate time in 8.1 branch (for 6.2).

Comment 6 Ade Lee 2011-04-26 01:23:48 UTC
Checked into 8.1:

[vakwetu@goofy-vm4 common]$ svn ci -m "Bugzilla Bug 698885 - Race conditions during IPA installation"
Sending        common/src/com/netscape/cmscore/registry/PluginRegistry.java
Transmitting file data .
Committed revision 1987.

Comment 7 Ade Lee 2011-08-04 17:07:47 UTC
Checked into 6.2:

[vakwetu@goofy-vm6 base]$ svn ci -m "Resolves #698796: Race conditions during IPA installation" 
Sending        base/common/src/com/netscape/cmscore/registry/PluginRegistry.java
Transmitting file data .
Committed revision 2103.

Comment 9 Kashyap Chamarthy 2011-11-08 19:37:00 UTC
VERIFIED.

##############################################################################
[root@neptune ~]# rpm -qi ipa-server
Name        : ipa-server                   Relocations: (not relocatable)
Version     : 2.1.4                             Vendor: (none)
Release     : 101.20111108T0212zgitc6d1996.el6   Build Date: Tue 08 Nov 2011 08:00:27 AM IST
##############################################################################
[root@neptune ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.2 Beta (Santiago)
[root@neptune ~]# 
##############################################################################
[root@neptune ~]# ipa-server-install --setup-dns -N

The log file for this installation can be found in /var/log/ipaserver-install.log
==============================================================================
This program will set up the IPA Server.

This includes:
  * Configure a stand-alone CA (dogtag) for certificate management
  * Create and configure an instance of Directory Server
  * Create and configure a Kerberos Key Distribution Center (KDC)
  * Configure Apache (httpd)
  * Configure DNS (bind)

Excluded by options:
  * Configure the Network Time Daemon (ntpd)

To accept the default shown in brackets, press the Enter key.

Existing BIND configuration detected, overwrite? [no]: yes
Enter the fully qualified domain name of the computer
on which you're setting up server software. Using the form
<hostname>.<domainname>
Example: master.example.com.


Server host name [neptune.lab.eng.pnq.redhat.com]: 

Warning: skipping DNS resolution of host neptune.lab.eng.pnq.redhat.com
The domain name has been calculated based on the host name.

Please confirm the domain name [lab.eng.pnq.redhat.com]: 

The IPA Master Server will be configured with
Hostname:    neptune.lab.eng.pnq.redhat.com
IP address:  10.65.201.47
Domain name: lab.eng.pnq.redhat.com

The kerberos protocol requires a Realm name to be defined.
This is typically the domain name converted to uppercase.

Please provide a realm name [LAB.ENG.PNQ.REDHAT.COM]: 
Certain directory server operations require an administrative user.
This user is referred to as the Directory Manager and has full access
to the Directory for system management tasks and will be added to the
instance of directory server created for IPA.
The password must be at least 8 characters long.

Directory Manager password: 
Password (confirm): 
Password mismatch!

Directory Manager password: 
Password (confirm): 
Password mismatch!

Directory Manager password: 
Password (confirm): 

The IPA server requires an administrative user, named 'admin'.
This user is a regular system account used for IPA server administration.

IPA admin password: 
Password (confirm): 

Do you want to configure DNS forwarders? [yes]: 
Enter the IP address of DNS forwarder to use, or press Enter to finish.
Enter IP address for a DNS forwarder: 10.65.201.47
DNS forwarder 10.65.201.47 added
Enter IP address for a DNS forwarder: 
Do you want to configure the reverse zone? [yes]: 
Please specify the reverse zone name [201.65.10.in-addr.arpa.]: 
Using reverse zone 201.65.10.in-addr.arpa.

The following operations may take some minutes to complete.
Please wait until the prompt is returned.

Configuring directory server for the CA: Estimated time 30 minutes 30 seconds
  [1/3]: creating directory server user
  [2/3]: creating directory server instance
  [3/3]: restarting directory server
done configuring pkids.
Configuring certificate server: Estimated time 33 minutes 30 seconds
  [1/17]: creating certificate server user
  [2/17]: creating pki-ca instance
  [3/17]: configuring certificate server instance
  [4/17]: disabling nonces
  [5/17]: creating CA agent PKCS#12 file in /root
  [6/17]: creating RA agent certificate database
  [7/17]: importing CA chain to RA certificate database
  [8/17]: fixing RA database permissions
  [9/17]: setting up signing cert profile
  [10/17]: set up CRL publishing
  [11/17]: set certificate subject base
  [12/17]: configuring certificate server to start on boot
  [13/17]: restarting certificate server
  [14/17]: requesting RA certificate from CA
  [15/17]: issuing RA agent certificate
  [16/17]: adding RA agent as a trusted user
  [17/17]: Configure HTTP to proxy connections
done configuring pki-cad.
Configuring directory server: Estimated time 31 minutes
  [1/35]: creating directory server user
  [2/35]: creating directory server instance
  [3/35]: adding default schema
  [4/35]: enabling memberof plugin
  [5/35]: enabling referential integrity plugin
  [6/35]: enabling winsync plugin
  [7/35]: configuring replication version plugin
  [8/35]: enabling IPA enrollment plugin
  [9/35]: enabling ldapi
  [10/35]: configuring uniqueness plugin
  [11/35]: configuring uuid plugin
  [12/35]: configuring modrdn plugin
  [13/35]: enabling entryUSN plugin
  [14/35]: configuring lockout plugin
  [15/35]: creating indices
  [16/35]: configuring ssl for ds instance
  [17/35]: configuring certmap.conf
  [18/35]: configure autobind for root
  [19/35]: configure new location for managed entries
  [20/35]: restarting directory server
  [21/35]: adding default layout
  [22/35]: adding delegation layout
  [23/35]: adding replication acis
  [24/35]: creating container for managed entries
  [25/35]: configuring user private groups
  [26/35]: configuring netgroups from hostgroups
  [27/35]: creating default Sudo bind user
  [28/35]: creating default Auto Member layout
  [29/35]: creating default HBAC rule allow_all
  [30/35]: initializing group membership
  [31/35]: adding master entry
  [32/35]: configuring Posix uid/gid generation
  [33/35]: enabling compatibility plugin
Restarting IPA to initialize updates before performing deletes:
  [1/2]: stopping directory server
  [2/2]: starting directory server
done configuring dirsrv.
  [34/35]: tuning directory server
  [35/35]: configuring directory to start on boot
done configuring dirsrv.
Configuring Kerberos KDC: Estimated time 30 minutes 30 seconds
  [1/14]: setting KDC account password
  [2/14]: adding sasl mappings to the directory
  [3/14]: adding kerberos entries to the DS
  [4/14]: adding default ACIs
  [5/14]: configuring KDC
  [6/14]: adding default keytypes
  [7/14]: adding default password policy
  [8/14]: creating a keytab for the directory
  [9/14]: creating a keytab for the machine
  [10/14]: exporting the kadmin keytab
  [11/14]: adding the password extension to the directory
  [12/14]: adding the kerberos master key to the directory
  [13/14]: starting the KDC
  [14/14]: configuring KDC to start on boot
done configuring krb5kdc.
Configuring ipa_kpasswd
  [1/2]: starting ipa_kpasswd 
  [2/2]: configuring ipa_kpasswd to start on boot
done configuring ipa_kpasswd.
Configuring the web interface: Estimated time 31 minutes
  [1/13]: disabling mod_ssl in httpd
  [2/13]: setting mod_nss port to 443
  [3/13]: setting mod_nss password file
  [4/13]: enabling mod_nss renegotiate
  [5/13]: adding URL rewriting rules
  [6/13]: configuring httpd
  [7/13]: setting up ssl
  [8/13]: setting up browser autoconfig
  [9/13]: publish CA cert
  [10/13]: creating a keytab for httpd
  [11/13]: configuring SELinux for httpd
  [12/13]: restarting httpd
  [13/13]: configuring httpd to start on boot
done configuring httpd.
Applying LDAP updates
Restarting IPA to initialize updates before performing deletes:
  [1/2]: stopping directory server
  [2/2]: starting directory server
done configuring dirsrv.
Restarting the directory server
Restarting the KDC
Restarting the web server
Configuring named:
  [1/9]: adding DNS container
  [2/9]: setting up our zone
  [3/9]: setting up reverse zone
  [4/9]: setting up our own record
  [5/9]: setting up kerberos principal
  [6/9]: setting up named.conf
  [7/9]: restarting named
  [8/9]: configuring named to start on boot
  [9/9]: changing resolv.conf to point to ourselves
done configuring named.
==============================================================================
Setup complete

Next steps:
        1. You must make sure these network ports are open:
                TCP Ports:
                  * 80, 443: HTTP/HTTPS
                  * 389, 636: LDAP/LDAPS
                  * 88, 464: kerberos
                  * 53: bind
                UDP Ports:
                  * 88, 464: kerberos
                  * 53: bind

        2. You can now obtain a kerberos ticket using the command: 'kinit admin'
           This ticket will allow you to use the IPA tools (e.g., ipa user-add)
           and the web user interface.
        3. Kerberos requires time synchronization between clients
           and servers for correct operation. You should consider enabling ntpd.

Be sure to back up the CA certificate stored in /root/cacert.p12
This file is required to create replicas. The password for this
file is the Directory Manager password
[root@neptune ~]# 
##############################################################################

Comment 10 errata-xmlrpc 2011-12-06 16:29:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2011-1655.html