Bug 1306989 - Crash seen with pki-common pkg during IPA server install
Crash seen with pki-common pkg during IPA server install
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: pki-core (Show other bugs)
Unspecified Unspecified
unspecified Severity high
: rc
: ---
Assigned To: Ade Lee
Asha Akkiangady
Aneta Šteflová Petrová
: Regression
Depends On:
  Show dependency treegraph
Reported: 2016-02-12 06:40 EST by Kaleem
Modified: 2016-05-10 20:32 EDT (History)
10 users (show)

See Also:
Fixed In Version: pki-core-9.0.3-48.el6
Doc Type: Bug Fix
Doc Text:
The ipa-server-install utility no longer terminates unexpectedly due to unexpected comment lines in CS.cfg An attempt to install an Identity Management server previously sometimes failed due to a problem with the _pki-common_ package. The fail occurred because the `CS.cfg` certificate authority (CA) configuration file which was being parsed contained unexpected comment lines before configuration. This problem has been fixed by making the parsing code ignore comment and blank lines.
Story Points: ---
Clone Of:
Last Closed: 2016-05-10 20:32:58 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
mail text received for crash (69.01 KB, text/plain)
2016-02-12 06:40 EST, Kaleem
no flags Details
crash dump (1.44 MB, application/x-gzip)
2016-02-12 06:43 EST, Kaleem
no flags Details
CS.cfg (75.87 KB, text/plain)
2016-02-22 02:07 EST, Kaleem
no flags Details
ipaserver-install.log with extra prints (3.66 MB, application/x-gzip)
2016-03-03 19:41 EST, Scott Poore
no flags Details

External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:0909 normal SHIPPED_LIVE pki-core bug fix update 2016-05-10 18:51:49 EDT

  None (edit)
Description Kaleem 2016-02-12 06:40:17 EST
Created attachment 1123467 [details]
mail text received for crash

Description of problem:
Following crash seen with pki-common pkg during IPA server install

:__init__.py:55:load:IndexError: list index out of range
:Traceback (most recent call last):
:  File "/usr/sbin/pki-server", line 89, in <module>
:    cli.execute(sys.argv)
:  File "/usr/sbin/pki-server", line 84, in execute
:    super(PKIServerCLI, self).execute(args)
:  File "/usr/lib/python2.6/site-packages/pki/cli.py", line 195, in execute
:    module.execute(module_args)
:  File "/usr/lib/python2.6/site-packages/pki/server/cli/upgrade.py", line 79, in execute
:    subsystem = pki.server.PKISubsystem(subsystem_dir)
:  File "/usr/lib/python2.6/site-packages/pki/server/__init__.py", line 44, in __init__
:    self.load()
:  File "/usr/lib/python2.6/site-packages/pki/server/__init__.py", line 55, in load
:    value = parts[1]
:IndexError: list index out of range

Version-Release number of selected component (if applicable):
pki-common.noarch 0:9.0.3-47.el6 

How reproducible:

Steps to Reproduce:
1. IPA install 

Actual results:
Crash seen

Expected results:
No crash expected

Additional info:
Please find the attached files for more reference.
Comment 2 Kaleem 2016-02-12 06:43 EST
Created attachment 1123469 [details]
crash dump
Comment 3 Nathan Kinder 2016-02-16 16:55:11 EST
Was this a fresh IPA install, or an upgrade?  Is there anything special with regards to how you are installing IPA?
Comment 4 Kaleem 2016-02-16 22:16:59 EST
(In reply to Nathan Kinder from comment #3)
> Was this a fresh IPA install, or an upgrade?  Is there anything special with
> regards to how you are installing IPA?

It was a fresh IPA install. We are doing normal IPA install by using following in automated execution

/usr/sbin/ipa-server-install --setup-dns --forwarder= --hostname=apollo.testrelm.test -r TESTRELM.TEST -n testrelm.test -p xxxxxxxx -a xxxxxxxx --ip-address= -U

Do let me know if any other info required.
Comment 5 Endi Sukma Dewata 2016-02-18 13:43:41 EST
Hi, I was not able to reproduce the problem using the same build.

In the bug description the code was trying to parse the properties in CS.cfg and some how failed because one of the lines doesn't contain the property value. Could you post the CS.cfg? I couldn't find it in the attachment. Thanks.
Comment 6 Kaleem 2016-02-22 02:07 EST
Created attachment 1129165 [details]

Please find the attached CS.cfg.
Comment 7 Endi Sukma Dewata 2016-02-27 00:00:37 EST
Thanks. I don't see anything wrong with the attached CS.cfg.

Could you add some print statements to the loop in /usr/lib/python2.6/site-packages/pki/server/__init__.py:52 as follows?

  print 'Parsing %s:' % self.cs_conf
  for line in lines:
      print '* %s' % line
      parts = line.split('=', 1)
      name = parts[0]
      value = parts[1]
      self.config[name] = value

Then run the installation again. The output should appear in /var/log/ipaserver-install.log. Please post the line from CS.cfg that's causing the parsing error. Thanks.
Comment 10 Scott Poore 2016-03-03 19:41 EST
Created attachment 1132990 [details]
ipaserver-install.log with extra prints
Comment 11 Scott Poore 2016-03-03 19:43:46 EST

I ran an install with the extra prints you indicated.  I couldn't see anything other than the extra data printed in the install log.  I included the whole log though in case I missed something.  As far as I've seen, ipa-server-install is completing as expected when this crash occurs.  

Let Kaleem or myself know if you need us to try something else.

Comment 13 Ade Lee 2016-03-09 10:01:50 EST
The only thing I can think is happening here is that this is a timing issue.
The system is calling the upgrade scripts and reading in the CS.cfg file before the filesystem has finished writing to it.

In the case where you added more debug statements above, the crash did not occur - presumably because the timing had changed.

Can you try the following test?

1. Remove the debug code Endi asked you to add above.
2. In the file /usr/share/pki/scripts/functions, change the line that says -->

   pki-server upgrade --subsystem "$PKI_INSTANCE_PATH" --silent


   pki-server upgrade --subsystem "$PKI_INSTANCE_PATH" --silent

3. See if the issue recurs.
Comment 14 Jenny Galipeau 2016-03-10 09:58:20 EST
Tom ... How do we get this on the RHEL 6.8 blocker list?  Who can provide "Blocker" flag "+"?
Comment 16 Ade Lee 2016-03-10 12:29:44 EST
The root cause of this is actually quite straightforward.

It turns out that the CS.cfg file which is being parsed contains some comment lines before configuration, and these lines cause the exception to be raised and the alert to be issued.

This is a trivial case which can be resolved by making the parsing code ignore comment and blank lines.
Comment 18 Ade Lee 2016-03-11 10:54:40 EST
Fix checked into rhel 6.8 branch ..

Counting objects: 62, done.
Compressing objects: 100% (31/31), done.
Writing objects: 100% (32/32), 5.18 KiB, done.
Total 32 (delta 18), reused 0 (delta 0)
To ssh://git.app.eng.bos.redhat.com/srv/git/pki.git
   3637b1f..dd5039f  DOGTAG_9_0_RHEL_BRANCH -> DOGTAG_9_0_RHEL_BRANCH

Matt will be creating builds.
Comment 19 Matthew Harmsen 2016-03-11 12:59:47 EST
I would change the existing doc text to something like this:

The ipa-server-install utility sometimes terminates unexpectedly

An attempt to install an Identity Management server can fail due to a problem with the *pki-common* package. The fail occurs because the `CS.cfg` certificate authority (CA) configuration file which is being parsed contains unexpected comment lines before configuration. This issue has been resolved by making
the parsing code ignore comment and blank lines.
Comment 21 Scott Poore 2016-03-13 18:23:05 EDT

Version ::


Results ::

:: [  BEGIN   ] :: Running ' /usr/sbin/ipa-server-install --setup-dns --forwarder=$FORWARDER --hostname=cisco-c240m3-01.testrelm.test -r TESTRELM.TEST -n testrelm.test -p Secret123 -a Secret123 --ip-address=$IPADDRESS -U'

The log file for this installation can be found in /var/log/ipaserver-install.log
This program will set up the IPA Server.

This includes:
  * Configure a stand-alone CA (dogtag) for certificate management
  * Configure the Network Time Daemon (ntpd)
  * Create and configure an instance of Directory Server
  * Create and configure a Kerberos Key Distribution Center (KDC)
  * Configure Apache (httpd)
  * Configure DNS (bind)

To accept the default shown in brackets, press the Enter key.

Warning: skipping DNS resolution of host cisco-c240m3-01.testrelm.test
Using reverse zone 70.16.10.in-addr.arpa.

The IPA Master Server will be configured with:
Hostname:      cisco-c240m3-01.testrelm.test
IP address:    $IPADDRESS
Domain name:   testrelm.test
Realm name:    TESTRELM.TEST

BIND DNS server will be configured to serve IPA domain with:
Forwarders:    $FORWARDER
Reverse zone:  $REVNET.in-addr.arpa.

Configuring NTP daemon (ntpd)
  [1/4]: stopping ntpd
  [2/4]: writing configuration
  [3/4]: configuring ntpd to start on boot
  [4/4]: starting ntpd
Done configuring NTP daemon (ntpd).
Configuring directory server for the CA (pkids): Estimated time 30 seconds
  [1/3]: creating directory server user
  [2/3]: creating directory server instance
  [3/3]: restarting directory server
Done configuring directory server for the CA (pkids).
Configuring certificate server (pki-cad): Estimated time 3 minutes 30 seconds
  [1/21]: creating certificate server user
  [2/21]: creating pki-ca instance
  [3/21]: configuring certificate server instance
  [4/21]: disabling nonces
  [5/21]: creating CA agent PKCS#12 file in /root
  [6/21]: creating RA agent certificate database
  [7/21]: importing CA chain to RA certificate database
  [8/21]: fixing RA database permissions
  [9/21]: setting up signing cert profile
  [10/21]: set up CRL publishing
  [11/21]: set certificate subject base
  [12/21]: enabling Subject Key Identifier
  [13/21]: setting audit signing renewal to 2 years
  [14/21]: configuring certificate server to start on boot
  [15/21]: restarting certificate server
  [16/21]: requesting RA certificate from CA
  [17/21]: issuing RA agent certificate
  [18/21]: adding RA agent as a trusted user
  [19/21]: configure certificate renewals
  [20/21]: configure Server-Cert certificate renewal
  [21/21]: Configure HTTP to proxy connections
Done configuring certificate server (pki-cad).
Configuring directory server (dirsrv): Estimated time 1 minute
  [1/38]: creating directory server user
  [2/38]: creating directory server instance
  [3/38]: adding default schema
  [4/38]: enabling memberof plugin
  [5/38]: enabling winsync plugin
  [6/38]: configuring replication version plugin
  [7/38]: enabling IPA enrollment plugin
  [8/38]: enabling ldapi
  [9/38]: disabling betxn plugins
  [10/38]: configuring uniqueness plugin
  [11/38]: configuring uuid plugin
  [12/38]: configuring modrdn plugin
  [13/38]: enabling entryUSN plugin
  [14/38]: configuring lockout plugin
  [15/38]: creating indices
  [16/38]: enabling referential integrity plugin
  [17/38]: configuring ssl for ds instance
  [18/38]: configuring certmap.conf
  [19/38]: configure autobind for root
  [20/38]: configure new location for managed entries
  [21/38]: restarting directory server
  [22/38]: adding default layout
  [23/38]: adding delegation layout
  [24/38]: adding replication acis
  [25/38]: creating container for managed entries
  [26/38]: configuring user private groups
  [27/38]: configuring netgroups from hostgroups
  [28/38]: creating default Sudo bind user
  [29/38]: creating default Auto Member layout
  [30/38]: adding range check plugin
  [31/38]: creating default HBAC rule allow_all
  [32/38]: Upload CA cert to the directory
  [33/38]: initializing group membership
  [34/38]: adding master entry
  [35/38]: configuring Posix uid/gid generation
  [36/38]: enabling compatibility plugin
  [37/38]: tuning directory server
  [38/38]: configuring directory to start on boot
Done configuring directory 
MARK-LWD-LOOP -- 2016-03-13 13:40:05 --
server (dirsrv).
Configuring Kerberos KDC (krb5kdc): Estimated time 30 seconds
  [1/10]: adding sasl mappings to the directory
  [2/10]: adding kerberos container to the directory
  [3/10]: configuring KDC
  [4/10]: initialize kerberos container
  [5/10]: adding default ACIs
  [6/10]: creating a keytab for the directory
  [7/10]: creating a keytab for the machine
  [8/10]: adding the password extension to the directory
  [9/10]: starting the KDC
  [10/10]: configuring KDC to start on boot
Done configuring Kerberos KDC (krb5kdc).
Configuring kadmin
  [1/2]: starting kadmin 
  [2/2]: configuring kadmin to start on boot
Done configuring kadmin.
Configuring ipa_memcached
  [1/2]: starting ipa_memcached 
  [2/2]: configuring ipa_memcached to start on boot
Done configuring ipa_memcached.
Configuring the web interface (httpd): Estimated time 1 minute
  [1/14]: setting mod_nss port to 443
  [2/14]: setting mod_nss protocol list to TLSv1.0 - TLSv1.2
  [3/14]: setting mod_nss password file
  [4/14]: enabling mod_nss renegotiate
  [5/14]: adding URL rewriting rules
  [6/14]: configuring httpd
  [7/14]: setting up ssl
  [8/14]: setting up browser autoconfig
  [9/14]: publish CA cert
  [10/14]: creating a keytab for httpd
  [11/14]: clean up any existing httpd ccache
  [12/14]: configuring SELinux for httpd
  [13/14]: restarting httpd
  [14/14]: configuring httpd to start on boot
Done configuring the web interface (httpd).
Applying LDAP updates
Restarting the directory server
Restarting the KDC
Configuring DNS (named)
  [1/9]: adding DNS container
  [2/9]: setting up our zone
  [3/9]: setting up reverse zone
  [4/9]: setting up our own record
  [5/9]: setting up kerberos principal
  [6/9]: setting up named.conf
  [7/9]: restarting named
  [8/9]: configuring named to start on boot
  [9/9]: changing resolv.conf to point to ourselves
Done configuring DNS (named).

Global DNS configuration in LDAP server is empty
You can use 'dnsconfig-mod' command to set global DNS options that
would override settings in local named.conf files

Restarting the web server
Setup complete

Next steps:
    1. You must make sure these network ports are open:
        TCP Ports:
          * 80, 443: HTTP/HTTPS
          * 389, 636: LDAP/LDAPS
          * 88, 464: kerberos
          * 53: bind
        UDP Ports:
          * 88, 464: kerberos
          * 53: bind
          * 123: ntp

    2. You can now obtain a kerberos ticket using the command: 'kinit admin'
       This ticket will allow you to use the IPA tools (e.g., ipa user-add)
       and the web user interface.

Be sure to back up the CA certificate stored in /root/cacert.p12
This file is required to create replicas. The password for this
file is the Directory Manager password
:: [   PASS   ] :: Command ' /usr/sbin/ipa-server-install --setup-dns --forwarder=$FORWARDER --hostname=cisco-c240m3-01.testrelm.test -r TESTRELM.TEST -n testrelm.test -p Secret123 -a Secret123 --ip-address=$IPADDRESS -U' (Expected 0, got 0)

No ABRT crash dump seen from this run.
Comment 23 errata-xmlrpc 2016-05-10 20:32:58 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.