Bug 1140855

Created attachment 936673 [details]
ipa server install log

Enabled CS debug using steps:

edit the template cs.cfg at /usr/share/pki/ca/conf/CS.cfg with 
debug.append=false
debug.enabled=true
debug.filename=/var/lib/pki-ca/logs/debug
debug.hashkeytypes=
debug.level=0
debug.showcaller=false

reinstalled, and Matthew took a look at the logs at /var/log/pki-ca/debug on the beaker machine

With following 389 build, ipa-server installation seems to be successful

389-ds-base-1.2.11.15-45.el6.x86_64


[root@rhel66-master ~]# rpm -q sssd ipa-server 389-ds-base pki-ca
sssd-1.11.6-29.el6.x86_64
ipa-server-3.0.0-42.el6.x86_64
389-ds-base-1.2.11.15-45.el6.x86_64
pki-ca-9.0.3-37.el6.noarch
[root@rhel66-master ~]# 

[root@rhel66-master ~]# service ipa status
Directory Service: RUNNING
KDC Service: RUNNING
KPASSWD Service: RUNNING
DNS Service: RUNNING
MEMCACHE Service: RUNNING
HTTP Service: RUNNING
CA Service: RUNNING
[root@rhel66-master ~]#

Installation on my 6.6 VM also worked without any issue. Given that installation works fine for both me and Kaleem as QE representative, closing this Bugzilla.

Please feel free to reopen it if the problem reoccur.

I'm seeing this bug again in today 6.6 build.

[root@cloud-qe-1-vm-1 log]# rpm -qa ipa-server 389-ds-base
389-ds-base-1.2.11.15-46.el6.x86_64
ipa-server-3.0.0-42.el6.x86_64

I'm not sure what other RPMs to check version here.

2014-09-15T13:23:41Z DEBUG stderr=[Fatal Error] :1:8: The string "--" is not permitted within comments.
org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 8; The string "--" is not permitted within 
comments.
        at org.apache.xerces.parsers.DOMParser.parse(DOMParser.java:239)
        at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:283)
        at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121)
        at ParseXML.parse(ParseXML.java:258)
        at ConfigureCA.getStatus(ConfigureCA.java:205)
        at ConfigureCA.checkStatus(ConfigureCA.java:221)
        at ConfigureCA.checkStatus(ConfigureCA.java:216)
        at ConfigureCA.DomainPanel(ConfigureCA.java:332)
        at ConfigureCA.ConfigureCAInstance(ConfigureCA.java:1173)
        at ConfigureCA.main(ConfigureCA.java:1672)

This was definitely a weird one to diagnose.

I worked on one of QE's systems, and attempted to install the CA using
both pkisilent (which uses httpclient) and my firefox browser.  Both
showed the following weirdness.

The client would attempt to POST something like the following:
https://cloud-qe-1-vm-1.testrelm.test:9445//ca/admin/console/config/wizard?

Data: 
sdomainURL=https%3A%2F%2Fcloud-qe-1-vm-1.testrelm.test%3A9445&sdomainName=IPA&choice=newdomain&p=3&op=next&xml=true

On the server side though, I would see logs like the following:

[15/Sep/2014:15:31:59][http-9445-4]: WizardServlet::service() param name='choice' value=''
[15/Sep/2014:15:31:59][http-9445-4]: WizardServlet::service() param name='p' value='3'

Or if attempting to connect to an existing domain:
[15/Sep/2014:15:28:32][http-9445-4]: WizardServlet::service() param name='sdomainN' value=''
[15/Sep/2014:15:28:32][http-9445-4]: WizardServlet::service() param name='p' value='3'

So, it appears that the data has been truncated.  For example,
"sdomainN" is not the parameter name - it is supposed to be
"sdomainName".

As this was happening from both pkisilent and the browser, it seemed
clear that this was a problem on the server side.

Now we had not changed anything in dogtag in RHEL6 for a very long time.
This only started happening recently - and so, we needed to look and see
what else had changed.

Nathan pointed out that a new tomcat build was created for RHEL6 on the
12th.  This build was on the affected machines (6.0.24-79).  I
downgraded the tomcat build to tomcat6-6.0.24-78.el6_5, and lo and
behold, everything started working again.

Transferring the bug to tomcat.

Ade

*** Bug 1141819 has been marked as a duplicate of this bug. ***

I did a quick test and it easily reproducible. With

ipa-server-3.0.0-42.el6.x86_64
pki-ca-9.0.3-37.el6.noarch
tomcat6-6.0.24-78.el6_5.noarch

the installation works:

# ipa-server-install
...
Configuring certificate server (pki-cad): Estimated time 3 minutes 30 seconds
  [1/21]: creating certificate server user
  [2/21]: creating pki-ca instance
  [3/21]: configuring certificate server instance
  [4/21]: disabling nonces
...

As soon as I update also tomcat 6, it crashes:

# yum update
...
========================================================================================================
 Package                     Arch       Version           Repository                               Size
========================================================================================================
Updating:
 tomcat6                     x86_64     6.0.24-79.el6     RHEL-6.6-nightly-workstation-x86_64      91 k
 tomcat6-el-2.1-api          x86_64     6.0.24-79.el6     RHEL-6.6-nightly-workstation-x86_64      46 k
 tomcat6-jsp-2.1-api         x86_64     6.0.24-79.el6     RHEL-6.6-nightly-workstation-x86_64      82 k
 tomcat6-lib                 x86_64     6.0.24-79.el6     RHEL-6.6-nightly-workstation-x86_64     2.9 M
 tomcat6-servlet-2.5-api     x86_64     6.0.24-79.el6     RHEL-6.6-nightly-workstation-x86_64      97 k

Transaction Summary
========================================================================================================
Upgrade       5 Package(s)
...

# rpm -q ipa-server pki-ca tomcat6
ipa-server-3.0.0-42.el6.x86_64
pki-ca-9.0.3-37.el6.noarch
tomcat6-6.0.24-79.el6.x86_64


# ipa-server-install -p kokos123 -a kokos123 --setup-dns --forwarder={10.16.255.2,10.11.5.19}
...
Configuring certificate server (pki-cad): Estimated time 3 minutes 30 seconds
  [1/21]: creating certificate server user
  [2/21]: creating pki-ca instance
  [3/21]: configuring certificate server instance
ipa         : CRITICAL failed to configure ca instance Command '/usr/bin/perl /usr/bin/pkisilent ConfigureCA -cs_hostname vm-086.idm.lab.bos.redhat.com -cs_port 9445 -client_certdb_dir /tmp/tmp-lAOSQR -client_certdb_pwd XXXXXXXX -preop_pin Mech62SCbLJ9TfcB3tln -domain_name IPA -admin_user admin -admin_email root@localhost -admin_password XXXXXXXX -agent_name ipa-ca-agent -agent_key_size 2048 -agent_key_type rsa -agent_cert_subject CN=ipa-ca-agent,O=IDM.LAB.BOS.REDHAT.COM -ldap_host vm-086.idm.lab.bos.redhat.com -ldap_port 7389 -bind_dn cn=Directory Manager -bind_password XXXXXXXX -base_dn o=ipaca -db_name ipaca -key_size 2048 -key_type rsa -key_algorithm SHA256withRSA -save_p12 true -backup_pwd XXXXXXXX -subsystem_name pki-cad -token_name internal -ca_subsystem_cert_subject_name CN=CA Subsystem,O=IDM.LAB.BOS.REDHAT.COM -ca_subsystem_cert_subject_name CN=CA Subsystem,O=IDM.LAB.BOS.REDHAT.COM -ca_ocsp_cert_subject_name CN=OCSP Subsystem,O=IDM.LAB.BOS.REDHAT.COM -ca_server_cert_subject_name CN=vm-086.idm.lab.bos.redhat.com,O=IDM.LAB.BOS.REDHAT.COM -ca_audit_signing_cert_subject_name CN=CA Audit,O=IDM.LAB.BOS.REDHAT.COM -ca_sign_cert_subject_name CN=Certificate Authority,O=IDM.LAB.BOS.REDHAT.COM -external false -clone false' returned non-zero exit status 255
Configuration of CA failed


I see Tomcat6 was rebuilt for Bug 1140301 quite late in the game, though the bug description does not indicate that the change could break IPA.

Was able to install a master, replica, client successfully using the nightly RHEL-6.6-20140917 build, which included tomcat6.x86_64 0:6.0.24-80.el6.

marking verified - from #comment 14

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1618.html