Bug 1162173

From my researches in the Internet and code I found out that this could potentially be fixed by fixing the "slot_substitution" for "server.xml" BUT:
a) I could not find the hook where the slot substitution for server.xml is invoked and
b) I'm not sure whether there is any semantic meaning having the host-name in the server.xml comments <!--...--> or whether it is (as it should be) just a comment.

Therefore I'm now going for a symptom fix (even I don't like it) i.e. patching: "org/apache/catalina/startup/Catalina.java" and "com/netscape/cmscore/apps/CMSEngine.java" to remove the double-hyphens "--" in the stream just before passing it to the XML-Reader.

Created attachment 958788 [details]
Workaround / temporary fix for bug 1162173 - more details see bug

Regarding the attachment and comment above:
As already announced I could not find a sustainable solution and therefore I've implemented now a workaround respective fixed the symptoms until a sustainable solution is provided (and the installation went fine).
Basically I've fixed the "org/apache/catalina/startup/Catalina.java" class to remove the double hyphens in the "server.xml" file and persist this change to disk for future.
In the attachment I've provided the "catalina.jar" which can be copied to "/usr/share/tomcat/lib/" to temporarily fix this issue.
The following changes have been made to "Catalina.java":

diff -r java/org/apache/catalina/startup/Catalina.java eclipse/apache-tomcat-7.0.56-src/src/org/apache/catalina/startup/Catalina.java
19a20,21
> import java.io.ByteArrayInputStream;
> import java.io.ByteArrayOutputStream;
21a24
> import java.io.FileOutputStream;
23a27
> import java.io.InputStreamReader;
24a29
> import java.io.OutputStreamWriter;
613c618,626
<             inputSource.setByteStream(inputStream);
---
> 			// TODO find sustainable fix for this workaround regarding:
> 			// https://bugzilla.redhat.com/show_bug.cgi?id=1162173
> 			// https://bugzilla.redhat.com/show_bug.cgi?id=1160555
> 			ByteArrayOutputStream baos = replaceDoubleHyphenInComment(
> 					inputStream, file);
> 			inputSource.setByteStream(new ByteArrayInputStream(baos
> 					.toByteArray()));
> 
> 			// inputSource.setByteStream(inputStream);
654a668,715
> 	private ByteArrayOutputStream replaceDoubleHyphenInComment(
> 			InputStream inputStream, File file) throws IOException {
> 		// TODO find sustainable fix for this workaround regarding:
> 		// https://bugzilla.redhat.com/show_bug.cgi?id=1162173
> 		// https://bugzilla.redhat.com/show_bug.cgi?id=1160555
> 		// Actually the correct fix would be not to have double hyphens
> 		// "--" in server.xml file but instead escaping it correctly.
> 		// But at this point in time trying a quick symptom fix
> 		// and removing the double hyphens before passing to SAX parser.
> 		char[] cbuf = new char[4096];
> 		StringBuffer sb = new StringBuffer();
> 		InputStreamReader isr = new InputStreamReader(inputStream);
> 		int isrReadLen;
> 		boolean modified = false;
> 		while ((isrReadLen = isr.read(cbuf)) == cbuf.length) {
> 			sb.append(cbuf);
> 		}
> 		if (isrReadLen > 0) {
> 			sb.append(cbuf, 0, isrReadLen);
> 		}
> 		isr.close();
> 		int commentStart = sb.indexOf("<!--", 0) + 4;
> 		int commentEnd = sb.indexOf("-->", commentStart);
> 		while (commentStart != 3) {
> 			if (sb.substring(commentStart, commentEnd).contains("--")) {
> 				modified = true;
> 				sb.replace(
> 						commentStart,
> 						commentEnd,
> 						sb.substring(commentStart, commentEnd).replaceAll("--",
> 								"**"));
> 			}
> 			commentStart = sb.indexOf("<!--", commentEnd + 3) + 4;
> 			commentEnd = sb.indexOf("-->", commentStart);
> 		}
> 		ByteArrayOutputStream baos = new ByteArrayOutputStream();
> 		OutputStreamWriter osw = new OutputStreamWriter(baos);
> 		osw.write((sb.toString().toCharArray()));
> 		osw.close();
> 
> 		if (modified && file != null) {
> 			FileOutputStream fos = new FileOutputStream(file);
> 			osw = new OutputStreamWriter(fos);
> 			osw.write((sb.toString().toCharArray()));
> 			osw.close();
> 		}
> 		return baos;
> 	}

Is this fix available in upstream Dogtag? I see that reporter kindly provided patch already. We would like to test the fix in FreeIPA too.

I've cloned this bug to Trac: https://fedorahosted.org/pki/ticket/1260

Per discussion with mharmsen the proper fix is to modify pkidaemon to obtain the info directly from the CS.cfg and then remove these lines from server.xml.

This bug is proposed to be fixed in Dogtag 10.3 timeframe. If it needs to be fixed sooner please mention that in the Trac ticket. Thanks.

Associated with upstream PKI Ticket #1260 slated for Dogtag 10.3 (RHCS 9.1); propose moving to RHEL 7.3?

(In reply to Matthew Harmsen from comment #8)
> Associated with upstream PKI Ticket #1260 slated for Dogtag 10.3 (RHCS 9.1);
> propose moving to RHEL 7.3?

Moving to RHEL 7.3.

Shouldn't this bugzilla be on tomcat component?

(In reply to Jan Pazdziora from comment #11)
> Shouldn't this bugzilla be on tomcat component?

Actually, no -- the reason that this problem is occurring is because we are obtaining this information out of commented lines in 'server.xml' (and XML uses the nasty "--" in their comment style). See https://fedorahosted.org/pki/ticket/1260 for details.

We are currently slated to fix this in 10.3 (so hopefully RHEL 7.3).

Per discussions in the RHEL 7.3 Triage meeting of 01/06/2016: priority low

ftweedal pushed to master:
* 8beb5cfa4cd81fbf47ea8cd6839b793c2a12284e

[root@nightcrawler ~]# rpm -qi pki-ca
Name        : pki-ca
Version     : 10.3.3
Release     : 8.el7
Architecture: noarch
Install Date: Wed 31 Aug 2016 03:15:38 PM EDT
Group       : System Environment/Daemons
Size        : 2430595
License     : GPLv2
Signature   : (none)
Source RPM  : pki-core-10.3.3-8.el7.src.rpm
Build Date  : Tue 30 Aug 2016 03:23:27 PM EDT
Build Host  : ppc-015.build.eng.bos.redhat.com
Relocations : (not relocatable)
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
Vendor      : Red Hat, Inc.
URL         : http://pki.fedoraproject.org/
Summary     : Certificate System - Certificate Authority

Did not see pkispawn CA failure when the hostname had "--" in it.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2396.html

Hi,
I just wanted to inform you that the fix is still not working.
I just did a fresh install "yum install ipa-server bind bind-dyndb-ldap" and got the following packages (which are the correct packages and versions according to: "https://rhn.redhat.com/errata/RHBA-2016-2396.html"):
...
    Dep-Install pki-base-10.3.3-14.el7_3.noarch                                  @updates
    Dep-Install pki-base-java-10.3.3-14.el7_3.noarch                             @updates
    Dep-Install pki-ca-10.3.3-14.el7_3.noarch                                    @updates
    Dep-Install pki-kra-10.3.3-14.el7_3.noarch                                   @updates
    Dep-Install pki-server-10.3.3-14.el7_3.noarch                                @updates
    Dep-Install pki-tools-10.3.3-14.el7_3.x86_64                                 @updates
    Dep-Install policycoreutils-python-2.5-9.el7.x86_64                          @updates
...

And during "ipa-server-install --no-ntp --idstart=5000 --idmax=9999" at the very beginning I got the following ERROR:
...
Server host name [directory--s0-v1.becke.ch]: 

ipa.ipapython.install.cli.install_tool(Server): ERROR    Invalid hostname 'directory--s0-v1.becke.ch', only letters, numbers, '-' are allowed. DNS label may not start or end with '-'

ipa.ipapython.install.cli.install_tool(Server): ERROR    The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information
...

And when looking in: /var/log/ipaserver-install.log
...
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/install.py", line 267, in decorated
    func(installer)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/install.py", line 460, in install_check
    sys.exit(str(e) + "\n")

2017-01-07T09:39:05Z DEBUG The ipa-server-install command failed, exception: SystemExit: Invalid hostname 'directory--s0-v1.becke.ch', only letters, numbers, '-' are allowed. DNS label may not start or end with '-'

2017-01-07T09:39:05Z ERROR Invalid hostname 'directory--s0-v1.becke.ch', only letters, numbers, '-' are allowed. DNS label may not start or end with '-'
...

Kind Regards
Raoul

That error message seems to be generated by IPA. Could you try editing the util.py as described in the original bug description?

The IPA part of the bug: https://bugzilla.redhat.com/show_bug.cgi?id=1160555

It is fixed upstream, it will be in RHEL 7.4

Yes this is correct I could quickly fix it myself as described above:

Replace regex repetition character "?" with "*": See:
]# diff util.py /usr/lib/python2.7/site-packages/ipalib/util.py
231c231
< label_regex = r'^[%(base)s%(extra)s]([%(base)s%(extra)s%(middle)s]?[%(base)s%(extra)s])*$' \
---
> label_regex = r'^[%(base)s%(extra)s]([%(base)s%(extra)s%(middle)s]*[%(base)s%(extra)s])*$' \