After fixing this issue the installation failed later on due to this fix:
...
[2/22]: configuring certificate server instance
ipa         : CRITICAL failed to configure ca instance Command '/usr/sbin/pkispawn -s CA -f /tmp/tmp72m3kU' returned non-zero exit status 1
Configuration of CA failed
...

Which was caused by: /var/log/pki/pki-tomcat/catalina.out
...
WARNING: Catalina.start using conf/server.xml: The string "--" is not permitted within comments.
Nov 07, 2014 10:12:05 PM org.apache.tomcat.util.digester.Digester fatalError
SEVERE: Parse Fatal Error at line 31 column 40: The string "--" is not permitted within comments.
org.xml.sax.SAXParseException; systemId: file:/var/lib/pki/pki-tomcat/conf/server.xml; lineNumber: 31; columnNumber: 40; The string "--" is not permitted within comments.
        at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)
        at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source)
...

So there seems to be an issue with tomcat pki that it cannot handle double hyphens "--" in hostname!

Thanks for investigation. Looks like pki bug is due - I will clone it.

Upstream ticket:
https://fedorahosted.org/freeipa/ticket/4710

Implemented a workaround ("symptom fix") for this issue. All details can be found in defect: https://bugzilla.redhat.com/show_bug.cgi?id=1162173

Thank you taking your time and submitting this request for Red Hat Enterprise Linux. Unfortunately, this bug was not given a priority and was deferred both in the upstream project and in Red Hat Enterprise Linux.

Given that we are unable to fulfill this request in following Red Hat Enterprise Linux releases, I am closing the Bugzilla as WONTFIX. To request that Red Hat re-considers the decision, please re-open the Bugzilla via appropriate support channels and provide additional business and/or technical details about its importance to you.

Note that you can still track this request or even contribute patches in the referred upstream Trac ticket.

This was fixed in Dogtag, so we fixed it in IPA as well :-)

Fixed upstream
master:
https://fedorahosted.org/freeipa/changeset/15cfd0ee20fd05735473d3677b6f9f349339197e

Verified
ipa server installed successfully with hostname "mgmt9-bug--test.testrelm.test"

ipa-server-4.5.0-13.el7.x86_64
pki-kra-10.4.1-4.el7.noarch
pki-base-10.4.1-4.el7.noarch
pki-base-java-10.4.1-4.el7.noarch
pki-tools-10.4.1-4.el7.x86_64
pki-server-10.4.1-4.el7.noarch
pki-ca-10.4.1-4.el7.noarch


# ipa-server-install --setup-dns --forwarder=10.16.36.29 --reverse-zone=4.16.10.in-addr.arpa. --allow-zone-overlap --hostname=mgmt9-bug--test.testrelm.test -r TESTRELM.TEST -n testrelm.test -p <XXXXX> -a <XXXXX> --ip-address=10.16.4.19 -U

The log file for this installation can be found in /var/log/ipaserver-install.log
==============================================================================
This program will set up the IPA Server.

This includes:
  * Configure a stand-alone CA (dogtag) for certificate management
  * Configure the Network Time Daemon (ntpd)
  * Create and configure an instance of Directory Server
  * Create and configure a Kerberos Key Distribution Center (KDC)
  * Configure Apache (httpd)
  * Configure DNS (bind)
  * Configure the KDC to enable PKINIT

WARNING: conflicting time&date synchronization service 'chronyd' will be disabled
in favor of ntpd

Warning: skipping DNS resolution of host mgmt9-bug--test.testrelm.test
Checking DNS domain testrelm.test., please wait ...
Checking DNS forwarders, please wait ...
Using reverse zone(s) 4.16.10.in-addr.arpa.

The IPA Master Server will be configured with:
Hostname:       mgmt9-bug--test.testrelm.test
IP address(es): 10.16.4.19
Domain name:    testrelm.test
Realm name:     TESTRELM.TEST

BIND DNS server will be configured to serve IPA domain with:
Forwarders:       10.16.36.29
Forward policy:   only
Reverse zone(s):  4.16.10.in-addr.arpa.

Adding [10.16.4.19 mgmt9-bug--test.testrelm.test] to your /etc/hosts file
Configuring NTP daemon (ntpd)
  [1/4]: stopping ntpd
  [2/4]: writing configuration
  [3/4]: configuring ntpd to start on boot
  [4/4]: starting ntpd
Done configuring NTP daemon (ntpd).
Configuring directory server (dirsrv). Estimated time: 30 seconds
  [1/45]: creating directory server instance
  [2/45]: enabling ldapi
  [3/45]: configure autobind for root
  [4/45]: stopping directory server
  [5/45]: updating configuration in dse.ldif
  [6/45]: starting directory server
  [7/45]: adding default schema
  [8/45]: enabling memberof plugin
  [9/45]: enabling winsync plugin
  [10/45]: configuring replication version plugin
  [11/45]: enabling IPA enrollment plugin
  [12/45]: configuring uniqueness plugin
  [13/45]: configuring uuid plugin
  [14/45]: configuring modrdn plugin
  [15/45]: configuring DNS plugin
  [16/45]: enabling entryUSN plugin
  [17/45]: configuring lockout plugin
  [18/45]: configuring topology plugin
  [19/45]: creating indices
  [20/45]: enabling referential integrity plugin
  [21/45]: configuring certmap.conf
  [22/45]: configure new location for managed entries
  [23/45]: configure dirsrv ccache
  [24/45]: enabling SASL mapping fallback
  [25/45]: restarting directory server
  [26/45]: adding sasl mappings to the directory
  [27/45]: adding default layout
  [28/45]: adding delegation layout
  [29/45]: creating container for managed entries
  [30/45]: configuring user private groups
  [31/45]: configuring netgroups from hostgroups
  [32/45]: creating default Sudo bind user
  [33/45]: creating default Auto Member layout
  [34/45]: adding range check plugin
  [35/45]: creating default HBAC rule allow_all
  [36/45]: adding entries for topology management
  [37/45]: initializing group membership
  [38/45]: adding master entry
  [39/45]: initializing domain level
  [40/45]: configuring Posix uid/gid generation
  [41/45]: adding replication acis
  [42/45]: activating sidgen plugin
  [43/45]: activating extdom plugin
  [44/45]: tuning directory server
  [45/45]: configuring directory to start on boot
Done configuring directory server (dirsrv).
Configuring Kerberos KDC (krb5kdc)
  [1/10]: adding kerberos container to the directory
  [2/10]: configuring KDC
  [3/10]: initialize kerberos container
WARNING: Your system is running out of entropy, you may experience long delays
  [4/10]: adding default ACIs
  [5/10]: creating a keytab for the directory
  [6/10]: creating a keytab for the machine
  [7/10]: adding the password extension to the directory
  [8/10]: creating anonymous principal
  [9/10]: starting the KDC
  [10/10]: configuring KDC to start on boot
Done configuring Kerberos KDC (krb5kdc).
Configuring kadmin
  [1/2]: starting kadmin 
  [2/2]: configuring kadmin to start on boot
Done configuring kadmin.
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/29]: configuring certificate server instance
  [2/29]: exporting Dogtag certificate store pin
  [3/29]: stopping certificate server instance to update CS.cfg
  [4/29]: backing up CS.cfg
  [5/29]: disabling nonces
  [6/29]: set up CRL publishing
  [7/29]: enable PKIX certificate path discovery and validation
  [8/29]: starting certificate server instance
  [9/29]: configure certmonger for renewals
  [10/29]: requesting RA certificate from CA
  [11/29]: setting up signing cert profile
  [12/29]: setting audit signing renewal to 2 years
  [13/29]: restarting certificate server
  [14/29]: publishing the CA certificate
  [15/29]: adding RA agent as a trusted user
  [16/29]: authorizing RA to modify profiles
  [17/29]: authorizing RA to manage lightweight CAs
  [18/29]: Ensure lightweight CAs container exists
  [19/29]: configure certificate renewals
  [20/29]: configure Server-Cert certificate renewal
  [21/29]: Configure HTTP to proxy connections
  [22/29]: restarting certificate server
  [23/29]: updating IPA configuration
  [24/29]: enabling CA instance
  [25/29]: migrating certificate profiles to LDAP
  [26/29]: importing IPA certificate profiles
  [27/29]: adding default CA ACL
  [28/29]: adding 'ipa' CA entry
  [29/29]: configuring certmonger renewal for lightweight CAs
Done configuring certificate server (pki-tomcatd).
Configuring directory server (dirsrv)
  [1/3]: configuring TLS for DS instance
  [2/3]: adding CA certificate entry
  [3/3]: restarting directory server
Done configuring directory server (dirsrv).
Configuring ipa-otpd
  [1/2]: starting ipa-otpd 
  [2/2]: configuring ipa-otpd to start on boot
Done configuring ipa-otpd.
Configuring ipa-custodia
  [1/5]: Generating ipa-custodia config file
  [2/5]: Making sure custodia container exists
  [3/5]: Generating ipa-custodia keys
  [4/5]: starting ipa-custodia 
  [5/5]: configuring ipa-custodia to start on boot
Done configuring ipa-custodia.
Configuring the web interface (httpd)
  [1/22]: stopping httpd
  [2/22]: setting mod_nss port to 443
  [3/22]: setting mod_nss cipher suite
  [4/22]: setting mod_nss protocol list to TLSv1.0 - TLSv1.2
  [5/22]: setting mod_nss password file
  [6/22]: enabling mod_nss renegotiate
  [7/22]: enabling mod_nss OCSP
  [8/22]: adding URL rewriting rules
  [9/22]: configuring httpd
  [10/22]: setting up httpd keytab
  [11/22]: configuring Gssproxy
  [12/22]: setting up ssl
  [13/22]: configure certmonger for renewals
  [14/22]: importing CA certificates from LDAP
  [15/22]: publish CA cert
  [16/22]: clean up any existing httpd ccaches
  [17/22]: configuring SELinux for httpd
  [18/22]: create KDC proxy config
  [19/22]: enable KDC proxy
  [20/22]: starting httpd
  [21/22]: configuring httpd to start on boot
  [22/22]: enabling oddjobd
Done configuring the web interface (httpd).
Configuring Kerberos KDC (krb5kdc)
  [1/1]: installing X509 Certificate for PKINIT
Done configuring Kerberos KDC (krb5kdc).
Applying LDAP updates
Upgrading IPA:. Estimated time: 1 minute 30 seconds
  [1/9]: stopping directory server
  [2/9]: saving configuration
  [3/9]: disabling listeners
  [4/9]: enabling DS global lock
  [5/9]: starting directory server
  [6/9]: upgrading server
  [7/9]: stopping directory server
  [8/9]: restoring configuration
  [9/9]: starting directory server
Done.
Restarting the KDC
Configuring DNS (named)
  [1/12]: generating rndc key file
WARNING: Your system is running out of entropy, you may experience long delays
  [2/12]: adding DNS container
  [3/12]: setting up our zone
  [4/12]: setting up reverse zone
  [5/12]: setting up our own record
  [6/12]: setting up records for other masters
  [7/12]: adding NS record to the zones
  [8/12]: setting up kerberos principal
  [9/12]: setting up named.conf
  [10/12]: setting up server configuration
  [11/12]: configuring named to start on boot
  [12/12]: changing resolv.conf to point to ourselves
Done configuring DNS (named).
Restarting the web server to pick up resolv.conf changes
Configuring DNS key synchronization service (ipa-dnskeysyncd)
  [1/7]: checking status
  [2/7]: setting up bind-dyndb-ldap working directory
  [3/7]: setting up kerberos principal
  [4/7]: setting up SoftHSM
  [5/7]: adding DNSSEC containers
  [6/7]: creating replica keys
  [7/7]: configuring ipa-dnskeysyncd to start on boot
Done configuring DNS key synchronization service (ipa-dnskeysyncd).
Restarting ipa-dnskeysyncd
Restarting named
Updating DNS system records
Configuring client side components
Using existing certificate '/etc/ipa/ca.crt'.
Client hostname: mgmt9-bug--test.testrelm.test
Realm: TESTRELM.TEST
DNS Domain: testrelm.test
IPA Server: mgmt9-bug--test.testrelm.test
BaseDN: dc=testrelm,dc=test

Skipping synchronizing time with NTP server.
New SSSD config will be created
Configured sudoers in /etc/nsswitch.conf
Configured /etc/sssd/sssd.conf
trying https://mgmt9-bug--test.testrelm.test/ipa/json
Forwarding 'schema' to json server 'https://mgmt9-bug--test.testrelm.test/ipa/json'
trying https://mgmt9-bug--test.testrelm.test/ipa/session/json
Forwarding 'ping' to json server 'https://mgmt9-bug--test.testrelm.test/ipa/session/json'
Forwarding 'ca_is_enabled' to json server 'https://mgmt9-bug--test.testrelm.test/ipa/session/json'
Systemwide CA database updated.
Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_ecdsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_ed25519_key.pub
Forwarding 'host_mod' to json server 'https://mgmt9-bug--test.testrelm.test/ipa/session/json'
SSSD enabled
Configured /etc/openldap/ldap.conf
Configured /etc/ssh/ssh_config
Configured /etc/ssh/sshd_config
Configuring testrelm.test as NIS domain.
Client configuration complete.
The ipa-client-install command was successful

==============================================================================
Setup complete

Next steps:
	1. You must make sure these network ports are open:
		TCP Ports:
		  * 80, 443: HTTP/HTTPS
		  * 389, 636: LDAP/LDAPS
		  * 88, 464: kerberos
		  * 53: bind
		UDP Ports:
		  * 88, 464: kerberos
		  * 53: bind
		  * 123: ntp

	2. You can now obtain a kerberos ticket using the command: 'kinit admin'
	   This ticket will allow you to use the IPA tools (e.g., ipa user-add)
	   and the web user interface.

Be sure to back up the CA certificates stored in /root/cacert.p12
These files are required to create replicas. The password for these
files is the Directory Manager password

# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.4 Beta (Maipo

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2304