RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2022483 - Unable to join RHEL 8.5 Replica to RHEL 7.9 Master for migration purposes
Summary: Unable to join RHEL 8.5 Replica to RHEL 7.9 Master for migration purposes
Keywords:
Status: CLOSED ERRATA
Alias: None
Deadline: 2022-02-14
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: ipa
Version: 8.5
Hardware: All
OS: All
unspecified
urgent
Target Milestone: rc
: ---
Assignee: Rob Crittenden
QA Contact: ipa-qe
URL:
Whiteboard:
Depends On:
Blocks: 2050540
TreeView+ depends on / blocked
 
Reported: 2021-11-11 19:07 UTC by Mike Ralph
Modified: 2022-05-10 14:34 UTC (History)
10 users (show)

Fixed In Version: idm-DL1-8060020220210180711.92098735
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2050540 (view as bug list)
Environment:
Last Closed: 2022-05-10 14:09:17 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Error during replica install for ca (14.90 KB, text/plain)
2021-11-11 19:07 UTC, Mike Ralph
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker FREEIPA-7326 0 None None None 2021-11-11 19:07:32 UTC
Red Hat Issue Tracker RHELPLAN-102555 0 None None None 2021-11-11 19:07:36 UTC
Red Hat Product Errata RHEA-2022:1884 0 None None None 2022-05-10 14:09:52 UTC

Description Mike Ralph 2021-11-11 19:07:04 UTC
Created attachment 1841265 [details]
Error during replica install for ca

Description of problem:
  RHEL 8.5 IdM replica fails to install when trying to join a RHEL 7.9 IdM environment. It fails during "Configuring certificate server (pki-tomcatd)" Step 27 of 29 "importing IPA certificate profiles" for No route to host.

Version-Release number of selected component (if applicable):
  RHEL 7.9 master
  RHEL 8.5 replica


How reproducible:
  Consistantly


Steps to Reproduce:
1. Install RHEL 7.9 IdM master
2. Attempt to install RHEL 8.5 IdM replica


Actual results:
  Fails at step 27 during CA installation


Expected results:
  Completes install


Additional info:

Comment 1 Rob Crittenden 2021-11-11 20:48:16 UTC
I can't reproduce this. This is a very low-level error, EHOSTUNREACH. It could be a firewall issue but it is almost certainly localized to your lab.

Comment 2 Mike Ralph 2021-11-11 20:58:05 UTC
Interesting as I have already had two other people complain of this issue. All servers tested are on the same IP subnet. RHEL 7.9 replicas have no issue joining the RHEL 7.9 master. The RHEL 8.5 replicas have no issues joining the RHEL 8.5 master. They only difference is RHEL 8.5 replicas trying to join the RHEL 7.9 master. 

The only changes made to the lab was the introduction of RHEL 8.5. The one way I fixed this was downgrading the three openjdk packages that IdM installs. One person from Brazil fixed it by downgrading pki-ca and pki-kra.

How did you try and reproduce it? I am running two VMs on a ESXi host, 2 cpus and 4gigs of memory with a 30gb drive. RHEL 7.9 ISO and RHEL 8.5 ISO. I have also tested with a RHEL 8.4 ISO that was upgraded to RHEL 8.5.

Comment 3 Rob Crittenden 2021-11-11 21:06:46 UTC
If you had a workaround you should have mentioned that in the original report.

I am running:

pki-ca-10.11.2-2.module+el8.5.0+12735+8eb38ccc.noarch
java-1.8.0-openjdk-headless-1.8.0.312.b07-1.el8_4.x86_64

Do you have FIPS enabled?

Comment 4 Mike Ralph 2021-11-11 21:16:24 UTC
FIPS is not enabled. The packages that I have are:

java-1.8.0-openjdk.x86_64             1:1.8.0.312.b07-1.el8_4
java-1.8.0-openjdk-devel.x86_64       1:1.8.0.312.b07-1.el8_4
java-1.8.0-openjdk-headless.x86_64    1:1.8.0.312.b07-1.el8_4
pki-ca.noarch                         10.11.2-2.module+el8.5.0+12735+8eb38ccc                                                                      
pki-kra.noarch                        10.11.2-2.module+el8.5.0+12735+8eb38ccc

Comment 5 Alexander Bokovoy 2021-11-11 21:32:22 UTC
Similar report on freeipa-users@ list week. https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org/thread/2WHHIALAH7EPFQAXLE26YTUICOGFC5OP/

Comment 7 Mike Ralph 2021-11-12 17:39:31 UTC
I have just tried downgrading the pki-ca/kra packages and still get the same failure so I think, in my experience, this is just a openjdk issue.

Comment 9 Florence Blanc-Renaud 2021-12-07 07:48:23 UTC
This issue is puzzling as I can't reproduce it either. I tried with the same packages as stated in comment #c6 (that failed for Mike) but the installation went through without any issue.

Mike, can you provide the full logs from both server and replica (/var/log/httpd/ and /var/log/pki/)?

Also check that both the server and replica are properly configured from DNS standpoint:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/linux_domain_identity_authentication_and_policy_guide/installing-ipa#dns-reqs
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/installing_identity_management/preparing-the-system-for-ipa-server-installation_installing-identity-management#host-name-and-dns-requirements-for-ipa_preparing-the-system-for-ipa-server-installation

Comment 10 Mike Ralph 2021-12-08 02:15:14 UTC
(In reply to Florence Blanc-Renaud from comment #9)
> This issue is puzzling as I can't reproduce it either. I tried with the same
> packages as stated in comment #c6 (that failed for Mike) but the
> installation went through without any issue.
> 
> Mike, can you provide the full logs from both server and replica
> (/var/log/httpd/ and /var/log/pki/)?
> 
> Also check that both the server and replica are properly configured from DNS
> standpoint:
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/
> html/linux_domain_identity_authentication_and_policy_guide/installing-
> ipa#dns-reqs
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/
> html/installing_identity_management/preparing-the-system-for-ipa-server-
> installation_installing-identity-management#host-name-and-dns-requirements-
> for-ipa_preparing-the-system-for-ipa-server-installation

Florence,
This is the same ansible script I use to install all of my idm labs and it has always worked in the past whether it was RHEL 7.9 or RHEL 8.4 or below.

Comment 12 Florence Blanc-Renaud 2021-12-16 13:30:56 UTC
Hi Mike,
please attach the logs from the master (or a full sos report from both master and replica).
You mention an ansible script, can you also provide it?

Comment 16 Florence Blanc-Renaud 2021-12-17 15:56:50 UTC
The only strange thing I could spot was that /etc/resolv.conf on the master should point to itself (since it's a DNS server), but that cannot explain the issue.

The replica installation takes place between 2021-11-18T13:28:12Z and 2021-11-18T13:32:15Z.
On the server, the /var/log/pki/pki-tomcat/localhost_access_log.2021-11-18.txt file properly shows the interactions between the replica and the server, until the failing operation that doesn't get logged at all, probably because the operation doesn't even reach the PKI server on the master. For instance:
192.168.110.78 - admin-idm8.lab7.example.com [18/Nov/2021:08:30:16 -0500] "GET /ca/rest/account/login HTTP/1.1" 200 222
192.168.110.78 - admin-idm8.lab7.example.com [18/Nov/2021:08:30:26 -0500] "GET /ca/rest/securityDomain/installToken?hostname=idm8.lab7.example.com&subsystem=CA HTTP/1.1" 200 31
^^ above shows that the replica is able to connect to the master and use /ca/rest/account/login rest api (note that the communication may happen on a different port, though).

When the replica installation works properly, we should see something similar to:
<replica IP@> - ipara [date] "GET /ca/rest/account/login HTTP/1.1" 200 218
<replica IP@> - ipara [date] "POST /ca/rest/profiles/raw HTTP/1.1" 409 158

It's possible to use curl to simulate the call:
[idm8]# curl -v --cert /var/lib/ipa/ra-agent.pem --key /var/lib/ipa/ra-agent.key https://idm7.lab7.example.com:8443/ca/rest/account/login

Could you try this command (either with -v or with --trace /tmp/output.txt) and check if it provides additional information? And check if the firewall is blocking the port 8443 with

[idm8]# telnet idm7.lab7.example.com 8443

Comment 17 Mike Ralph 2021-12-17 18:57:06 UTC
According to the documentation, port 8443 is not necessary to open (https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/linux_domain_identity_authentication_and_policy_guide/installing-ipa#prereq-ports) and it is not part of any of the services that the doc say you can enable.

Running the curl from idm8 to both idm and the rep:
root@idm8 ~]# curl -v --cert /var/lib/ipa/ra-agent.pem --key /var/lib/ipa/ra-agent.key https://idm.lab7.example.com:8443/ca/rest/account/login
*   Trying 192.168.110.72...
* TCP_NODELAY set
* connect to 192.168.110.72 port 8443 failed: No route to host
* Failed to connect to idm.lab7.example.com port 8443: No route to host
* Closing connection 0
curl: (7) Failed to connect to idm.lab7.example.com port 8443: No route to host
[root@idm8 ~]# curl -v --cert /var/lib/ipa/ra-agent.pem --key /var/lib/ipa/ra-agent.key https://rep.lab7.example.com:8443/ca/rest/account/login
*   Trying 192.168.110.73...
* TCP_NODELAY set
* connect to 192.168.110.73 port 8443 failed: No route to host
* Failed to connect to rep.lab7.example.com port 8443: No route to host
* Closing connection 0
curl: (7) Failed to connect to rep.lab7.example.com port 8443: No route to host

Only tcp6 listening for port 8443
[root@idm ~]# netstat -an | grep 8443
tcp6       0      0 :::8443                 :::*                    LISTEN     
[root@idm ~]#

Comment 18 Alexander Bokovoy 2021-12-17 19:59:27 UTC
Mike, tcp6 listeners are special on Linux. Please see ipv6(7) manual page:


       IPv4 connections can be handled with the v6 API by using the v4-mapped-on-v6 address type; thus a program needs to support only this API type to support both protocols.  This is handled transparently by the address handling functions in the C library.

       IPv4 and IPv6 share the local port space.  When you get an IPv4 connection or packet to an IPv6 socket, its source address will be mapped to v6 and it will be mapped to v6.

That ':::8443' means listening on both IPv4 and IPv6 on all interfaces on port 8443.

Please see https://vda.li/drafts/firewall-considerations.txt for detailed traffic flow considerations.

Comment 19 Sneha Veeranki 2022-01-14 14:53:59 UTC
Unable to install RHCS CA even after openjdk was downgraded from java-1.8.0-openjdk java-1.8.0-openjdk-devel java-1.8.0-openjdk-headless to

# rpm -qa java*
java-1.8.0-openjdk-devel-1.8.0.292.b10-1.el8_4.x86_64
javapackages-filesystem-5.3.0-1.module+el8+2447+6f56d9a6.noarch
java-11-openjdk-javadoc-zip-11.0.13.0.8-4.el8_5.x86_64
javapackages-tools-5.3.0-1.module+el8+2447+6f56d9a6.noarch
java-1.8.0-openjdk-1.8.0.292.b10-1.el8_4.x86_64
java-1.8.0-openjdk-javadoc-1.8.0.312.b07-2.el8_5.noarch
java-11-openjdk-javadoc-11.0.13.0.8-4.el8_5.x86_64
java-17-openjdk-javadoc-17.0.1.0.12-2.el8_5.x86_64
java-1.8.0-openjdk-headless-1.8.0.292.b10-1.el8_4.x86_64
java-1.8.0-openjdk-javadoc-zip-1.8.0.312.b07-2.el8_5.noarch
java-17-openjdk-javadoc-zip-17.0.1.0.12-2.el8_5.x86_64

Tested on RHEL 85 on a FIPS + STIG enabled machine

# rpm -qa redhat-pki*
redhat-pki-base-10.12.4-1.module+el8pki+13844+b6277488.noarch
redhat-pki-base-java-10.12.4-1.module+el8pki+13844+b6277488.noarch
redhat-pki-kra-10.12.4-1.module+el8pki+13844+b6277488.noarch
redhat-pki-console-theme-10.12.4-1.module+el8pki+13844+b6277488.noarch
redhat-pki-server-theme-10.12.4-1.module+el8pki+13844+b6277488.noarch
redhat-pki-console-10.12.4-1.module+el8pki+13844+b6277488.noarch
redhat-pki-server-10.12.4-1.module+el8pki+13844+b6277488.noarch
redhat-pki-ca-10.12.4-1.module+el8pki+13844+b6277488.noarch
redhat-pki-ocsp-10.12.4-1.module+el8pki+13844+b6277488.noarch
redhat-pki-10.12.4-1.module+el8pki+13844+b6277488.x86_64
redhat-pki-javadoc-10.12.4-1.module+el8pki+13844+b6277488.noarch
redhat-pki-symkey-10.12.4-1.module+el8pki+13844+b6277488.x86_64
redhat-pki-tools-10.12.4-1.module+el8pki+13844+b6277488.x86_64
redhat-pki-acme-10.12.4-1.module+el8pki+13844+b6277488.noarch
redhat-pki-tks-10.12.4-1.module+el8pki+13844+b6277488.noarch
redhat-pki-tps-10.12.4-1.module+el8pki+13844+b6277488.x86_64


CA logs:

Loading deployment configuration from ca.cfg.
Installation log: /var/log/pki/pki-ca-spawn.20220114095127.log
Installing CA into /var/lib/pki/topology-02-CA.

Installation failed:
<!doctype html><html lang="en"><head><title>HTTP Status 500 – Internal Server Error</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 500 – Internal Server Error</h1><hr class="line" /><p><b>Type</b> Exception Report</p><p><b>Message</b> org.apache.jasper.JasperException: Unable to compile class for JSP</p><p><b>Description</b> The server encountered an unexpected condition that prevented it from fulfilling the request.</p><p><b>Exception</b></p><pre>org.apache.jasper.JasperException: org.apache.jasper.JasperException: Unable to compile class for JSP
        org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:605)
        org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:423)
        org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:385)
        org.apache.jasper.servlet.JspServlet.service(JspServlet.java:329)
        javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
        sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        java.lang.reflect.Method.invoke(Method.java:498)
        org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)
        org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)
        java.security.AccessController.doPrivileged(Native Method)
        javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
        org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)
        org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:170)
        java.security.AccessController.doPrivileged(Native Method)
        org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        java.lang.reflect.Method.invoke(Method.java:498)
        org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)
        org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)
        java.security.AccessController.doPrivileged(Native Method)
        javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
        org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)
        org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
</pre><p><b>Root Cause</b></p><pre>org.apache.jasper.JasperException: Unable to compile class for JSP
        org.apache.jasper.compiler.DefaultErrorHandler.javacError(DefaultErrorHandler.java:117)
        org.apache.jasper.compiler.ErrorDispatcher.javacError(ErrorDispatcher.java:228)
        org.apache.jasper.compiler.AntCompiler.generateClass(AntCompiler.java:266)
        org.apache.jasper.compiler.Compiler.compile(Compiler.java:392)
        org.apache.jasper.compiler.Compiler.compile(Compiler.java:362)
        org.apache.jasper.compiler.Compiler.compile(Compiler.java:346)
        org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:605)
        org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:400)
        org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:385)
        org.apache.jasper.servlet.JspServlet.service(JspServlet.java:329)
        javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
        sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        java.lang.reflect.Method.invoke(Method.java:498)
        org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)
        org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)
        java.security.AccessController.doPrivileged(Native Method)
        javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
        org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)
        org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:170)
        java.security.AccessController.doPrivileged(Native Method)
        org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        java.lang.reflect.Method.invoke(Method.java:498)
        org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)
        org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)
        java.security.AccessController.doPrivileged(Native Method)
        javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
        org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)
        org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
</pre><p><b>Root Cause</b></p><pre>Compile failed; see the compiler error output for details.
        org.apache.tools.ant.taskdefs.Javac.compile(Javac.java:1425)
        org.apache.tools.ant.taskdefs.Javac.execute(Javac.java:1133)
        org.apache.jasper.compiler.AntCompiler.generateClass(AntCompiler.java:232)
        org.apache.jasper.compiler.Compiler.compile(Compiler.java:392)
        org.apache.jasper.compiler.Compiler.compile(Compiler.java:362)
        org.apache.jasper.compiler.Compiler.compile(Compiler.java:346)
        org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:605)
        org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:400)
        org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:385)
        org.apache.jasper.servlet.JspServlet.service(JspServlet.java:329)
        javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
        sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        java.lang.reflect.Method.invoke(Method.java:498)
        org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)
        org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)
        java.security.AccessController.doPrivileged(Native Method)
        javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
        org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)
        org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
</pre><p><b>Root Cause</b></p><pre>Compile failed; see the compiler error output for details.
        org.apache.tools.ant.taskdefs.Javac.compile(Javac.java:1425)
        org.apache.tools.ant.taskdefs.Javac.execute(Javac.java:1133)
        org.apache.jasper.compiler.AntCompiler.generateClass(AntCompiler.java:232)
        org.apache.jasper.compiler.Compiler.compile(Compiler.java:392)
        org.apache.jasper.compiler.Compiler.compile(Compiler.java:362)
        org.apache.jasper.compiler.Compiler.compile(Compiler.java:346)
        org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:605)
        org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:400)
        org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:385)
        org.apache.jasper.servlet.JspServlet.service(JspServlet.java:329)
        javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
        sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        java.lang.reflect.Method.invoke(Method.java:498)
        org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)
        org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)
        java.security.AccessController.doPrivileged(Native Method)
        javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
        org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)
        org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:170)
        java.security.AccessController.doPrivileged(Native Method)
        org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        java.lang.reflect.Method.invoke(Method.java:498)
        org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)
        org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)
        java.security.AccessController.doPrivileged(Native Method)
        javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
        org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)
        org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
</pre><p><b>Note</b> The full stack trace of the root cause is available in the server logs.</p><hr class="line" /><h3>Apache Tomcat/9.0.30</h3></body></html>

Please check the CA logs in /var/log/pki/topology-02-CA/ca.

Comment 20 Rob Crittenden 2022-01-14 15:07:22 UTC
(In reply to Sneha Veeranki from comment #19)
> Unable to install RHCS CA even after openjdk was downgraded from
> java-1.8.0-openjdk java-1.8.0-openjdk-devel java-1.8.0-openjdk-headless to

This does not appear to be the same issue but it's hard to tell with no logs available.

Given the error "Unable to compile class for JSP" I'd suggest filing a new bug against either pki-core (recommended) or the jdk.

I'd also suggest re-trying without the STIG enabled to try to simplify the reproducer.

Comment 21 Rob Crittenden 2022-01-27 19:01:47 UTC
Flo provided me a good analysis offline.

Importing the profiles overrides the normal CA port to 8443 because Apache is not setup yet. I think the intention was to use the local CA though, not the remote one. Otherwise I think we'd have run into this before. I can either see about twiddling with 443 vs 8443 or checking on the hostname being used for the connection.

Note that to properly test this firewalld needs to be enabled on the 7.9 server and configured to only allow the standard ports. With that reproduction is trivial.

Comment 22 Rob Crittenden 2022-01-27 19:27:55 UTC
Confirmed Flo's theory. Because the CA installation isn't "complete" because the entry doesn't exist in cn=masters yet this is seen:

Lookup failed: Preferred host replica.example.test does not provide CA.

So it falls back to another configured CA and in this case we have only one, the 7.9 CA. So if we use 443 instead of 8443 the profile add is attempted and fails with an "Invalid profile" because it knows nothing of ACME, but this failure is logged and warned.

Comment 23 Rob Crittenden 2022-01-28 15:27:20 UTC
Linked to upstream issue https://pagure.io/freeipa/issue/9100

Comment 24 Rob Crittenden 2022-02-03 18:37:28 UTC
Fixed upstream
master:
https://pagure.io/freeipa/c/8c241869dda779d38fee4c23e44c89eb81a2a4d4

Comment 25 Florence Blanc-Renaud 2022-02-04 08:33:48 UTC
Fixed upstream
ipa-4-9:
https://pagure.io/freeipa/c/edb216849e4f47d6cae95981edf0c3fe2653fd7a

Comment 26 Florence Blanc-Renaud 2022-02-04 08:47:22 UTC
In order to reproduce:
----------------------

- install a RHEL 7.9 master with firewalld enabled, open only the ports listed in https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/linux_domain_identity_authentication_and_policy_guide/installing-ipa#prereq-ports:
# firewall-cmd --permanent --add-service={freeipa-ldap,freeipa-ldaps,dns}

- from the replica, ensure that server:8443 is not accessible:
# dnf install netcat
# nc -zv server.ipa.test 8443
nc: connect to server.ipa.test (10.0.144.249) port 8443 (tcp) failed: No route to host

- install the replica with --setup-ca option

Without the fix, the installation fails while configuring the CA instance on the replica. With the fix, the installation succeeds.

Comment 34 anuja 2022-03-04 10:09:25 UTC
Verified using stpeps mentioned in comment #26
Using versions:
ipa-server : ipa-server-4.6.8-5.el7_9.10.x86_64
ipa-replica : ipa-server-4.9.8-6.module+el8.6.0+14224+4c38d4ea.x86_64

+++++++++++++++++++ipa-server++++++++++++++++++++++++++++++=

[root@master ~]# firewall-cmd --list-services
dhcpv6-client dns freeipa-ldap freeipa-ldaps ssh
[root@master ~]# rpm -qa ipa-server
ipa-server-4.6.8-5.el7_9.10.x86_64
+++++++++++++++++++ipa-replica++++++++++++++++++++++++++++++=
[root@replica ~]# nc -zv master.testrealm.test 8443
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: No route to host.
[root@replica ~]# rpm -qa ipa-server
ipa-server-4.9.8-6.module+el8.6.0+14224+4c38d4ea.x86_64
[root@replica ~]# head /var/log/ipareplica-install.log 
2022-02-28T09:55:19Z DEBUG Logging to /var/log/ipareplica-install.log
2022-02-28T09:55:19Z DEBUG ipa-replica-install was invoked with arguments [] and options: {'unattended': True, 'ip_addresses': [CheckedIPAddress('x.x.x.x')], 'domain_name': 'testrealm.test', 'servers': None, 'realm_name': None, 'host_name': 'replica.testrealm.test', 'principal': 'admin', 'hidden_replica': False, 'setup_adtrust': False, 'setup_ca': True, 'setup_kra': False, 'setup_dns': True, 'no_pkinit': False, 'no_ui_redirect': False, 'dirsrv_config_file': None, 'skip_mem_check': False, 'dirsrv_cert_files': None, 'http_cert_files': None, 'pkinit_cert_files': None, 'dirsrv_cert_name': None, 'http_cert_name': None, 'pkinit_cert_name': None, 'keytab': None, 'mkhomedir': False, 'force_join': False, 'ntp_servers': None, 'ntp_pool': None, 'no_ntp': False, 'force_ntpd': False, 'ssh_trust_dns': False, 'no_ssh': False, 'no_sshd': False, 'no_dns_sshfp': False, 'skip_schema_check': False, 'pki_config_override': None, 'allow_zone_overlap': False, 'reverse_zones': None, 'no_reverse': False, 'auto_reverse': False, 'forwarders': None, 'no_forwarders': True, 'auto_forwarders': False, 'forward_policy': None, 'no_dnssec_validation': False, 'no_host_dns': False, 'add_agents': False, 'enable_compat': False, 'no_msdcs': False, 'skip_conncheck': False, 'add_sids': False, 'netbios_name': None, 'rid_base': None, 'secondary_rid_base': None, 'verbose': False, 'quiet': False, 'log_file': None}
2022-02-28T09:55:19Z DEBUG IPA version 4.9.8-6.module+el8.6.0+14224+4c38d4ea
2022-02-28T09:55:19Z DEBUG IPA platform rhel
2022-02-28T09:55:19Z DEBUG IPA os-release Red Hat Enterprise Linux 8.6 (Ootpa)
2022-02-28T09:55:19Z DEBUG svmem(total=3913011200, available=3380109312, percent=13.6, used=271208448, free=2258661376, active=454959104, inactive=975781888, buffers=4018176, cached=1379123200, shared=10469376)
2022-02-28T09:55:19Z DEBUG Available memory is 3380109312B
2022-02-28T09:55:19Z DEBUG Searching for an interface of IP address: ::1
2022-02-28T09:55:19Z DEBUG Testing local IP address: ::1/128 (interface: lo)
2022-02-28T09:55:19Z DEBUG Starting external process
[root@replica ~]# 

[root@replica ~]# tail /var/log/ipareplica-install.log 
2022-02-28T10:02:21Z DEBUG stdout=
2022-02-28T10:02:21Z DEBUG stderr=
2022-02-28T10:02:21Z DEBUG Starting external process
2022-02-28T10:02:21Z DEBUG args=['/bin/systemctl', 'is-active', 'ipa.service']
2022-02-28T10:02:21Z DEBUG Process finished, return code=0
2022-02-28T10:02:21Z DEBUG stdout=active

2022-02-28T10:02:21Z DEBUG stderr=
2022-02-28T10:02:21Z DEBUG Restart of ipa.service complete
2022-02-28T10:02:21Z INFO The ipa-replica-install command was successful
[root@replica ~]#

Comment 36 errata-xmlrpc 2022-05-10 14:09:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (idm:client and idm:DL1 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:1884


Note You need to log in before you can comment on or make changes to this bug.