Bug 2061424

Summary: Update to tomcat-9.0.59 breaks CA installation in IPA
Product: [Fedora] Fedora Reporter: Chris Kelley <ckelley>
Component: tomcatAssignee: Coty Sutherland <csutherl>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: rawhideCC: alee, awilliam, coolsvap, csutherl, frenaud, gzaronikas, huwang, ivan.afonichev, java-sig-commits, krzysztof.daniel, robatino
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: openqa
Fixed In Version: tomcat-9.0.59-2.fc37 tomcat-9.0.59-3.fc37 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-03-08 20:33:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2009537    

Description Chris Kelley 2022-03-07 14:15:23 UTC 
Description of problem:
The recent update from tomcat-9.0.56 to tomcat-9.0.59 in Rawhide prevents CA installation in IPA. Reverting to 56 fixes the installation issue.

## Pull request that introduces this issue
https://src.fedoraproject.org/rpms/tomcat/pull-request/8

## More detail in the IPA/Dogtag issues raised for this
https://pagure.io/freeipa/issue/9122
https://github.com/dogtagpki/pki/issues/3927

Version-Release number of selected component (if applicable):


How reproducible:
Very

Steps to Reproduce:
1. See upstream issues or details
2.
3.

Actual results:
CA installation with IPA fails with latest Rawhide

Expected results:
CA installation with IPA succeeds with latest Rawhide

Additional info:
Comment 1 Adam Williamson 2022-03-08 01:22:31 UTC 
Yeah, openQA hit this too. Only just got around to checking the logs, been busy with F36. This is a clear F37 Beta blocker, per "It must be possible to configure a Fedora Server system installed according to the above criteria as a FreeIPA domain controller, using the official deployment tools provided in the distribution FreeIPA packages" - https://fedoraproject.org/wiki/Basic_Release_Criteria#FreeIPA_server_requirements

Mar 07 02:36:37 ipa001.test.openqa.fedoraproject.org systemd[1]: Starting pki-tomcatd - PKI Tomcat Server pki-tomcat...
Mar 07 02:36:39 ipa001.test.openqa.fedoraproject.org pcscd[2161]: 70379504 auth.c:137:IsClientAuthorized() Process 3000 (user: 17) is NOT authorized for action: access_pcsc
Mar 07 02:36:39 ipa001.test.openqa.fedoraproject.org pcscd[2161]: 00000614 winscard_svc.c:335:ContextThread() Rejected unauthorized PC/SC client
Mar 07 02:36:39 ipa001.test.openqa.fedoraproject.org pcscd[2161]: 00059234 auth.c:137:IsClientAuthorized() Process 3000 (user: 17) is NOT authorized for action: access_pcsc
Mar 07 02:36:39 ipa001.test.openqa.fedoraproject.org pcscd[2161]: 00000248 winscard_svc.c:335:ContextThread() Rejected unauthorized PC/SC client
Mar 07 02:36:39 ipa001.test.openqa.fedoraproject.org pcscd[2161]: 00073970 auth.c:137:IsClientAuthorized() Process 3000 (user: 17) is NOT authorized for action: access_pcsc
Mar 07 02:36:39 ipa001.test.openqa.fedoraproject.org pcscd[2161]: 00000225 winscard_svc.c:335:ContextThread() Rejected unauthorized PC/SC client
Mar 07 02:36:39 ipa001.test.openqa.fedoraproject.org pcscd[2161]: 00061217 auth.c:137:IsClientAuthorized() Process 3000 (user: 17) is NOT authorized for action: access_pcsc
Mar 07 02:36:39 ipa001.test.openqa.fedoraproject.org pcscd[2161]: 00000224 winscard_svc.c:335:ContextThread() Rejected unauthorized PC/SC client
Mar 07 02:36:40 ipa001.test.openqa.fedoraproject.org systemd[1]: Started pki-tomcatd - PKI Tomcat Server pki-tomcat.
Mar 07 02:36:40 ipa001.test.openqa.fedoraproject.org audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pki-tomcatd@pki-tomcat comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar 07 02:36:40 ipa001.test.openqa.fedoraproject.org server[3051]: Java virtual machine used: /usr/lib/jvm/jre-17-openjdk/bin/java
Mar 07 02:36:40 ipa001.test.openqa.fedoraproject.org server[3051]: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:
Mar 07 02:36:40 ipa001.test.openqa.fedoraproject.org server[3051]: main class used: org.apache.catalina.startup.Bootstrap
Mar 07 02:36:40 ipa001.test.openqa.fedoraproject.org server[3051]: flags used: -Dcom.redhat.fips=false
Mar 07 02:36:40 ipa001.test.openqa.fedoraproject.org server[3051]: options used: -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager     -Djava.security.manager     -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy
Mar 07 02:36:40 ipa001.test.openqa.fedoraproject.org server[3051]: arguments used: start
Mar 07 02:36:40 ipa001.test.openqa.fedoraproject.org server[3051]: WARNING: A command line option has enabled the Security Manager
Mar 07 02:36:40 ipa001.test.openqa.fedoraproject.org server[3051]: WARNING: The Security Manager is deprecated and will be removed in a future release
Mar 07 02:36:41 ipa001.test.openqa.fedoraproject.org pcscd[2161]: 02095788 auth.c:137:IsClientAuthorized() Process 3051 (user: 17) is NOT authorized for action: access_pcsc
Mar 07 02:36:41 ipa001.test.openqa.fedoraproject.org pcscd[2161]: 00000297 winscard_svc.c:335:ContextThread() Rejected unauthorized PC/SC client
Mar 07 02:36:41 ipa001.test.openqa.fedoraproject.org pcscd[2161]: 00040568 auth.c:137:IsClientAuthorized() Process 3051 (user: 17) is NOT authorized for action: access_pcsc
Mar 07 02:36:41 ipa001.test.openqa.fedoraproject.org pcscd[2161]: 00000190 winscard_svc.c:335:ContextThread() Rejected unauthorized PC/SC client
Mar 07 02:36:41 ipa001.test.openqa.fedoraproject.org pcscd[2161]: 00040523 auth.c:137:IsClientAuthorized() Process 3051 (user: 17) is NOT authorized for action: access_pcsc
Mar 07 02:36:41 ipa001.test.openqa.fedoraproject.org pcscd[2161]: 00000651 winscard_svc.c:335:ContextThread() Rejected unauthorized PC/SC client
Mar 07 02:36:41 ipa001.test.openqa.fedoraproject.org pcscd[2161]: 00041781 auth.c:137:IsClientAuthorized() Process 3051 (user: 17) is NOT authorized for action: access_pcsc
Mar 07 02:36:41 ipa001.test.openqa.fedoraproject.org pcscd[2161]: 00000647 winscard_svc.c:335:ContextThread() Rejected unauthorized PC/SC client
Mar 07 02:36:42 ipa001.test.openqa.fedoraproject.org server[3051]: WARNING: Some of the specified [protocols] are not supported by the SSL engine and have been skipped: [[TLSv1, TLSv1.1]]
Mar 07 02:36:46 ipa001.test.openqa.fedoraproject.org ns-slapd[2405]: [07/Mar/2022:05:36:46.711707018 -0500] - ERR - ipa-topology-plugin - ipa_topo_util_get_replica_conf: server configuration missing
Mar 07 02:36:46 ipa001.test.openqa.fedoraproject.org ns-slapd[2405]: [07/Mar/2022:05:36:46.716634612 -0500] - ERR - ipa-topology-plugin - ipa_topo_util_get_replica_conf: cannot create replica
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]: WARNING: The SHA1 algorithm used in com.netscape.ca.serviceCheckChallenge::<init>:1631 is deprecated. Use a more secure algorithm.
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]: WARNING: The SHA1 algorithm used in com.netscape.cmscore.authentication.ChallengePhraseAuthentication::init:116 is deprecated. Use a more secure algorithm.
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]: SEVERE: The required Server component failed to start so Tomcat is unable to start.
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina]]
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         at org.apache.catalina.util.LifecycleBase.handleSubClassException(LifecycleBase.java:440)
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:198)
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         at org.apache.catalina.core.StandardService.startInternal(StandardService.java:432)
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:927)
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         at org.apache.catalina.startup.Catalina.start(Catalina.java:772)
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         at java.base/java.lang.reflect.Method.invoke(Method.java:568)
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:345)
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:476)
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]: Caused by: java.lang.ExceptionInInitializerError
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         at org.apache.tomcat.util.threads.TaskThreadFactory.newThread(TaskThreadFactory.java:63)
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:630)
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:920)
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         at java.base/java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1593)
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:346)
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:680)
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         at org.apache.tomcat.util.threads.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:138)
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:951)
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:263)
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         ... 11 more
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]: Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field private java.security.AccessControlContext java.lang.Thread.inheritedAccessControlContext accessible: module java.base does not "opens java.lang" to unnamed module @35851384
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         at java.base/java.lang.reflect.Field.setAccessible(Field.java:172)
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         at org.apache.tomcat.util.security.PrivilegedSetAccessControlContext.<clinit>(PrivilegedSetAccessControlContext.java:41)
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]:         ... 21 more
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]: WARNING: When running on Java 9 or later you need to add "--add-opens=java.base/java.io=null" to the JVM command line arguments to enable ObjectStream cache memory leak protection. Alternatively, you can suppress this warning by disabling ObjectStream class cache memory leak protection.
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]: WARNING: When running on Java 9 or later you need to add "--add-opens=java.base/java.lang=null" to the JVM command line arguments to enable ThreadLocal memory leak detection. Alternatively, you can suppress this warning by disabling ThreadLocal memory leak detection.
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]: WARNING: When running on Java 9 or later you need to add "--add-opens=java.rmi/sun.rmi.transport=null" to the JVM command line arguments to enable RMI Target memory leak detection. Alternatively, you can suppress this warning by disabling RMI Target memory leak detection.
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]: WARNING: When running on Java 9 or later you need to add "--add-opens=java.base/java.io=null" to the JVM command line arguments to enable ObjectStream cache memory leak protection. Alternatively, you can suppress this warning by disabling ObjectStream class cache memory leak protection.
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]: WARNING: When running on Java 9 or later you need to add "--add-opens=java.base/java.lang=null" to the JVM command line arguments to enable ThreadLocal memory leak detection. Alternatively, you can suppress this warning by disabling ThreadLocal memory leak detection.
Mar 07 02:36:47 ipa001.test.openqa.fedoraproject.org server[3051]: WARNING: When running on Java 9 or later you need to add "--add-opens=java.rmi/sun.rmi.transport=null" to the JVM command line arguments to enable RMI Target memory leak detection. Alternatively, you can suppress this warning by disabling RMI Target memory leak detection.
Mar 07 02:36:48 ipa001.test.openqa.fedoraproject.org server[3051]: WARNING: The web application [ca] appears to have started a thread named [AsyncLoader watchdog] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
Mar 07 02:36:48 ipa001.test.openqa.fedoraproject.org server[3051]:  java.base.2/java.lang.Object.wait(Native Method)
Mar 07 02:36:48 ipa001.test.openqa.fedoraproject.org server[3051]:  java.base.2/java.lang.Object.wait(Object.java:338)
Mar 07 02:36:48 ipa001.test.openqa.fedoraproject.org server[3051]:  java.base.2/java.util.TimerThread.mainLoop(Timer.java:537)
Mar 07 02:36:48 ipa001.test.openqa.fedoraproject.org server[3051]:  java.base.2/java.util.TimerThread.run(Timer.java:516)
Mar 07 02:36:48 ipa001.test.openqa.fedoraproject.org server[3051]: WARNING: When running on Java 9 or later you need to add "--add-opens=java.base/java.io=null" to the JVM command line arguments to enable ObjectStream cache memory leak protection. Alternatively, you can suppress this warning by disabling ObjectStream class cache memory leak protection.
Mar 07 02:36:48 ipa001.test.openqa.fedoraproject.org server[3051]: WARNING: When running on Java 9 or later you need to add "--add-opens=java.base/java.lang=null" to the JVM command line arguments to enable ThreadLocal memory leak detection. Alternatively, you can suppress this warning by disabling ThreadLocal memory leak detection.
Mar 07 02:36:48 ipa001.test.openqa.fedoraproject.org server[3051]: WARNING: When running on Java 9 or later you need to add "--add-opens=java.rmi/sun.rmi.transport=null" to the JVM command line arguments to enable RMI Target memory leak detection. Alternatively, you can suppress this warning by disabling RMI Target memory leak detection.
Comment 2 Adam Williamson 2022-03-08 16:27:21 UTC 
Note: as well as affecting initial deployment, it looks like FreeIPA also often fails to start correctly on existing systems updated/upgraded to this version of tomcat.
Comment 3 Coty Sutherland 2022-03-08 19:21:29 UTC 
I provided a note about the issue on https://github.com/dogtagpki/pki/issues/3927#issuecomment-1062117221.

I'm going to push a fix shortly, build, and test to verify.
Comment 4 Fedora Update System 2022-03-08 20:30:52 UTC 
FEDORA-2022-bfc7b60b73 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2022-bfc7b60b73
Comment 5 Fedora Update System 2022-03-08 20:33:40 UTC 
FEDORA-2022-bfc7b60b73 has been pushed to the Fedora 37 stable repository.
If problem still persists, please make note of it in this bug report.
Comment 6 Coty Sutherland 2022-03-08 20:55:02 UTC 
FYI, I confirmed that the updated package resolves the issue and the `ipa-server-install --install` call completes successfully in my test env.
Comment 7 Fedora Update System 2022-03-10 20:50:41 UTC 
FEDORA-2022-de1aefcca1 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2022-de1aefcca1
Comment 8 Fedora Update System 2022-03-10 20:52:38 UTC 
FEDORA-2022-de1aefcca1 has been pushed to the Fedora 37 stable repository.
If problem still persists, please make note of it in this bug report.