Bug 1866570 - FreeIPA deployment fails in current Rawhide due to various issues with Java 11
Summary: FreeIPA deployment fails in current Rawhide due to various issues with Java 11
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: pki-core
Version: 33
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: Matthew Harmsen
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: openqa AcceptedBlocker
: 1868822 (view as bug list)
Depends On:
Blocks: BetaBlocker, F33BetaBlocker Java11
TreeView+ depends on / blocked
 
Reported: 2020-08-06 00:04 UTC by Adam Williamson
Modified: 2020-08-24 21:21 UTC (History)
19 users (show)

Fixed In Version: pki-core-10.9.2-2.fc34, pki-core-10.9.2-2.fc33
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-08-24 17:12:33 UTC
Type: Bug


Attachments (Terms of Use)
full /var/log tarball from a recent failure (1.55 MB, application/octet-stream)
2020-08-06 00:06 UTC, Adam Williamson
no flags Details

Description Adam Williamson 2020-08-06 00:04:23 UTC
In current Fedora Rawhide, FreeIPA deployment fails, with a traceback in pki-tomcat:

2020-08-03 07:50:47 [main] INFO: CA subsystem started
2020-08-03 07:50:48 [https-jsse-nio-8443-exec-3] SEVERE: Servlet.service() for servlet [Resteasy] in context with path [/ca] threw exception [javax/xml/bind/annotation/XmlElement] with root cause
java.lang.ClassNotFoundException: javax.xml.bind.annotation.XmlElement
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	at com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector.<init>(JaxbAnnotationIntrospector.java:139)
	at com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector.<init>(JaxbAnnotationIntrospector.java:126)
	at com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector.<init>(JaxbAnnotationIntrospector.java:118)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
	at java.base/java.lang.Class.newInstance(Class.java:584)
	at com.fasterxml.jackson.jaxrs.json.JsonMapperConfigurator._resolveIntrospector(JsonMapperConfigurator.java:111)
	at com.fasterxml.jackson.jaxrs.json.JsonMapperConfigurator._resolveIntrospectors(JsonMapperConfigurator.java:84)
	at com.fasterxml.jackson.jaxrs.cfg.MapperConfiguratorBase._setAnnotations(MapperConfiguratorBase.java:120)
	at com.fasterxml.jackson.jaxrs.json.JsonMapperConfigurator.getDefaultMapper(JsonMapperConfigurator.java:45)
	at com.fasterxml.jackson.jaxrs.base.ProviderBase.locateMapper(ProviderBase.java:933)
	at org.jboss.resteasy.plugins.providers.jackson.ResteasyJackson2Provider.readFrom(ResteasyJackson2Provider.java:116)
	at org.jboss.resteasy.core.interception.AbstractReaderInterceptorContext.readFrom(AbstractReaderInterceptorContext.java:66)
	at org.jboss.resteasy.core.interception.ServerReaderInterceptorContext.readFrom(ServerReaderInterceptorContext.java:61)
	at org.jboss.resteasy.core.interception.AbstractReaderInterceptorContext.proceed(AbstractReaderInterceptorContext.java:56)
	at org.jboss.resteasy.core.MessageBodyParameterInjector.inject(MessageBodyParameterInjector.java:151)
	at org.jboss.resteasy.core.MethodInjectorImpl.injectArguments(MethodInjectorImpl.java:92)
	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:115)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:236)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:406)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:213)
	at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:228)
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)
	at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:550)
	at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)
	at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:170)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:225)
	at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:47)
	at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:149)
	at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:145)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)
	at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:550)
	at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)
	at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:191)
	at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:47)
	at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:149)
	at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:145)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
	at com.netscape.cms.tomcat.ExternalAuthenticationValve.invoke(ExternalAuthenticationValve.java:82)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:690)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1589)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.base/java.lang.Thread.run(Thread.java:834)

Googling around, this seems to be a Java 11 change with a known fix:

https://stackoverflow.com/questions/58373147/xmlelement-annotation-not-found

but I believe it's also already generally known - see e.g. https://www.spinics.net/lists/fedora-devel/msg268597.html - that dogtag is not compatible with Java 11 for other reasons.

However, I can't find an actual Fedora bug on this yet, so I'm filing one. This is a clear release blocker per Basic criterion "It must be possible to configure a Fedora Server system installed according to the above criteria as a FreeIPA domain controller, using the official deployment tools provided in the distribution FreeIPA packages..." Also marking this as blocking the "Java 11 as system Java" Change at least for now.

Comment 1 Adam Williamson 2020-08-06 00:06:36 UTC
Created attachment 1710583 [details]
full /var/log tarball from a recent failure

Comment 2 Alexander Bokovoy 2020-08-06 06:03:39 UTC
https://src.fedoraproject.org/rpms/pki-core/c/29e9dc66f9579ea93392ec5586a6e42925010dbe attempts to get it working with Java 11 so we can assume it is going to be usable once we fix all the dependencies and incompatibilities.
Let's use this bug for this purpose.

Comment 3 Alex Scheel 2020-08-06 13:31:37 UTC
Note that Dogtag depends on glassfish-jaxb-api, which provides the missing APIs in question:

https://koji.fedoraproject.org/koji/rpminfo?rpmID=22300038

So I'm really confused by the above log message; that shouldn't happen assuming it is getting loaded on the classpath (and it has historically).


However, if you look in a different log file, you see messages like this earlier:

2020-08-03 07:50:45 [main] WARN: RESTEASY002145: NoClassDefFoundError: Unable to load builtin provider org.jboss.resteasy.plugins.providers.jaxb.XmlJAXBContextFinder from jar:file:/usr/share/java/resteasy/resteasy-jaxb-provider.jar!/META-INF/services/javax.ws.rs.ext.Providers
... snip stack trace ...

2020-08-03 07:50:45 [main] WARN: RESTEASY002145: NoClassDefFoundError: Unable to load builtin provider org.jboss.resteasy.plugins.providers.atom.AtomEntryProvider from jar:file:/usr/share/java/resteasy/resteasy-atom-provider.jar!/META-INF/services/javax.ws.rs.ext.Providers
... snip stack trace ...



so if anything doesn't like the new Java11 stuff, it looks like it is actually resteasy.

That's been on our plate to upgrade for a while, but requires packaging new stuff that isn't yet in Fedora.

Comment 4 Adam Williamson 2020-08-06 17:57:11 UTC
Alex: per Alexander pki-core definitely *does* have Java 11 incompatibilities that need resolving even if the specific one I mentioned in the comment isn't as it seems. That's why I intentionally made the *summary* of the bug quite generic. So I don't think simply reassigning the bug to resteasy is the best idea, unless we're going to assign it back when the resteasy issue is resolved...

Comment 5 Alex Scheel 2020-08-06 20:25:28 UTC
Adam -- not sure I follow. My reading of what Alexander says (and our experience with Debian) shows that the Dogtag proper doesn't have JDK 11 issues. Fedora's packaging of Dogtag's dependencies are what have Java 11 issues at the current moment. Most telling is that, while Debian also ships Dogtag at a similar version, they don't have this particular issue, per discussion with Timo on #freeipa. (He has one remaining Tomcat bug, but that came up recently due to versions of bnd mismatched with what upstream Tomcat was targeting). 

If there are specific issues with Dogtag, these should be noted and resolved. However, if the issues are in the dependencies, I think it is disingenuous to call them Dogtag bugs, any more so than a bug in glibc shouldn't result in several hundred bugs filed against every component that uses glibc.

Short of rewriting Dogtag to avoid one of our most heavily used libraries (resteasy provides all our REST API infrastructure), there's no way to avoid this on Dogtag's part. We should document the bug in the component the bug is in, and provide a fix for that component. Even though Dogtag is the only consumer of Resteasy in the published Fedora repositories.

If you -- or anyone else! -- would like to contribute to rewriting Dogtag to avoid resteasy, consider this a warm invitation :-) I'll happily merge the code upstream because it will save us many person-months of maintenance on a JBoss package they don't support in Fedora and RHEL.


If you want to file a tracker bug blocking Fedora 33 because FreeIPA is broken, shouldn't that tracker bug be filed against FreeIPA, as that is the release-blocking component? Then we can file a separate bug against resteasy and mark it blocking this bug, and so on, until all Dogtag dependencies are fixed.

My 2c.

Comment 6 Fabio Valentini 2020-08-06 20:36:41 UTC
So maybe comparing versions of dogtag-pki dependencies between what we have in fedora (which is not working) and what is in debian (which is working) can be a starting point? I can prioritize those packages (if they are mine or the Stewardship/Java SIG's), to close the difference, which should help.

Comment 7 Adam Williamson 2020-08-06 21:13:47 UTC
Alex: I think we may be reading "so we can assume it is going to be usable once we fix all the dependencies and incompatibilities" differently. But the larger point is that, ultimately, dogtag-in-FreeIPA-in-Fedora needs to work with Java 11 and that's what this bug is for, it wasn't intended to be for one specific sub-component of that. If just fixing resteasy isn't going to make FreeIPA work again, then this bug isn't fixed when resteasy is fixed...

Comment 8 Geoffrey Marr 2020-08-10 22:17:07 UTC
Discussed during the 2020-08-10 blocker review meeting: [0]

The decision to classify this bug as an "AcceptedBlocker" was made as it violates the following Basic criterion:

"It must be possible to configure a Fedora Server system installed according to the above criteria as a FreeIPA domain controller, using the official deployment tools provided in the distribution FreeIPA packages..."

[0] https://meetbot.fedoraproject.org/fedora-blocker-review/2020-08-10/f33-blocker-review.2020-08-10-16.17.txt

Comment 9 Ben Cotton 2020-08-11 15:20:02 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 33 development cycle.
Changing version to 33.

Comment 10 Adam Williamson 2020-08-24 17:12:33 UTC
So we haven't had a Rawhide compose for a while, but in the more recent F33 composes, this appears to be mostly fixed. FreeIPA server deployment on a fresh install is working, and most client tests are also working. Kickstart client enrolment is failing, but that may well be on anaconda side. Server upgrade from F32 to F33 is also failing.

Let's close this bug, and I'll file new separate bugs for the remaining issues.

Comment 11 Alex Scheel 2020-08-24 18:29:35 UTC
*** Bug 1868822 has been marked as a duplicate of this bug. ***

Comment 12 Alex Scheel 2020-08-24 18:30:38 UTC
Note that this was ultimately updated in pki-core to add some shims. We still need to update resteasy...

Comment 13 Adam Williamson 2020-08-24 21:21:32 UTC
just to circle back on the issues noted in #c10 - upgrade bug is https://bugzilla.redhat.com/show_bug.cgi?id=1871990 , kickstart enrolment failure turned out to be actually a bug in the tests, nothing wrong with FreeIPA, I've fixed that and re-run them.


Note You need to log in before you can comment on or make changes to this bug.