In current Fedora Rawhide, FreeIPA deployment fails, with a traceback in pki-tomcat: 2020-08-03 07:50:47 [main] INFO: CA subsystem started 2020-08-03 07:50:48 [https-jsse-nio-8443-exec-3] SEVERE: Servlet.service() for servlet [Resteasy] in context with path [/ca] threw exception [javax/xml/bind/annotation/XmlElement] with root cause java.lang.ClassNotFoundException: javax.xml.bind.annotation.XmlElement at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) at com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector.<init>(JaxbAnnotationIntrospector.java:139) at com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector.<init>(JaxbAnnotationIntrospector.java:126) at com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector.<init>(JaxbAnnotationIntrospector.java:118) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) at java.base/java.lang.Class.newInstance(Class.java:584) at com.fasterxml.jackson.jaxrs.json.JsonMapperConfigurator._resolveIntrospector(JsonMapperConfigurator.java:111) at com.fasterxml.jackson.jaxrs.json.JsonMapperConfigurator._resolveIntrospectors(JsonMapperConfigurator.java:84) at com.fasterxml.jackson.jaxrs.cfg.MapperConfiguratorBase._setAnnotations(MapperConfiguratorBase.java:120) at com.fasterxml.jackson.jaxrs.json.JsonMapperConfigurator.getDefaultMapper(JsonMapperConfigurator.java:45) at com.fasterxml.jackson.jaxrs.base.ProviderBase.locateMapper(ProviderBase.java:933) at org.jboss.resteasy.plugins.providers.jackson.ResteasyJackson2Provider.readFrom(ResteasyJackson2Provider.java:116) at org.jboss.resteasy.core.interception.AbstractReaderInterceptorContext.readFrom(AbstractReaderInterceptorContext.java:66) at org.jboss.resteasy.core.interception.ServerReaderInterceptorContext.readFrom(ServerReaderInterceptorContext.java:61) at org.jboss.resteasy.core.interception.AbstractReaderInterceptorContext.proceed(AbstractReaderInterceptorContext.java:56) at org.jboss.resteasy.core.MessageBodyParameterInjector.inject(MessageBodyParameterInjector.java:151) at org.jboss.resteasy.core.MethodInjectorImpl.injectArguments(MethodInjectorImpl.java:92) at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:115) at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295) at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249) at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:236) at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:406) at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:213) at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:228) at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) at javax.servlet.http.HttpServlet.service(HttpServlet.java:733) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:550) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:170) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:225) at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:47) at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:149) at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:145) at java.base/java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:550) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:191) at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:47) at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:149) at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:145) at java.base/java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) at com.netscape.cms.tomcat.ExternalAuthenticationValve.invoke(ExternalAuthenticationValve.java:82) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:690) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1589) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Thread.java:834) Googling around, this seems to be a Java 11 change with a known fix: https://stackoverflow.com/questions/58373147/xmlelement-annotation-not-found but I believe it's also already generally known - see e.g. https://www.spinics.net/lists/fedora-devel/msg268597.html - that dogtag is not compatible with Java 11 for other reasons. However, I can't find an actual Fedora bug on this yet, so I'm filing one. This is a clear release blocker per Basic criterion "It must be possible to configure a Fedora Server system installed according to the above criteria as a FreeIPA domain controller, using the official deployment tools provided in the distribution FreeIPA packages..." Also marking this as blocking the "Java 11 as system Java" Change at least for now.
Created attachment 1710583 [details] full /var/log tarball from a recent failure
https://src.fedoraproject.org/rpms/pki-core/c/29e9dc66f9579ea93392ec5586a6e42925010dbe attempts to get it working with Java 11 so we can assume it is going to be usable once we fix all the dependencies and incompatibilities. Let's use this bug for this purpose.
Note that Dogtag depends on glassfish-jaxb-api, which provides the missing APIs in question: https://koji.fedoraproject.org/koji/rpminfo?rpmID=22300038 So I'm really confused by the above log message; that shouldn't happen assuming it is getting loaded on the classpath (and it has historically). However, if you look in a different log file, you see messages like this earlier: 2020-08-03 07:50:45 [main] WARN: RESTEASY002145: NoClassDefFoundError: Unable to load builtin provider org.jboss.resteasy.plugins.providers.jaxb.XmlJAXBContextFinder from jar:file:/usr/share/java/resteasy/resteasy-jaxb-provider.jar!/META-INF/services/javax.ws.rs.ext.Providers ... snip stack trace ... 2020-08-03 07:50:45 [main] WARN: RESTEASY002145: NoClassDefFoundError: Unable to load builtin provider org.jboss.resteasy.plugins.providers.atom.AtomEntryProvider from jar:file:/usr/share/java/resteasy/resteasy-atom-provider.jar!/META-INF/services/javax.ws.rs.ext.Providers ... snip stack trace ... so if anything doesn't like the new Java11 stuff, it looks like it is actually resteasy. That's been on our plate to upgrade for a while, but requires packaging new stuff that isn't yet in Fedora.
Alex: per Alexander pki-core definitely *does* have Java 11 incompatibilities that need resolving even if the specific one I mentioned in the comment isn't as it seems. That's why I intentionally made the *summary* of the bug quite generic. So I don't think simply reassigning the bug to resteasy is the best idea, unless we're going to assign it back when the resteasy issue is resolved...
Adam -- not sure I follow. My reading of what Alexander says (and our experience with Debian) shows that the Dogtag proper doesn't have JDK 11 issues. Fedora's packaging of Dogtag's dependencies are what have Java 11 issues at the current moment. Most telling is that, while Debian also ships Dogtag at a similar version, they don't have this particular issue, per discussion with Timo on #freeipa. (He has one remaining Tomcat bug, but that came up recently due to versions of bnd mismatched with what upstream Tomcat was targeting). If there are specific issues with Dogtag, these should be noted and resolved. However, if the issues are in the dependencies, I think it is disingenuous to call them Dogtag bugs, any more so than a bug in glibc shouldn't result in several hundred bugs filed against every component that uses glibc. Short of rewriting Dogtag to avoid one of our most heavily used libraries (resteasy provides all our REST API infrastructure), there's no way to avoid this on Dogtag's part. We should document the bug in the component the bug is in, and provide a fix for that component. Even though Dogtag is the only consumer of Resteasy in the published Fedora repositories. If you -- or anyone else! -- would like to contribute to rewriting Dogtag to avoid resteasy, consider this a warm invitation :-) I'll happily merge the code upstream because it will save us many person-months of maintenance on a JBoss package they don't support in Fedora and RHEL. If you want to file a tracker bug blocking Fedora 33 because FreeIPA is broken, shouldn't that tracker bug be filed against FreeIPA, as that is the release-blocking component? Then we can file a separate bug against resteasy and mark it blocking this bug, and so on, until all Dogtag dependencies are fixed. My 2c.
So maybe comparing versions of dogtag-pki dependencies between what we have in fedora (which is not working) and what is in debian (which is working) can be a starting point? I can prioritize those packages (if they are mine or the Stewardship/Java SIG's), to close the difference, which should help.
Alex: I think we may be reading "so we can assume it is going to be usable once we fix all the dependencies and incompatibilities" differently. But the larger point is that, ultimately, dogtag-in-FreeIPA-in-Fedora needs to work with Java 11 and that's what this bug is for, it wasn't intended to be for one specific sub-component of that. If just fixing resteasy isn't going to make FreeIPA work again, then this bug isn't fixed when resteasy is fixed...
Discussed during the 2020-08-10 blocker review meeting: [0] The decision to classify this bug as an "AcceptedBlocker" was made as it violates the following Basic criterion: "It must be possible to configure a Fedora Server system installed according to the above criteria as a FreeIPA domain controller, using the official deployment tools provided in the distribution FreeIPA packages..." [0] https://meetbot.fedoraproject.org/fedora-blocker-review/2020-08-10/f33-blocker-review.2020-08-10-16.17.txt
This bug appears to have been reported against 'rawhide' during the Fedora 33 development cycle. Changing version to 33.
So we haven't had a Rawhide compose for a while, but in the more recent F33 composes, this appears to be mostly fixed. FreeIPA server deployment on a fresh install is working, and most client tests are also working. Kickstart client enrolment is failing, but that may well be on anaconda side. Server upgrade from F32 to F33 is also failing. Let's close this bug, and I'll file new separate bugs for the remaining issues.
*** Bug 1868822 has been marked as a duplicate of this bug. ***
Note that this was ultimately updated in pki-core to add some shims. We still need to update resteasy...
just to circle back on the issues noted in #c10 - upgrade bug is https://bugzilla.redhat.com/show_bug.cgi?id=1871990 , kickstart enrolment failure turned out to be actually a bug in the tests, nothing wrong with FreeIPA, I've fixed that and re-run them.