https://bodhi.fedoraproject.org/updates/FEDORA-2023-7ff08f7d6f - containing resteasy-3.0.26-23.fc38 , jboss-jaxrs-2.0-api-1.0.0-23.fc38 and jackson-modules-base-2.14.2-3.fc38 - breaks FreeIPA server deployment in Rawhide. With that update included, FreeIPA deployment fails while attempting to configure the CA. /var/log/pki/pki-tomcat/ca/debug.2023-02-06.log shows this traceback: 2023-02-06 10:14:30 [https-jsse-nio-8443-exec-6] SEVERE: Servlet.service() for servlet [Resteasy] in context with path [/ca] threw exception [jakarta/xml/bind/annotation/XmlElement] with root cause java.lang.ClassNotFoundException: jakarta.xml.bind.annotation.XmlElement at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:587) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520) at com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector.<init>(JaxbAnnotationIntrospector.java:137) at com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector.<init>(JaxbAnnotationIntrospector.java:124) at com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector.<init>(JaxbAnnotationIntrospector.java:116) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) at java.base/java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:128) at java.base/jdk.internal.reflect.ReflectionFactory.newInstance(ReflectionFactory.java:347) at java.base/java.lang.Class.newInstance(Class.java:645) at com.fasterxml.jackson.jaxrs.json.JsonMapperConfigurator._resolveIntrospector(JsonMapperConfigurator.java:111) at com.fasterxml.jackson.jaxrs.json.JsonMapperConfigurator._resolveIntrospectors(JsonMapperConfigurator.java:84) at com.fasterxml.jackson.jaxrs.cfg.MapperConfiguratorBase._setAnnotations(MapperConfiguratorBase.java:120) at com.fasterxml.jackson.jaxrs.json.JsonMapperConfigurator.getDefaultMapper(JsonMapperConfigurator.java:45) at com.fasterxml.jackson.jaxrs.base.ProviderBase.locateMapper(ProviderBase.java:910) at org.jboss.resteasy.plugins.providers.jackson.ResteasyJackson2Provider.readFrom(ResteasyJackson2Provider.java:116) at org.jboss.resteasy.core.interception.AbstractReaderInterceptorContext.readFrom(AbstractReaderInterceptorContext.java:66) at org.jboss.resteasy.core.interception.ServerReaderInterceptorContext.readFrom(ServerReaderInterceptorContext.java:61) at org.jboss.resteasy.core.interception.AbstractReaderInterceptorContext.proceed(AbstractReaderInterceptorContext.java:56) at org.jboss.resteasy.core.MessageBodyParameterInjector.inject(MessageBodyParameterInjector.java:151) at org.jboss.resteasy.core.MethodInjectorImpl.injectArguments(MethodInjectorImpl.java:92) at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:115) at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295) at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249) at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:236) at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:406) at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:213) at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:228) at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) at javax.servlet.http.HttpServlet.service(HttpServlet.java:779) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280) at java.base/java.security.AccessController.doPrivileged(AccessController.java:712) at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:584) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:170) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:221) at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:145) at java.base/java.security.AccessController.doPrivileged(AccessController.java:569) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:143) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280) at java.base/java.security.AccessController.doPrivileged(AccessController.java:712) at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:584) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:187) at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:145) at java.base/java.security.AccessController.doPrivileged(AccessController.java:569) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:143) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:177) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) at com.netscape.cms.tomcat.ExternalAuthenticationValve.invoke(ExternalAuthenticationValve.java:83) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:891) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1784) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Thread.java:833) this is obviously in the same area as https://bugzilla.redhat.com/show_bug.cgi?id=2158907 , but I figured we could do with a new bug. We have untagged the builds from the update from Rawhide until this is resolved somehow.
edit: forgot to mention, the system journal also has this warning again: Feb 6 10:14:14 ipa002 server[3475]: WARNING: Problem with JAR file [/usr/share/pki/server/common/lib/jaxb-api.jar], exists: [false], canRead: [false] as explained in 2158907 , that file is part of dogtag-pki-server and is a symlink to /usr/share/java/jaxb-api.jar . Possibly just adding a direct dep on jaxb-api2 to dogtag-pki-server would be enough to resolve this for now; I'll test that.
hmm, no. Adding that explicit dep does ensure jaxb-api2 gets installed and makes the /usr/share/pki/server/common/lib/jaxb-api.jar warning go away, but FreeIPA server deployment still fails with the same traceback.
Note that the PR https://src.fedoraproject.org/rpms/dogtag-pki/pull-request/3 was merged but the package was not rebuilt since then.
dogtag-pki should symlink the real dependencies of resteasy, which were updated. Proposed fix: https://src.fedoraproject.org/rpms/dogtag-pki/pull-request/4
I have rebuilt dogtag with Marián's patch (thanks!) in a new side-tag (f38-build-side-62834) with the builds from the previous update. I'm not sure what to do next though. I tried to submit a new update with the original 3 builds and the new Dogtag build but Bodhi didn't like that, telling me the builds already exist. If I submit just the new Dogtag build it will not build. What's the play?
This bug appears to have been reported against 'rawhide' during the Fedora Linux 38 development cycle. Changing version to 38.
Yeah, a package can only be in one update, so since an update with resteasy+friends already exists, you can't create a new update also containing those builds. What we need to do is edit the other builds into that update. Another option is to do bump+rebuilds of those three packages for the new update, but in this case it's probably easier for someone with provenpackager privs (hi) to just edit the existing one. I'll take care of it.
agh, actually, since that update is stable already, we can't do that. instead, we can just create a new update for dogtag-pki and retag the packages from that update at the same time (they were manually untagged because of this bug). I'll co-ordinate that with releng.
OK, in that case, Endi is about to make another update for PKI before the branch. Can we wait until he has finished that before making the new update and tagging back in please? Or would it be fine for him to do his update on top of your provenpackager bit?
No problem, I can wait for that.
actually, to get it out of the way, we went ahead and dealt with it. all the builds from the resteasy update plus dogtag-pki-11.2.0-3.fc38.1 are now stable and tested to work OK (note you should probably drop that .1 from the NVR for the next build, it was added by the mass rebuild bot). endi can submit a new build whenever, it won't cause any problems.
Great, thanks for pushing that through! I hadn't even noticed that .1 - didn't know that was even a thing. I'll ask Endi to drop it, thanks for spotting that.
yeah, IIRC the mass rebuild basically runs `rpmdev-bumpspec` on the spec file, which *tries* to just increment the release counter by 1, but when a spec implements the release field in indirect ways (like dogtag-pki does) and it can't figure out how to do that, it just sticks a .1 on the end as a fallback. If you don't notice that and take it off again, it'll stick there till the next mass rebuild when the bot will probably bump it to .2 or something :P I think there may actually be something you can do to clue rpmdev-bumpspec in to what number it needs to bump (%release_number in this case) but I don't know offhand exactly how you do it...
It looks like you can manipulate the end of the Release tag in various ways but not update an arbitrary field and then pass it in to Release. Which is fine, it ain't broke after all.