jaxb-api-4.0.0-1.fc38 has broken dogtag-pki (and thus FreeIPA) in Rawhide. The details are a bit complex and the best way forward is not clear. I'll try to outline things as best I can. The relevant dependency chain is: dogtag-pki-server -> dogtag-pki-tools -> dogtag-pki-java -> pki-resteasy-core -> "mvn(javax.xml.bind:jaxb-api)" Until recently, "mvn(javax.xml.bind:jaxb-api)" was provided only by jaxb-api. Then jaxb-api2, which appears to be intended as a compatibility package for things that do not work with the new version of jaxb-api, appeared in mid-December, which also provides it. dogtag-pki-server, however, actually has a much more direct dependency on jaxb-api, though it isn't specified in the spec file (I think it probably should be). dogtag-pki-server includes a file `/usr/share/pki/server/common/lib/jaxb-api.jar`, which is actually just a symlink pointing to `/usr/share/java/jaxb-api.jar`. That file is, or was, a part of jaxb-api: the old jaxb-api-2.3.3-7.fc37 package included it. That file appears to be associated with the "mvn(javax.xml.bind:jaxb-api)" provide, because the old jaxb-api-2.3.3-7.fc37 both contained `/usr/share/java/jaxb-api.jar` and provided "mvn(javax.xml.bind:jaxb-api)", while the new jaxb-api-4.0.0-1.fc38 has neither the file nor the provide. With the new jaxb-api-4.0.0-1.fc38 build in Rawhide, what happens when trying to install FreeIPA is this. dogtag-pki-server gets installed, triggering the dep chain. Since the jaxb-api package no longer provides "mvn(javax.xml.bind:jaxb-api)", the jaxb-api2 package is instead installed to satisfy that requirement (previously, jaxb-api would have won the "race" to satisfy the requirement because its name is shorter. Yes, really.) However, jaxb-api2 does not contain a file `/usr/share/java/jaxb-api.jar`. Instead it has `/usr/share/java/jaxb-api-2.3.3.jar`. So the symlink in dogtag-pki-server that points to `/usr/share/java/jaxb-api.jar` is left dangling, and this causes dogtag not to work - we get a fatal error in /var/log/pki/pki-tomcat/ca/debug.2023-01-06.log , and the system log shows an error: Jan 6 14:50:26 ipa002 server[3187]: WARNING: Problem with JAR file [/usr/share/pki/server/common/lib/jaxb-api.jar], exists: [false], canRead: [false] I'm not sure what the best resolution for this problem is. It's easy enough to put the `/usr/share/java/jaxb-api.jar` symlink back in jaxb-api (pointing to the real file), but it's a bit trickier to add the provide back (presumably you need to use the maven macros to do so, but I'm not sure exactly how). Even if you sort that out, is it the *right* thing to do? Can we expect things that worked with the 2.3.3 library to work with the 4.0.0 library - is it sensible to advertise it as "the same thing"? Possibly we should instead have dogtag-pki-server specifically require the jaxb-api2 package somehow, and install its symlink pointing to `/usr/share/java/jaxb-api-2.3.3.jar` instead. Or perhaps the solution is something else I hadn't thought of. I figured it was better to file a bug and leave it to the experts. It's also not clear whether pki-resteasy-core is "OK". As things stand, its dependency on "mvn(javax.xml.bind:jaxb-api)" is now provided by jaxb-api2. Is that what we want? Will it be able the find the now-versioned library file from that package? For now I have asked releng to untag the jaxb-api2 build: https://pagure.io/releng/issue/11202 . I think that should prevent the bug appearing in Rawhide until we can figure out what to do about it.
Looking at the logs to file this bug report, I also noted a similar issue with the update to jakarta-activation. Again, a new version has been built which drops a filename and a provide compared to the previous version, this time the file is `/usr/share/java/jakarta-activation/jakarta.activation.jar` and the provide is "mvn(com.sun.activation:jakarta.activation)" - both present in 1.2.2-7 and absent in 2.1.1-1 . Again, a compatibility package has been added - jakarta-activation1 - which has the provide, but a slightly different filename: it has `/usr/share/java/jakarta-activation1/jakarta.activation.jar` (note `jakarta-activation1/` not `jakarta-activation/`). Again, dogtag-pki-server has a symlink to the old name: /usr/share/pki/server/common/lib/javax.activation.jar -> /usr/share/java/jakarta-activation/jakarta.activation.jar. And again, this now triggers an error message: WARNING: Problem with JAR file [/usr/share/pki/server/common/lib/javax.activation.jar], exists: [false], canRead: [false] I *think* that case happens not to actually break FreeIPA, though, or at least the bits of it that openQA exercises, as looking at timestamps, we seem to have had a few successful runs of the openQA tests after jakarta-activation-2.1.1-1.fc38 was tagged. The failures seem to have started right after jaxb-api-4.0.0-1.fc38 was tagged.
The error that FreeIPA actually fails with (included for info and searching purposes) is this: 2023-01-06 14:50:40 [https-jsse-nio-8443-exec-4] SEVERE: Servlet.service() for servlet [Resteasy] in context with path [/ca] threw exception [javax/xml/bind/annotation/XmlElement] with root cause java.lang.ClassNotFoundException: javax.xml.bind.annotation.XmlElement and of course, "javax.xml.bind" seems to be exactly the thing we're dealing with here.
FYI now jackson-jaxrs-providers FTBFS because it can't find jakarta.activation:jakarta.activation-api:jar:1.2.0 https://koschei.fedoraproject.org/package/jackson-jaxrs-providers?collection=f38.
I believe this can be solved by a downstream patch to package `dogtag-pki` in file `pki-11.2.0/base/CMakeLists.txt` at lines 156 and 158. Please try it along with installed jaxb-api2 package (and removed jaxb-api package) and let me know if the issue does not happen anymore. If so, we can discuss what file paths would be used by version 2 and 4.
I see a potential solution which would keep the files unchanged. See the upstream jar filenames: https://repo1.maven.org/maven2/javax/xml/bind/jaxb-api/2.3.1/ https://repo1.maven.org/maven2/jakarta/xml/bind/jakarta.xml.bind-api/2.3.3/ https://repo1.maven.org/maven2/jakarta/xml/bind/jakarta.xml.bind-api/4.0.0/ (Note that version 2.3.1 is API-compatible with 2.3.2 and 2.3.3, just the project owners have changed, the project still uses namespace "javax", later versions use "jakarta") It would be possible to associate "/usr/share/java/jaxb-api.jar" with version 2 and "/usr/share/java/jakarta.xml.bind-api.jar" with version 4. There is still the issue of versions 2.3.2 and 2.3.3 with jar filenames starting with "jakarta" but them representing major version 2.
There is my proposed solution: Compatibility package with version 2, functional replacement for the package jaxb-api version 2, contains the file "/usr/share/java/jaxb-api.jar". https://copr.fedorainfracloud.org/coprs/mkoncek/jaxb-api2/build/5216257/ Updated version of jaxb-api to version 4. This is not needed by you. https://copr.fedorainfracloud.org/coprs/mkoncek/jaxb-api/build/5216271/
Sorry, I haven't been able to follow up in detail as I'm on PTO, but the solution from comment #5 sounds reasonable. I can try and test it but can't guarantee to do it before I get back from PTO next week. If you're reasonably confident you could throw builds at Rawhide, and if they break stuff we'll find out with the next compose... The changes between jakarta-activation1-1.2.2-4.fc38 and jakarta-activation1-1.2.2-5.fc38 broke Rawhide today: https://openqa.fedoraproject.org/tests/1685426#step/role_deploy_domain_controller/25 dogtag-pki-java cannot be installed because nothing provides "mvn(javax.activation:activation)", which it requires.
oh, it looks like jakarta-activation1-1.2.2-6.fc38 is intended to fix the breakage from jakarta-activation1-1.2.2-5.fc38 , I guess - I'll see if tomorrow's Rawhide is OK.
jakarta-activation1-1.2.2-5.fc38 reverts the provision of mvn(javax.activation:activation) and resteasy package was rebuilt after this change. jakarta-activation1-1.2.2-6.fc38 only additionally installs the symlink in /usr/share/java/javax.activation.jar.
I just built the new jaxb-api2 and re-tagged jaxb-api (v4) in Rawhide. Let's hope nothing breaks.
I believe `dogtag-pki` package will need a rebuild, not because of changes in jaxb-api, but because of changes in jakarta-activation. File /usr/share/java/jakarta-activation/jakarta.activation.jar previously provided by jakarta-activation (v1) is no longer provided, but file /usr/share/java/javax.activation.jar is now provided by jakarta-activation1. The build of dogtag uses cmake `find_files` function but it should now find a different jar for jakarta-activation.
The re-tagging doesn't seem to have taken effect. Today's compose had jaxb-api-2.3.3-8.fc37.noarch.rpm and jaxb-api2-2.3.3-3.fc38.noarch.rpm. The tests passed, but I expect they used jaxb-api-2.3.3-8 so we still don't know if they work with the new jaxb-api2. It looks like a jaxb-api-4.0.0-2.fc38 build ran today and was tagged as usual via Bodhi, so that should show up in tomorrow's compose. I could manually trigger the openQA update tests on it, that's a slightly different scenario as the older jaxb-api build will still be in the main repo for dnf to use if it likes, but a later check in the openQA tests *should* catch if it does that.
OK, today's compose got jaxb-api-4.0.0-2.fc38 and jaxb-api2-2.3.3-3.fc38 and the FreeIPA tests passed OK. As mentioned somewhere above, for FreeIPA purposes, we did see some errors/warnings related to the activation stuff, but they don't seem to stop the FreeIPA stuff we test in openQA from working OK.
As far as I'm concerned this can be closed now, but I'll leave it open in case Marián wants it for anything. If not, just go ahead and close it as RAWHIDE. Thanks!
I believe the best way forward is to modify all the dependent packages that break and not keep compatibility stuff in neither of jaxb-api packages. Therefore I intend to remove the provision of .jar symlinks. You should use tools intended to find jars instead of finding them manually in CMake. These are `xmvn-resolve`, `find-jar`, `build-classpath`.
It would be best to co-ordinate that work with the packages that use the symlinks, rather than just dropping them and breaking the dependent packages and expecting them to adapt. We *will* file more blocker bugs and ask for untags if future updates break FreeIPA, for instance, as it's a release-blocking component. At present in Rawhide the deps are as follows: [adamw@xps13a nightlies]$ sudo dnf repoquery --whatrequires jaxb-api jakarta-xml-ws-0:2.3.3-1.fc37.noarch jaxb-core-0:4.0.1-2.fc38.noarch [adamw@xps13a nightlies]$ sudo dnf repoquery --whatrequires jaxb-api2 jackson-module-jaxb-annotations-0:2.14.1-2.fc38.noarch jboss-jaxrs-2.0-api-0:1.0.0-20.fc37.noarch openstack-java-nova-model-0:3.2.9-13.fc37.noarch pki-resteasy-core-0:3.0.26-19.fc38.noarch preflight-0:2.0.27-4.fc38.noarch I suspect all the things that depend on jaxb-api2 probably depend on the symlinks, if so they should all be adapted before the symlink is removed...
I am mostly involved in packaging maven dependencies and other packages which all use maven for their build so I may not see the full picture. In maven world you do not care about the jar location, only about the artifact coordinates and the mapping is stored in metadata files installed along with each package. Stable jar location is necessary when you have for example a java executable that builds its classpath when being invoked. Or, like in the case of dogtag-pki, when using non-traditional build systems (CMake). I inspected these the packages and they are all java libraries, so there is nothing to modify in them other than ensure that they use correct maven coordinates. But I agree that the compat packages should provide some stable jar location that wouldn't change with a package update.
See: https://src.fedoraproject.org/rpms/dogtag-pki/pull-request/3 The two updates to jaxb-api2 and jakarta-activation1 will cause installation of files: /usr/share/java/jaxb-api2/jakarta.xml.bind-api-2.jar /usr/share/java/jakarta-activation1/jakarta.activation-api-1.jar These files are resolved by the command `xmvn-resolve` and their file path will not change with possible further updates. I intend to remove the provision of artifact `javax.xml.bind:jaxb-api` but I will first open PRs for all the dependent packages.
You seem to have done the new jaxb-api2 build already, before giving anyone any reasonable amount of time to review and merge your PRs. So FreeIPA is broken in Rawhide again: https://openqa.fedoraproject.org/tests/1703193# I will request an untag, again. If the changes to jaxb-api2 and the dependent packages need to land together (i.e. we can't change dogtag-pki etc. *first* then change jaxb-api2 once dogtag-pki is done), then builds of all relevant packages should be done on a side tag and a single combined update containing all the packages created from that side tag. See https://docs.fedoraproject.org/en-US/package-maintainers/Package_Update_Guide/#_side_tags for info on the side tag workflow. If you need more permissions to get builds run or create the update, I can help.
https://pagure.io/releng/issue/11229 - untag request.
Alternatively, I guess, you could do a bridge build of jaxb-api2 which has both the old compat symlink and the new files, then consumers can adjust to use the new files, then once they're all adjusted, you can do another build of jaxb-api2 to drop the compat symlink.
Sorry for causing you problems. The update was indeed intended as a bridge update but there was a bug. The symlink exists but points to the wrong path.
New release has been built. I believe the previously cause issue if fixed. These PRs can now be merged: https://src.fedoraproject.org/rpms/xstream/pull-request/6 https://src.fedoraproject.org/rpms/jboss-jaxrs-2.0-api/pull-request/2 https://src.fedoraproject.org/rpms/jackson-modules-base/pull-request/10 https://src.fedoraproject.org/rpms/resteasy/pull-request/4 https://src.fedoraproject.org/rpms/dogtag-pki/pull-request/3
Thanks a lot for that, I'll check if there are any remaining problems with FreeIPA.
Looks like we have a sequel to this: https://bugzilla.redhat.com/show_bug.cgi?id=2167539 . Today's resteasy+friends update broke FreeIPA deployment again.
This bug appears to have been reported against 'rawhide' during the Fedora Linux 38 development cycle. Changing version to 38.
The original issue was resolved, this bug can be closed.