Bug 1965721

Summary: FreeIPA replica deployment fails since Fedora-Rawhide-20210527.n.0 (pki-core-10.11.0-0.2.alpha1.fc35...)
Product: [Fedora] Fedora Reporter: Adam Williamson <awilliam>
Component: pki-coreAssignee: Endi Sukma Dewata <edewata>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: rawhideCC: abokovoy, alee, alexander.m.scheel, cfu, ckelley, edewata, jmagne, jpazdziora, kwright, mharmsen, mkdineshprasanth
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard: openqa
Fixed In Version: pki-core-10.11.0-0.4.alpha2.fc35 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-06-22 22:05:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
log tarball from the replica none

Description Adam Williamson 2021-05-28 23:17:27 UTC 
Since Fedora-Rawhide-20210527.n.0, the FreeIPA replica tests are failing. Deployment of the replica fails with this error in the log:

INFO: Connecting to https://ipa002.test.openqa.fedoraproject.org:443
java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory
        at org.apache.http.impl.client.CloseableHttpClient.<init>(CloseableHttpClient.java:59)
        at org.apache.http.impl.client.AbstractHttpClient.<init>(AbstractHttpClient.java:232)
        at org.apache.http.impl.client.DefaultHttpClient.<init>(DefaultHttpClient.java:148)
        at com.netscape.certsrv.client.PKIConnection.<init>(PKIConnection.java:82)
        at com.netscape.certsrv.client.PKIClient.<init>(PKIClient.java:76)
        at com.netscape.certsrv.client.PKIClient.<init>(PKIClient.java:65)
        at com.netscape.cmstools.cli.MainCLI.getClient(MainCLI.java:588)
        at org.dogtagpki.cli.CLI.getClient(CLI.java:207)
        at com.netscape.cmstools.ca.CACLI.getSubsystemClient(CACLI.java:65)
        at com.netscape.cmstools.range.RangeRequestCLI.execute(RangeRequestCLI.java:77)
        at org.dogtagpki.cli.CommandCLI.execute(CommandCLI.java:57)
        at org.dogtagpki.cli.CLI.execute(CLI.java:357)
        at org.dogtagpki.cli.CLI.execute(CLI.java:357)
        at com.netscape.cmstools.cli.SubsystemCLI.execute(SubsystemCLI.java:76)
        at org.dogtagpki.cli.CLI.execute(CLI.java:357)
        at com.netscape.cmstools.cli.MainCLI.execute(MainCLI.java:660)
        at com.netscape.cmstools.cli.MainCLI.main(MainCLI.java:696)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        ... 17 more
CalledProcessError: Command '['pki', '-d', '/etc/pki/pki-tomcat/alias', '-f', '/etc/pki/pki-tomcat/password.conf', '-U', 'https://ipa002.test.openqa.fedoraproject.org:443', 'ca-range-request', 'request', '--install-token', '/tmp/tmpw21z8owd/install-token', '--output-format', 'json', '--debug']' returned non-zero exit status 255.

In that compose, several relevant packages changed. The changelogs look minor, but maybe there was an unlogged change. pki-core, tomcatjss, jss and idm-console-framework all were updated.

Attaching the full log tarball from the replica.
Comment 1 Adam Williamson 2021-05-28 23:18:14 UTC 
Created attachment 1787967 [details]
log tarball from the replica
Comment 2 Alexander Bokovoy 2021-05-31 18:31:14 UTC 
Note I am reproducing it trivially in FreeIPA Azure CI with Rawhide, for example on my personal instance: https://dev.azure.com/abbra1freeipa/freeipa/_build/results?buildId=547&view=results. The failures are definitive. Is this because org.apache.commons.logging.LogFactory cannot be found even though it is installed into /usr/share/java.
Comment 3 Endi Sukma Dewata 2021-06-10 15:40:48 UTC 
Please try again with pki-core-10.11.0-0.4.alpha2.fc35. Thanks.
Comment 4 Jan Pazdziora 2021-06-19 13:53:56 UTC 
Current failure on Fedora rawhide is

  [3/29]: creating ACIs for admin
  [4/29]: creating installation admin user
  [5/29]: configuring certificate server instance
Failed to configure CA instance
See the installation logs and the following files/directories for more information:
  /var/log/pki/pki-tomcat
  [error] RuntimeError: CA configuration failed.
CA configuration failed.
The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

with /var/log/ipareplica-install.log saying

DEBUG: Command: pki -d /etc/pki/pki-tomcat/alias -f /etc/pki/pki-tomcat/password.conf -U https://hp-z210-01.ml3.eng.bos.redhat.com:443 ca-range-request request --install-token /tmp/tmp63slqgcy/install-token --output-format json --debug
INFO: Connecting to https://hp-z210-01.ml3.eng.bos.redhat.com:443
WARN: RESTEASY002145: NoClassDefFoundError: Unable to load builtin provider org.jboss.resteasy.plugins.providers.IIOImageProvider from jar:file:/usr/share/java/resteasy/resteasy-jaxrs.jar!/META-INF/services/javax.ws.rs.ext.Providers
java.lang.NoClassDefFoundError: javax/annotation/Priority
        at org.jboss.resteasy.spi.ResteasyProviderFactory.getPriority(ResteasyProviderFactory.java:1425)
        at org.jboss.resteasy.spi.ResteasyProviderFactory.registerProvider(ResteasyProviderFactory.java:1463)
        at org.jboss.resteasy.spi.ResteasyProviderFactory.registerProvider(ResteasyProviderFactory.java:1398)
        at org.jboss.resteasy.plugins.providers.RegisterBuiltin.registerProviders(RegisterBuiltin.java:133)
        at org.jboss.resteasy.plugins.providers.RegisterBuiltin.register(RegisterBuiltin.java:45)
        at org.jboss.resteasy.client.jaxrs.ResteasyClientBuilder.getProviderFactory(ResteasyClientBuilder.java:334)
        at org.jboss.resteasy.client.jaxrs.ResteasyClientBuilder.build(ResteasyClientBuilder.java:367)
        at com.netscape.certsrv.client.PKIConnection.<init>(PKIConnection.java:198)
        at com.netscape.certsrv.client.PKIClient.<init>(PKIClient.java:76)
        at com.netscape.certsrv.client.PKIClient.<init>(PKIClient.java:65)
        at com.netscape.cmstools.cli.MainCLI.getClient(MainCLI.java:592)
        at org.dogtagpki.cli.CLI.getClient(CLI.java:207)
        at com.netscape.cmstools.ca.CACLI.getSubsystemClient(CACLI.java:66)
        at com.netscape.cmstools.range.RangeRequestCLI.execute(RangeRequestCLI.java:80)
        at org.dogtagpki.cli.CommandCLI.execute(CommandCLI.java:58)
        at org.dogtagpki.cli.CLI.execute(CLI.java:357)
        at org.dogtagpki.cli.CLI.execute(CLI.java:357)
        at com.netscape.cmstools.cli.SubsystemCLI.execute(SubsystemCLI.java:79)
        at org.dogtagpki.cli.CLI.execute(CLI.java:357)
        at com.netscape.cmstools.cli.MainCLI.execute(MainCLI.java:665)
        at com.netscape.cmstools.cli.MainCLI.main(MainCLI.java:701)
Caused by: java.lang.ClassNotFoundException: javax.annotation.Priority
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        ... 21 more

Is that something that should be tracked as part of this bugzilla, or separate bugzilla filed?
Comment 6 Adam Williamson 2021-06-20 06:49:42 UTC 
Jan: I was waiting till the Python 3.10 stuff was sorted out. I'll see where openQA is on Monday.
Comment 7 Adam Williamson 2021-06-22 22:05:48 UTC 
yeah, openQA is showing exactly the same as Jan posted. I think a new bug is clearest, technically the bug from this report is fixed now, I think.
Comment 8 Adam Williamson 2021-06-22 22:54:06 UTC 
Filed https://bugzilla.redhat.com/show_bug.cgi?id=1975010 .