Bug 2066019
| Summary: | Latest ose-jenkins-agent-base:v4.9.0 image fails to start on OpenShift due to FIPS error | ||
|---|---|---|---|
| Product: | OpenShift Developer Tools and Services | Reporter: | jamieclinton |
| Component: | Jenkins | Assignee: | Akram Ben Aissi <abenaiss> |
| Status: | CLOSED EOL | QA Contact: | Jitendar Singh <jitsingh> |
| Severity: | high | Docs Contact: | Rolfe Dlugy-Hegwer <rdlugyhe> |
| Priority: | high | ||
| Version: | 4.8 | CC: | aos-bugs, cdaley, dkarde, gmontero, jdelft, jitsingh, spandura |
| Target Milestone: | --- | ||
| Target Release: | 4.11 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Enhancement | |
| Doc Text: |
* This enhancement adds a new Jenkins environment variable, `JAVA_FIPS_OPTIONS`, that controls how the JVM operates when running on a FIPS node. For more information, see link:https://access.redhat.com/documentation/en-us/openjdk/11/html-single/configuring_openjdk_11_on_rhel_with_fips/index#config-fips-in-openjdk[OpenJDK support article] (link:https://bugzilla.redhat.com/show_bug.cgi?id=2066019[BZ#2066019])
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-02-07 15:34:40 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 2075135 | ||
|
Description
jamieclinton
2022-03-20 02:13:10 UTC
Why is it that nobody has responded to this bug yet? It appears as though the latest image of ose-jenkins-agent-base:v4.8.0 does not run in a FIPS-enabled OpenShift cluster. This is a big problem for us. Please try to recreate this bug on your end. Thanks, Jamie Correction, v4.9.0. So I have confirmed by bringing up "more recent versions of OCP" (I've tried 4.9 and the 4.11 currently under development) that with FIPS on for a cluster, both to start up Jenkins in general, as well as to launch agent based pods, you need to employ some form of https://access.redhat.com/documentation/en-us/openjdk/11/html-single/configuring_openjdk_11_on_rhel_with_fips/index#config-fips-in-openjdk in order to get things to come up. Employing the simplest of the options, setting the system property -Dcom.redhat.fips=false on both the Jenkins pod/container's JVM args, as well as the same for the PodTemplate I used for the agent based build, I could get things to work. Presumably someone could also set up the java trust store with a valid FIPS cert config, as is indicated in the openJDK article I noted above, and get things to work as well, and avoid the exception the customer is seeing. I have had trouble getting 4.8 clusters up today, so as of yet, have not been able to clearly discover the reason why "it works" on 4.8, but I would be shocked if it is not a JDK version difference of some sort. I will report back when I have been able to test on 4.8. The other possible difference I can think of, prior to actual attempts on 4.8, is differences in the PodTemplates used with the 4.8 image vs. the 4.9 image. But we can dive into that if my 4.8 testing results drive us in that direction. OK, using the most recent 4.8 level, 4.8.35, I still see the same issues I saw on 4.9 and 4.11, where I need to set the JVM arg -Dcom.redhat.fips=false on both the jenkins pod as well as the any agent pods launched from PodTemplates in order to avoid the FIPS related exception reported in this BZ's description. I also found the customer logs attached to the support case (probably since support did not open the BZ, they were not attached here as well ?), and I did see a JVM version difference between the customer's jenkins and what was present in my 4.8.35 based jenkins pod. 4.8.35 had /usr/lib/jvm/java-11-openjdk-11.0.14.1.1-1.el8_4.x86_64/bin/java and /usr/lib/jvm/java-11-openjdk-11.0.14.1.1-1.el8_4.x86_64/bin/javac The customer's had /usr/lib/jvm/java-11-openjdk-11.0.13.0.8-1.el8_4.x86_64/bin/java and /usr/lib/jvm/java-11-openjdk-11.0.13.0.8-1.el8_4.x86_64/bin/javac I have not sense at this time how disparate those openjdk version are, but it is "something". I see in the description the customer is at 4.8.22. I will now deploy that, and see what happens. And sure enough, at 4.8.22, my jenkins and agent pods were at /usr/lib/jvm/java-11-openjdk-11.0.13.0.8-1.el8_4.x86_64/bin/java and /usr/lib/jvm/java-11-openjdk-11.0.13.0.8-1.el8_4.x86_64/bin/javac and I did not have to set -Dcom.redhat.fips=false in order for either of them to work properly on my FIPS cluster. OK, so here is how we are moving forward: - Dipak Karde, I have added a needinfo for you. In your role as support, I want you to open an item associated with the customer case here on the openjdk team's queue, and see if they can explain why JVMs at 11.0.13.0.8-1 can run on FIPS clusters with disabling FIPS or setting up a JVM friendly FIPS trust store in some way per https://access.redhat.com/documentation/en-us/openjdk/11/html-single/configuring_openjdk_11_on_rhel_with_fips/index#config-fips-in-openjdk but we need such settings at 11.0.14.1.1-1. It would be good to know what exactly is going on here, is it expected, is there a bug in 11.0.14.1.1-1, whatever. - In the interim, I will be updating our images, templates, pod templates to default to -Dcom.redhat.fips=false but allow for override via environment variables. - I may change the default to something else if we get an answer from openjdk that steers us in such a direction. - Jamie Clinton: if you want to set -Dcom.redhat.fips=false on your own in the interim in your PodTemplate, it is fairly straight forward to do that from the Jenkins UI (add it to the front of the args before the computer.jnlp related stuff for example). But if you need pointers let us know. Also, if you upgrade your jenkins pod to one of these newer levels, you'll need to add the JENKINS_JAVA_OPTIONS env var on your jenkins container and set it to -Dcom.redhat.fips=false Hi Gabe,
I was able to get the new ose-jenkins-agent-base:v4.9.0 image to run successfully by adding the JAVA_TOOL_OPTIONS env var to the PodTemplate yaml file and setting the value to -Dcom.redhat.fips=false.
<org.csanchez.jenkins.plugins.kubernetes.model.KeyValueEnvVar>
<key>JAVA_TOOL_OPTIONS</key>
<value>-Dcom.redhat.fips=false</value>
</org.csanchez.jenkins.plugins.kubernetes.model.KeyValueEnvVar>
Thanks for the workaround!
Jamie
*** Bug 2065519 has been marked as a duplicate of this bug. *** verified ============= provisioned 4.11 cluster with private-templates/functionality-testing/aos-4_11/ipi-on-aws/versioned-installer-fips flexy template created jenkins using template created and triggered builds using sidecar agent images started 20 build and all builds completed succesfully checked jenkins master pod log and found "-Dcom.redhat.fips=false" in the below section which is desired behavior as per the PR + exec java -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Xmx512m -Dfile.encoding=UTF8 -Djavamelody.displayed-counters=log,error -Djava.util.logging.config.file=/var/lib/jenkins/logging.properties -Djavax.net.ssl.trustStore=/var/lib/jenkins/ca-anchors-keystore "-Dcom.redhat.fips=false" -Djdk.http.auth.tunneling.disabledSchemes= -Djdk.http.auth.proxying.disabledSchemes= -Duser.home=/var/lib/jenkins -Djavamelody.application-name=jenkins -Dhudson.security.csrf.GlobalCrumbIssuerConfiguration.DISABLE_CSRF_PROTECTION=true -Djenkins.install.runSetupWizard=false -jar /usr/lib/jenkins/jenkins.war Picked up JAVA_TOOL_OPTIONS: -XX:+UnlockExperimentalVMOptions -Dsun.zip.disableMemoryMapping=true the openjdk team explained why the behavior changed in https://issues.redhat.com/browse/OPENJDK-629?focusedCommentId=20095046&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-20095046 ... basically they had to address a bug with running openjdk on RHEL 8.6 and later with FIPS enabled. |