Description of problem: I tried to install the latest ose-jenkins:v4.8.0 image from the Red Hat catalog in our OpenShift 4.8 cluster: https://catalog.redhat.com/software/containers/openshift4/ose-jenkins/5cdd918ad70cc57c44b2d279?tag=v4.8.0-202203050155.p0.g8c40628.assembly.stream&push_date=1647430166000&container-tabs=gti A previous build of the ose-jenkins:v4.8.0 image (from several weeks ago) works just fine, but with this latest image, the pod comes up with "Jenkins is fully up and running," but after trying to display the Jenkins console, the browser displays "Oops! A problem occurred while processing the request." I've tried this repeatedly with the same result, including with the v4.9.0 and v4.10.0 images. The pod logs show following: 2022/03/18 00:56:21 [go-init] No pre-start command defined, skip 2022/03/18 00:56:21 [go-init] Main command launched : /usr/libexec/s2i/run Using JENKINS_SERVICE_NAME=jenkins Generating jenkins.model.JenkinsLocationConfiguration.xml using (/var/lib/jenkins/jenkins.model.JenkinsLocationConfiguration.xml.tpl) ... Jenkins URL set to: https://jenkins-jenkins-test.apps.ocpcarrier1.test.fdc.ibm in file: /var/lib/jenkins/jenkins.model.JenkinsLocationConfiguration.xml /usr/libexec/s2i/run: line 24: /jenkins.model.JenkinsLocationConfiguration.xml.tpl: No such file or directory CONTAINER_MEMORY_IN_MB='1024', using /usr/lib/jvm/java-11-openjdk-11.0.14.0.9-2.el8_4.x86_64/bin/java and /usr/lib/jvm/java-11-openjdk-11.0.14.0.9-2.el8_4.x86_64/bin/javac Administrative monitors that contact the update center will remain active Picked up JAVA_TOOL_OPTIONS: -XX:+UnlockExperimentalVMOptions -Dsun.zip.disableMemoryMapping=true -Dorg.jenkinsci.plugins.pipeline.modeldefinition.parser.RuntimeASTTransformer.SCRIPT_SPLITTING_TRANSFORMATION=true -Dorg.jenkinsci.plugins.getclient.GitClient.untrustedSSL=true Migrating slave image configuration to current version tag ... + exec java -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Xmx512m -Dfile.encoding=UTF8 -Djavamelody.displayed-counters=log,error -Djava.util.logging.config.file=/var/lib/jenkins/logging.properties -Djavax.net.ssl.trustStore=/var/lib/jenkins/ca-anchors-keystore -Djdk.http.auth.tunneling.disabledSchemes= -Djdk.http.auth.proxying.disabledSchemes= -Duser.home=/var/lib/jenkins -Djavamelody.application-name=JENKINS -Dhudson.security.csrf.GlobalCrumbIssuerConfiguration.DISABLE_CSRF_PROTECTION=true -Djenkins.install.runSetupWizard=false -jar /usr/lib/jenkins/jenkins.war Picked up JAVA_TOOL_OPTIONS: -XX:+UnlockExperimentalVMOptions -Dsun.zip.disableMemoryMapping=true -Dorg.jenkinsci.plugins.pipeline.modeldefinition.parser.RuntimeASTTransformer.SCRIPT_SPLITTING_TRANSFORMATION=true -Dorg.jenkinsci.plugins.getclient.GitClient.untrustedSSL=true Running from: /usr/lib/jenkins/jenkins.war webroot: EnvVars.masterEnvVars.get("JENKINS_HOME") 2022-03-18 00:56:24 INFO org.eclipse.jetty.util.log.Log initialized Logging initialized @525ms to org.eclipse.jetty.util.log.JavaUtilLog 2022-03-18 00:56:24 INFO winstone.Logger logInternal Beginning extraction from war file 2022-03-18 00:56:24 WARNING org.eclipse.jetty.server.handler.ContextHandler setContextPath Empty contextPath 2022-03-18 00:56:24 INFO org.eclipse.jetty.server.Server doStart jetty-9.4.43.v20210629; built: 2021-06-30T11:07:22.254Z; git: 526006ecfa3af7f1a27ef3a288e2bef7ea9dd7e8; jvm 11.0.14+9-LTS 2022-03-18 00:56:25 INFO org.eclipse.jetty.webapp.StandardDescriptorProcessor visitServlet NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet 2022-03-18 00:56:25 INFO org.eclipse.jetty.server.session.DefaultSessionIdManager doStart DefaultSessionIdManager workerName=node0 2022-03-18 00:56:25 INFO org.eclipse.jetty.server.session.DefaultSessionIdManager doStart No SessionScavenger set, using defaults 2022-03-18 00:56:25 INFO org.eclipse.jetty.server.session.HouseKeeper startScavenging node0 Scavenging every 600000ms 2022-03-18 00:56:25 INFO hudson.WebAppMain contextInitialized Jenkins home directory: /var/lib/jenkins found at: EnvVars.masterEnvVars.get("JENKINS_HOME") 2022-03-18 00:56:25 INFO org.eclipse.jetty.server.handler.ContextHandler doStart Started w.@2c5d601e{Jenkins v2.319.2,/,file:///var/lib/jenkins/war/,AVAILABLE}{/var/lib/jenkins/war} 2022-03-18 00:56:25 INFO org.eclipse.jetty.server.AbstractConnector doStart Started ServerConnector@6156496{HTTP/1.1, (http/1.1)}{0.0.0.0:8080} 2022-03-18 00:56:25 INFO org.eclipse.jetty.server.Server doStart Started @1895ms 2022-03-18 00:56:25 INFO winstone.Logger logInternal Winstone Servlet Engine running: controlPort=disabled 2022-03-18 00:56:26 INFO jenkins.InitReactorRunner$1 onAttained Started initialization 2022-03-18 00:56:26 INFO jenkins.InitReactorRunner$1 onAttained Listed all plugins WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$2 (file:/var/lib/jenkins/war/WEB-INF/lib/guice-4.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain) WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$2 WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 2022-03-18 00:56:33 INFO jenkins.InitReactorRunner$1 onAttained Prepared all plugins 2022-03-18 00:56:33 INFO jenkins.InitReactorRunner$1 onAttained Started all plugins 2022-03-18 00:56:33 WARNING hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1 error Failed to instantiate Key[type=org.jenkinsci.main.modules.instance_identity.PageDecoratorImpl, annotation=[none]]; skipping this component com.google.inject.ProvisionException: Unable to provision, see the following errors: 1) Error injecting constructor, java.lang.NullPointerException at org.jenkinsci.main.modules.instance_identity.PageDecoratorImpl.<init>(PageDecoratorImpl.java:21) 1 error at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:52) at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145) at hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:441) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012) at hudson.ExtensionFinder$GuiceFinder._find(ExtensionFinder.java:401) at hudson.ExtensionFinder$GuiceFinder.find(ExtensionFinder.java:392) at hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:352) at hudson.ExtensionList.load(ExtensionList.java:382) at hudson.ExtensionList.ensureLoaded(ExtensionList.java:318) at hudson.ExtensionList.getComponents(ExtensionList.java:182) at hudson.DescriptorExtensionList.load(DescriptorExtensionList.java:212) at hudson.ExtensionList.ensureLoaded(ExtensionList.java:318) at hudson.ExtensionList.iterator(ExtensionList.java:170) at jenkins.branch.CustomOrganizationFolderDescriptor.doAddSpecificDescriptors(CustomOrganizationFolderDescriptor.java:157) at jenkins.branch.CustomOrganizationFolderDescriptor.addSpecificDescriptors(CustomOrganizationFolderDescriptor.java:136) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104) at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:180) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1151) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.NullPointerException at java.base/java.util.Base64$Encoder.encode(Base64.java:267) at org.jenkinsci.main.modules.instance_identity.pem.PEMHelper.writeEncoded(PEMHelper.java:186) at org.jenkinsci.main.modules.instance_identity.pem.PEMHelper.encodePEM(PEMHelper.java:113) at org.jenkinsci.main.modules.instance_identity.InstanceIdentity.write(InstanceIdentity.java:96) at org.jenkinsci.main.modules.instance_identity.InstanceIdentity.<init>(InstanceIdentity.java:66) at org.jenkinsci.main.modules.instance_identity.InstanceIdentity.<init>(InstanceIdentity.java:40) at org.jenkinsci.main.modules.instance_identity.PageDecoratorImpl.<init>(PageDecoratorImpl.java:22) at org.jenkinsci.main.modules.instance_identity.PageDecoratorImpl$$FastClassByGuice$$68618106.newInstance(<generated>) at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61) at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:105) at com.google.inject.internal.ConstructorInjector.access$000(ConstructorInjector.java:32) at com.google.inject.internal.ConstructorInjector$1.call(ConstructorInjector.java:89) at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:115) at hudson.ExtensionFinder$GuiceFinder$SezpozModule.onProvision(ExtensionFinder.java:563) at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:126) at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:68) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:87) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267) at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ... 31 more 2022-03-18 00:56:34 WARNING hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1 error Failed to instantiate Key[type=org.jenkinsci.main.modules.sshd.SSHD, annotation=[none]]; skipping this component com.google.inject.ProvisionException: Unable to provision, see the following errors: 1) Error injecting constructor, java.lang.NullPointerException at org.jenkinsci.main.modules.instance_identity.InstanceIdentity.<init>(InstanceIdentity.java:40) while locating org.jenkinsci.main.modules.instance_identity.InstanceIdentity for field at org.jenkinsci.main.modules.sshd.SSHD.identity(SSHD.java:43) 1 error at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:52) at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145) at hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:441) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:54) at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:132) at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114) at com.google.inject.internal.ConstructorInjector.access$000(ConstructorInjector.java:32) at com.google.inject.internal.ConstructorInjector$1.call(ConstructorInjector.java:89) at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:115) at hudson.ExtensionFinder$GuiceFinder$SezpozModule.onProvision(ExtensionFinder.java:563) at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:126) at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:68) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:87) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267) at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145) at hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:441) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012) at hudson.ExtensionFinder$GuiceFinder._find(ExtensionFinder.java:401) at hudson.ExtensionFinder$GuiceFinder.find(ExtensionFinder.java:392) at hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:352) at hudson.ExtensionList.load(ExtensionList.java:382) at hudson.ExtensionList.ensureLoaded(ExtensionList.java:318) at hudson.ExtensionList.getComponents(ExtensionList.java:182) at hudson.DescriptorExtensionList.load(DescriptorExtensionList.java:212) at hudson.ExtensionList.ensureLoaded(ExtensionList.java:318) at hudson.ExtensionList.iterator(ExtensionList.java:170) at jenkins.branch.CustomOrganizationFolderDescriptor.doAddSpecificDescriptors(CustomOrganizationFolderDescriptor.java:157) at jenkins.branch.CustomOrganizationFolderDescriptor.addSpecificDescriptors(CustomOrganizationFolderDescriptor.java:136) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104) at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:180) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1151) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.NullPointerException at java.base/java.util.Base64$Encoder.encode(Base64.java:267) at org.jenkinsci.main.modules.instance_identity.pem.PEMHelper.writeEncoded(PEMHelper.java:186) at org.jenkinsci.main.modules.instance_identity.pem.PEMHelper.encodePEM(PEMHelper.java:113) at org.jenkinsci.main.modules.instance_identity.InstanceIdentity.write(InstanceIdentity.java:96) at org.jenkinsci.main.modules.instance_identity.InstanceIdentity.<init>(InstanceIdentity.java:66) at org.jenkinsci.main.modules.instance_identity.InstanceIdentity.<init>(InstanceIdentity.java:40) at org.jenkinsci.main.modules.instance_identity.InstanceIdentity$$FastClassByGuice$$af33f6d9.newInstance(<generated>) at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61) at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:105) at com.google.inject.internal.ConstructorInjector.access$000(ConstructorInjector.java:32) at com.google.inject.internal.ConstructorInjector$1.call(ConstructorInjector.java:89) at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:115) at hudson.ExtensionFinder$GuiceFinder$SezpozModule.onProvision(ExtensionFinder.java:563) at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:126) at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:68) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:87) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267) at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:54) at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:132) at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114) at com.google.inject.internal.ConstructorInjector.access$000(ConstructorInjector.java:32) at com.google.inject.internal.ConstructorInjector$1.call(ConstructorInjector.java:89) at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:115) at hudson.ExtensionFinder$GuiceFinder$SezpozModule.onProvision(ExtensionFinder.java:563) at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:126) at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:68) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:87) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267) at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ... 48 more Version-Release number of selected component (if applicable): ose-jenkins:v4.8.0 Jenkins 2.319.2 OpenShift 4.8.22 How reproducible: Always Steps to Reproduce: oc new-app -f "$DIR/jenkins-persistent-template.yaml" \ -p JENKINS_SERVICE_NAME="$JENKINS_SERVICE_NAME" \ -p IMAGE_REGISTRY="$IMAGE_REGISTRY" \ -p NAMESPACE="${NAMESPACE}" \ -p REGISTRY_SECRET="$REGISTRY_SECRET" Actual results: Jenkins console does not come up after deployment. Expected results: Jenkins console should work Additional info:
Can you try using the ocp-tools-4 Jenkins image? We are test-driving a new version of Jenkins that is meant to work with all supported OpenShift versions. https://catalog.redhat.com/software/containers/ocp-tools-4/jenkins-rhel8/5fe1f38288e9c2f788526306
The problem we've been having with deploying the latest ose-jenkins:v4.8.0 image (also v4.9.0 and v4.10.0) appears to be related to FIPS, since we have a FIPS-enabled OpenShift cluster. I can get it to come up cleanly by adding the following variable to JAVA_TOOL_OPTIONS in our template: -Dcom.redhat.fips=false. As soon as I remove that variable from the template, it fails with errors similar to what I originally posted above. As for the ocp-tools-4 Jenkins image, I haven't tried it yet, but it only has a "B" security health index, so I'd rather stick with the latest ose-jenkins image, which has an "A" health index.
my fix for https://bugzilla.redhat.com/show_bug.cgi?id=2066019 address both the core jenkins and agent image for the FIPs situation, so we can close this BZ as a dupe of that. Once QE starts verifies, we can start backporting. While the 4.11 version won't be available until 4.11 GAs in a couple of months, the 4.10 version of the fix, once merged, would be available at the next 4.10 errata. Looking at the container catalog for ocp-tools-4 imge I do not think the latest errata that has the necessary fixes has been posted yet (the last tag update is still March 22). I just pulled it and brought it up and confirmed it does not have Akram's fixes which we have verified internally. So I would hold off on that for now. *** This bug has been marked as a duplicate of bug 2066019 ***