Description of problem: I upgraded my cluster from 4.1.11 to 4.1.13. I had a Jenkins (persistent template) deployed before the upgrade. After the upgrade I can no longer log into Jenkins. In fact instead of the the Jenkins screen prompting me to "log in with OpenShift" I see a "Flaming Jenkins" "Oops" screen. And a stack trace on the Jenkins Page: A problem occurred while processing the request. Please check our bug tracker to see if a similar problem has already been reported. If it is already reported, please vote and put a comment on it to let us gauge the impact of the problem. If you think this is a new issue, please file a new issue. When you file an issue, make sure to add the entire stack trace, along with the version of Jenkins and relevant plugins. The users list might be also useful in understanding what has happened. Stack trace java.lang.NullPointerException at com.google.api.client.repackaged.com.google.common.base.Preconditions.checkNotNull(Preconditions.java:191) at com.google.api.client.util.Preconditions.checkNotNull(Preconditions.java:127) at com.google.api.client.auth.oauth2.AuthorizationCodeFlow$Builder.setTransport(AuthorizationCodeFlow.java:537) at com.google.api.client.auth.oauth2.AuthorizationCodeFlow$Builder.<init>(AuthorizationCodeFlow.java:486) at org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm.newOAuthSession(OpenShiftOAuth2SecurityRealm.java:945) at org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm.doCommenceLogin(OpenShiftOAuth2SecurityRealm.java:1165) at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627) at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145) at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:535) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747) Caused: javax.servlet.ServletException at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:797) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878) at org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:219) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:676) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at org.openshift.jenkins.plugins.openshiftlogin.OpenShiftPermissionFilter.doFilter(OpenShiftPermissionFilter.java:247) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:243) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:64) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1701) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1668) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:502) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.lang.Thread.run(Thread.java:748) When looking at the Jenkins logs I see the following error: Sep 09, 2019 4:09:37 PM org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm populateDefaults INFO: OpenShift OAuth returning true with namespace gpte-jenkins SA dir null default /run/secrets/kubernetes.io/serviceaccount SA name null default jenkins client ID null default system:serviceaccount:gpte-jenkins:jenkins secret null default eyJhb....... redirect null default https://oauth-openshift.apps.shared.na.openshift.opentlc.com server null default https://kubernetes.default:443 Sep 09, 2019 4:09:37 PM org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm useProviderOAuthEndpoint INFO: OpenShift OAuth server is 4.x, specifically OpenShiftVersionInfo: major: 1 minor: 13+ gitVersion: v1.13.4+a8c5f5b Sep 09, 2019 4:09:37 PM org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm newOAuthSession INFO: OpenShift OAuth using OAuth Provider specified endpoints for this login flow Sep 09, 2019 4:09:37 PM org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm transportToUse INFO: OpenShift OAuth got a SSL error when accessing the issuer's token endpoint when using the SA certificate Sep 09, 2019 4:09:37 PM org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm transportToUse INFO: OpenShift OAuth provider token endpoint failed unexpectedly using the JVMs default keystore java.lang.NullPointerException at org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm.transportToUse(OpenShiftOAuth2SecurityRealm.java:579) A newly deployed jenkins (different name, same namespace) works fine. Something happened and I have no idea what... Version-Release number of selected component (if applicable): 4.1.11/4.1.13, Jenkins 2.176.2 How reproducible: Not sure
Hi Wolfgang, We will have an urgent look at it. Can you please send the versions of the plugins that are deployed on your Jenkins instance as well? Also, please tell us the version of the Jenkins server used in the "old" deployment and the new ones. (You reported 2.176.2, an update to 2.176.3 is pending, I want to check if you are not using this one, or a previous one based on 2.168.x ).
Hi Wolfgang, as discussed by slack and then tested on your instance and on also on OpenShift Online for which we had a similar issue, the fix is pending. The PR fixing it is: https://github.com/openshift/jenkins-openshift-login-plugin/pull/80 I changed your bz to 4.2 for process purposes and I made a clone of it to 4.1. We will release the plugin very soon. Code freeze for 4.2 is in 2 days. You can find as an attachement the openshift-login.hpi plugin which contains the fix.
Created attachment 1614173 [details] openshift login.hpi fixing the issue
I don't reproduce this issue when upgrade cluster from 4.1.11 to 4.1.13 with jenkins-persistent app with htpasswd idp. However I will keep an eye on this issue.
Hi @XiuJuan Wang The issue does not occur with htpasswd IDP, you need to set up an Oauth IDP to see it. We managed to reproduce it: - on an OpenShift that we configured to use GitHub Oauth Identity provider - an OpenShift online (which is configured to use access.redhat.com as an OAuth provider)
Update jenkins from 4.2.0-0.nightly-2019-09-15-052022 to 4.2.0-0.nightly-2019-09-21-183303, could access jenkins console with persistent volume with google oauth, no error shown up.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2922
This happened again when upgrading from 4.1.18 to 4.1.20.