Bug 1751302

Summary:	Jenkins shows error instead of login screen after cluster upgrade from 4.1.11 to 4.1.13
Product:	OpenShift Container Platform	Reporter:	Akram Ben Aissi <abenaiss>
Component:	Jenkins	Assignee:	Akram Ben Aissi <abenaiss>
Status:	CLOSED ERRATA	QA Contact:	Wenjing Zheng <wzheng>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	4.2.0	CC:	calfonso, vbobade, wkulhane, wsun, xiuwang
Target Milestone:	---
Target Release:	4.1.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1750588
Clones:	1767479 (view as bug list)		Environment:
Last Closed:	2019-09-27 00:33:52 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1750588, 1754045
Bug Blocks:	1767479

Description Akram Ben Aissi 2019-09-11 16:42:13 UTC

+++ This bug was initially created as a clone of Bug #1750588 +++

Description of problem:

I upgraded my cluster from 4.1.11 to 4.1.13. I had a Jenkins (persistent template) deployed before the upgrade.

After the upgrade I can no longer log into Jenkins. In fact instead of the the Jenkins screen prompting me to "log in with OpenShift" I see a "Flaming Jenkins" "Oops" screen.

And a stack trace on the Jenkins Page:

A problem occurred while processing the request. Please check our bug tracker to see if a similar problem has already been reported. If it is already reported, please vote and put a comment on it to let us gauge the impact of the problem. If you think this is a new issue, please file a new issue. When you file an issue, make sure to add the entire stack trace, along with the version of Jenkins and relevant plugins. The users list might be also useful in understanding what has happened.

Stack trace
java.lang.NullPointerException
	at com.google.api.client.repackaged.com.google.common.base.Preconditions.checkNotNull(Preconditions.java:191)
	at com.google.api.client.util.Preconditions.checkNotNull(Preconditions.java:127)
	at com.google.api.client.auth.oauth2.AuthorizationCodeFlow$Builder.setTransport(AuthorizationCodeFlow.java:537)
	at com.google.api.client.auth.oauth2.AuthorizationCodeFlow$Builder.<init>(AuthorizationCodeFlow.java:486)
	at org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm.newOAuthSession(OpenShiftOAuth2SecurityRealm.java:945)
	at org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm.doCommenceLogin(OpenShiftOAuth2SecurityRealm.java:1165)
	at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)
	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408)
	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212)
	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145)
	at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:535)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)
Caused: javax.servlet.ServletException
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:797)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)
	at org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:219)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:676)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at org.openshift.jenkins.plugins.openshiftlogin.OpenShiftPermissionFilter.doFilter(OpenShiftPermissionFilter.java:247)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:243)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:64)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1701)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1668)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:502)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
	at java.lang.Thread.run(Thread.java:748)



When looking at the Jenkins logs I see the following error:


Sep 09, 2019 4:09:37 PM org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm populateDefaults
INFO: OpenShift OAuth returning true with namespace gpte-jenkins SA dir null default /run/secrets/kubernetes.io/serviceaccount SA name null default jenkins client ID null default system:serviceaccount:gpte-jenkins:jenkins secret null default eyJhb....... redirect null default https://oauth-openshift.apps.shared.na.openshift.opentlc.com server null default https://kubernetes.default:443
Sep 09, 2019 4:09:37 PM org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm useProviderOAuthEndpoint
INFO: OpenShift OAuth server is 4.x, specifically OpenShiftVersionInfo: major: 1 minor: 13+ gitVersion: v1.13.4+a8c5f5b
Sep 09, 2019 4:09:37 PM org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm newOAuthSession
INFO: OpenShift OAuth using OAuth Provider specified endpoints for this login flow
Sep 09, 2019 4:09:37 PM org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm transportToUse
INFO: OpenShift OAuth got a SSL error when accessing the issuer's token endpoint when using the SA certificate
Sep 09, 2019 4:09:37 PM org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm transportToUse
INFO: OpenShift OAuth provider token endpoint failed unexpectedly using the JVMs default keystore
java.lang.NullPointerException
at org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm.transportToUse(OpenShiftOAuth2SecurityRealm.java:579)


A newly deployed jenkins (different name, same namespace) works fine.


Something happened and I have no idea what...


Version-Release number of selected component (if applicable):
4.1.11/4.1.13, Jenkins 2.176.2

How reproducible:
Not sure

--- Additional comment from Akram Ben Aissi on 2019-09-10 07:02:29 UTC ---

Hi Wolfgang,

We will have an urgent look at it. 
Can you please send the versions of the plugins that are deployed on your  Jenkins instance as well?
Also, please tell us the version of the Jenkins server used in the "old" deployment and the new ones. (You reported 2.176.2, an update to 2.176.3 is pending, I want to check if you are not using this one, or a previous one based on 2.168.x ).

Comment 2 Wenjing Zheng 2019-09-23 08:41:44 UTC

Verified by upgrading from 4.1.6 to 4.1.7, jenkins console can be accessed with jenkins-persistent template on a cluster enabled google IDP.

Comment 4 errata-xmlrpc 2019-09-27 00:33:52 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2856