Bug 1850528

Summary: Unable to login openshift3/jenkins-2-rhel7:v3.11.232 - OAUTH failure
Product: OpenShift Container Platform Reporter: gekulkar <gekulkar>
Component: JenkinsAssignee: Vibhav Bobade <vbobade>
Status: CLOSED ERRATA QA Contact: Jitendar Singh <jitsingh>
Severity: low Docs Contact:
Priority: unspecified    
Version: 3.11.0CC: abenaiss, andbartl, aos-bugs, pbhattac, vbobade
Target Milestone: ---   
Target Release: 3.11.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-08-26 22:44:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description gekulkar 2020-06-24 12:49:13 UTC
Description of problem:
After upgrading Image to the latest(v3.11.232-3) unable to login to Jenkins.
After downgrading the image to the previous version we are able to log in.


Version-Release number of selected component (if applicable):
OpenShift v3.11

openshift-login:1.0.23 plugin --> Unable to login
openshift-login:1.0.22 plugin --> Able to login


How reproducible:
NA


Actual results:
unable to login.

+++++++++++++

020-06-18 14:09:31 INFO    org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm useProviderOAuthEndpoint Now checking if we are on an OpenShift3 cluster and the answer is:  true
2020-06-18 14:09:31 INFO    org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm initializeHttpsProxyAuthenticator Checking if HTTPS proxy initialization is required ... 
2020-06-18 14:09:51 INFO    org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm transportToUse OpenShift OAuth provider token endpoint failed unexpectedly using this pod's SA's certificate
java.net.SocketTimeoutException: connect timed out
	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
	at java.base/java.net.Socket.connect(Socket.java:609)
	at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:285)
	at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
	at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:265)
	at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:372)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:168)
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:143)
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:79)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:996)
	at org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm.transportToUse(OpenShiftOAuth2SecurityRealm.java:542)
	at org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm.populateDefaults(OpenShiftOAuth2SecurityRealm.java:406)
	at org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm.doCommenceLogin(OpenShiftOAuth2SecurityRealm.java:1019)
	at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710)
	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)
	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408)
	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212)
	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145)
	at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:535)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)
	at org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:219)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:676)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at org.openshift.jenkins.plugins.openshiftlogin.OpenShiftPermissionFilter.doFilter(OpenShiftPermissionFilter.java:236)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:243)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:76)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:117)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:118)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:36)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:566)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:500)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
	at java.base/java.lang.Thread.run(Thread.java:834)
+++++++++++++++++++++++++++++++++++++


Expected results:
Successful log in 


Additional info:
We can see similar Bugzilla for OCP v4.1 
~~~
https://bugzilla.redhat.com/show_bug.cgi?id=1760798
~~~

Comment 6 Vibhav Bobade 2020-07-01 05:57:53 UTC
Hi gekulkar,

I tested for the reproducibility of this bug request on the give Jenkins Image v3.11.232-3 and wasn't able to reproduce with the same steps you have provided. 

Is the cluster behind a proxy ? 
Is openshift-sync plugin being used with proxy settings ? 
How reproducible is this issue ?

Considering the timeout, this could also be a network performance issue on the cluster. 


Do let me know if you have any other information along with the one above that could help resolve this issue.

Regards, Vibhav

Comment 9 gekulkar 2020-07-02 12:53:37 UTC
Hello Vaibhav,

We have an update from the customer regarding your queries.

~~~

Is the cluster behind a proxy? 
no

Is openshift-sync plugin being used with proxy settings? 
no

Are you able to reproduce the issue?
yes, upgrading the openshift-login:1.0.22 to 1.0.23 introduces this issue reliably, downgrading back to 1.0.22 'fixes' the issue.

Were there any network discrepancies while you faced this issue? (Considering the timeout error)
no 

~~~ 

Thank you.

Regards,
Geetesh

Comment 21 Jitendar Singh 2020-08-13 12:34:41 UTC
VERIFIED
=====================================================
- Login - successful
- Pipeline build triggered - successful
- Sync Plugin - successful

==========================================================
oc get pods
NAME               READY   STATUS    RESTARTS   AGE
jenkins-1-deploy   1/1     Running   0          2m
jenkins-1-plrrr    0/1     Running   0          2m
 jsingh@localhost  ~/go/src/github.com/redhat-developer/jenkins   jenkins-test ●✚  oc get pods
NAME              READY   STATUS    RESTARTS   AGE
jenkins-1-plrrr   1/1     Running   0          2m
 jsingh@localhost  ~/go/src/github.com/redhat-developer/jenkins   jenkins-test ●✚  oc rsh jenkins-1-plrrr
sh-4.2$ cat /var/lib/jenkins/plugins/openshift-login/META-INF/MANIFEST.MF |grep Implementation-Version
Implementation-Version: 1.0.24
===========================
jsingh@localhost  ~/go/src/github.com/redhat-developer/jenkins   jenkins-test ●✚  oc get routes
NAME      HOST/PORT                                                               PATH   SERVICES   PORT    TERMINATION     WILDCARD
jenkins   jenkins-jenkins-test.apps.jenkins311maven.lab.upshift.rdu2.redhat.com          jenkins    <all>   edge/Redirect   None
 jsingh@localhost  ~/go/src/github.com/redhat-developer/jenkins   jenkins-test ●✚  oc new-app -f  https://raw.githubusercontent.com/openshift/origin/master/examples/jenkins/pipeline/maven-pipeline.yaml                    
--> Deploying template "jenkins-test/maven-pipeline" for "https://raw.githubusercontent.com/openshift/origin/master/examples/jenkins/pipeline/maven-pipeline.yaml" to project jenkins-test

     * With parameters:
        * Application Name=openshift-jee-sample
        * Source URL=https://github.com/openshift/openshift-jee-sample.git
        * Source Ref=master
        * GitHub Webhook Secret=hWR2Xht5ty8Hme7lsWsChlGXXQFbxxnrkqhuCsSJ # generated
        * Generic Webhook Secret=J5Mkepwi1TS7PMBQhN8paqj55urBPG37GgwvvhTe # generated

--> Creating resources ...
    imagestream.image.openshift.io "openshift-jee-sample" created
    imagestream.image.openshift.io "wildfly" created
    buildconfig.build.openshift.io "openshift-jee-sample" created
    buildconfig.build.openshift.io "openshift-jee-sample-docker" created
    deploymentconfig.apps.openshift.io "openshift-jee-sample" created
    service "openshift-jee-sample" created
    route.route.openshift.io "openshift-jee-sample" created
--> Success
    Use 'oc start-build openshift-jee-sample' to start a build.
    Use 'oc start-build openshift-jee-sample-docker' to start a build.
    Access your application via route 'openshift-jee-sample-jenkins-test.apps.jenkins311maven.lab.upshift.rdu2.redhat.com' 
    Run 'oc status' to view your app.
 jsingh@localhost  ~/go/src/github.com/redhat-developer/jenkins   jenkins-test ●✚  oc start-build openshift-jee-sample
build.build.openshift.io/openshift-jee-sample-1 started
jsingh@localhost  ~/go/src/github.com/redhat-developer/jenkins   jenkins-test ●✚  oc get pods
NAME                                  READY   STATUS      RESTARTS   AGE
jenkins-1-plrrr                       1/1     Running     0          11m
openshift-jee-sample-1-deploy         1/1     Running     0          37s
openshift-jee-sample-1-p9wjz          0/1     Running     0          32s
openshift-jee-sample-docker-1-build   0/1     Completed   0          1m
 jsingh@localhost  ~/go/src/github.com/redhat-developer/jenkins   jenkins-test ●✚  oc get pods                        
NAME                                  READY   STATUS      RESTARTS   AGE
jenkins-1-plrrr                       1/1     Running     0          12m
openshift-jee-sample-1-p9wjz          1/1     Running     0          1m
openshift-jee-sample-docker-1-build   0/1     Completed   0          1m

Comment 23 errata-xmlrpc 2020-08-26 22:44:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 3.11.272 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3245