Bug 1417756 - Jenkins connection test read timeout fails
Summary: Jenkins connection test read timeout fails
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Image
Version: 3.x
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: ---
Assignee: Gabe Montero
QA Contact: Wang Haoran
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-30 21:47 UTC by Chris Ryan
Modified: 2017-04-05 20:53 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-04-05 20:53:15 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Chris Ryan 2017-01-30 21:47:57 UTC
Description of problem:
When using the Jenkins pipeline 'Test Connection' button in the pipeline snippet generator, the connection fails due to the short timeout. 

Version-Release number of selected component (if applicable):
Online stg

How reproducible:
Always

Steps to Reproduce:
1. Create a jenkins application
oc new-app --file https://raw.githubusercontent.com/openshift/origin/master/examples/jenkins/application-template.json
2. Create a pipeline job, using the 'New Item' menu option
3. Navigate to the pipeline snippet generator page:
https://<jenkins_route>/job/<job_name>/pipeline-syntax/
4. Enter in the API Endpoint URL, and project, and click "Test Connection"

Actual results:
The test fails:
Jan 30, 2017 9:37:20 PM com.openshift.jenkins.plugins.pipeline.model.IOpenShiftPluginDescriptor doTestConnection
SEVERE: doTestConnection
com.openshift.restclient.OpenShiftException: Exception while trying to determine the health/ready response of the server
	at com.openshift.internal.restclient.DefaultClient.getServerReadyStatus(DefaultClient.java:251)
	at com.openshift.jenkins.plugins.pipeline.model.IOpenShiftPluginDescriptor.doTestConnection(IOpenShiftPluginDescriptor.java:125)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:324)
	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:167)
	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:100)
	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
	at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:233)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
	at org.kohsuke.stapler.MetaClass$11.dispatch(MetaClass.java:380)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
	at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:233)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
	at org.openshift.jenkins.plugins.openshiftlogin.OpenShiftPermissionFilter.doFilter(OpenShiftPermissionFilter.java:101)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
	at org.eclipse.jetty.server.Server.handle(Server.java:499)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: connect timed out
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at okhttp3.internal.platform.Platform.connectSocket(Platform.java:124)
	at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:187)
	at okhttp3.internal.connection.RealConnection.buildConnection(RealConnection.java:173)
	at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:114)
	at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:193)
	at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:129)
	at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:98)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:109)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:124)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at com.openshift.internal.restclient.okhttp.ResponseCodeInterceptor.intercept(ResponseCodeInterceptor.java:54)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:170)
	at okhttp3.RealCall.execute(RealCall.java:60)
	at com.openshift.internal.restclient.DefaultClient.getServerReadyStatus(DefaultClient.java:247)
	... 78 more


Expected results:
The test should succeed

Additional info:

Comment 1 Gabe Montero 2017-01-31 14:30:25 UTC
The hard coded connection timeout of 5 seconds is too aggressive for the more stressed environments like online stg.

the default okhttp3 conn timeout is 2 minutes.

the fix will minimally increase the timeout, hopefully allow it to be tunable.

Comment 2 Gabe Montero 2017-01-31 21:38:21 UTC
Hey Troy, per https://github.com/openshift/jenkins/pull/240, we'll need a new openshift-pipeline RPM for v1.0.42 and new jenkins images out on brew pulp for defect verification.

Comment 3 Gabe Montero 2017-01-31 21:39:06 UTC
Of course send bug back to me when complete and I'll move to QA - thanks.

Comment 4 Gabe Montero 2017-02-01 16:11:53 UTC
Troy pinged me that the rhel7 image update machinery is in the works.  I'll take this one back and monitor brew-pulp for the image's availability.

Thanks Troy!

Comment 5 Gabe Montero 2017-02-01 19:14:37 UTC
The brew-pulp openshift jenkins rhel7 images now have v1.0.42 of openshift-pipeline, which includes the fix for this issue.

QA should now be able to verify this bug by importing this new version of the openshfit jenkins rhel7 images into the appropriate environments.

Comment 6 Dongbo Yan 2017-02-06 08:35:19 UTC
test with /openshift3/jenkins-2-rhel7@sha256:1e70178a88330eb69996df3fa8d1658f360eb6753001e453990aedbad41632f8
openshift-pipeline	1.0.42

Comment 8 Gabe Montero 2017-02-06 14:17:39 UTC
Dongbo Yan - that is the correct behavior.

It is attempting to make a connection within the pod.  Public URLs like https://api.dev-preview-stg.openshift.com are not accessible within the pod.

I assert that if you open a terminal in the pod and attempt to ping api.dev-preview-stg.openshift.com it will not resolve.


Note You need to log in before you can comment on or make changes to this bug.