Bug 1902692

Created attachment 1734812 [details]
Jenkins error

Description of problem:

Customer has deployed Jenkins on Openshift 4.4.16, almost identical setup on Openshift 3 worked without issues:


	There are 20 namespaces/projects that jenkins scans every 5 secs and each namespace got couple of jobs so it would be around 100 jobs configured

	These jobs are regular pipelines doing the deployment in openshift.

	Customer starts the configure-* job from the openshift console and it calls the other job usually deploy-* which in turn calls the other jobs. 

	The issue is when we start the configure-* job sometimes it doesn't start and stays in new/pending status until we either restart jenkins or going to manage jenkins->configure system ->apply.

We can see the following exception in Jenkins pod's log:

~~~
	2020-11-25 20:57:34 WARNING io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1 onFailure Exec Failure
	java.io.InterruptedIOException: executor rejected
		at okhttp3.RealCall$AsyncCall.executeOn(RealCall.java:185)
		at okhttp3.Dispatcher.promoteAndExecute(Dispatcher.java:186)
		at okhttp3.Dispatcher.enqueue(Dispatcher.java:137)
		at okhttp3.RealCall.enqueue(RealCall.java:126)
		at okhttp3.internal.ws.RealWebSocket.connect(RealWebSocket.java:193)
		at okhttp3.OkHttpClient.newWebSocket(OkHttpClient.java:435)
		at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.runWatch(WatchConnectionManager.java:153)
		at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.access$1200(WatchConnectionManager.java:48)
		at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$2$1.execute(WatchConnectionManager.java:316)
		at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$NamedRunnable.run(WatchConnectionManager.java:405)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
		at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
		at java.base/java.lang.Thread.run(Thread.java:834)
	Caused by: java.util.concurrent.RejectedExecutionException: Task okhttp3.RealCall$AsyncCall@40b9daa5 rejected from java.util.concurrent.ThreadPoolExecutor@41225385[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 96]
		at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055)
		at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825)
		at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355)
		at okhttp3.RealCall$AsyncCall.executeOn(RealCall.java:182)
		... 15 more

~~~

We suggested to the customer to use the plugin versions described in 'https://github.com/openshift/jenkins/blob/release-4.4/2/contrib/openshift/base-plugins.txt', but the same exception is thrown.

Attaching Jobs list, configured jenkins plugin list and jenkins pod's log.



Version-Release number of selected component (if applicable):

Openshift 4.4

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info: