Bug 1925615 - 'java.util.concurrent.RejectedExecutionException' thrown on Jenkins deployment
Summary: 'java.util.concurrent.RejectedExecutionException' thrown on Jenkins deployment
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Jenkins
Version: 4.5
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.5.z
Assignee: Akram Ben Aissi
QA Contact: Jitendar Singh
URL:
Whiteboard:
Depends On: 1925610
Blocks: 1902692
TreeView+ depends on / blocked
 
Reported: 2021-02-05 16:47 UTC by Akram Ben Aissi
Modified: 2021-06-21 18:21 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1925610
Environment:
Last Closed: 2021-06-21 18:21:47 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Akram Ben Aissi 2021-02-05 16:47:15 UTC
+++ This bug was initially created as a clone of Bug #1925610 +++

+++ This bug was initially created as a clone of Bug #1925524 +++

This bug was initially created as a copy of Bug #1902692

I am copying this bug because: 



Description of problem:

Customer has deployed Jenkins on Openshift 4.4.16, almost identical setup on Openshift 3 worked without issues:


	There are 20 namespaces/projects that jenkins scans every 5 secs and each namespace got couple of jobs so it would be around 100 jobs configured

	These jobs are regular pipelines doing the deployment in openshift.

	Customer starts the configure-* job from the openshift console and it calls the other job usually deploy-* which in turn calls the other jobs. 

	The issue is when we start the configure-* job sometimes it doesn't start and stays in new/pending status until we either restart jenkins or going to manage jenkins->configure system ->apply.

We can see the following exception in Jenkins pod's log:

~~~
	2020-11-25 20:57:34 WARNING io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1 onFailure Exec Failure
	java.io.InterruptedIOException: executor rejected
		at okhttp3.RealCall$AsyncCall.executeOn(RealCall.java:185)
		at okhttp3.Dispatcher.promoteAndExecute(Dispatcher.java:186)
		at okhttp3.Dispatcher.enqueue(Dispatcher.java:137)
		at okhttp3.RealCall.enqueue(RealCall.java:126)
		at okhttp3.internal.ws.RealWebSocket.connect(RealWebSocket.java:193)
		at okhttp3.OkHttpClient.newWebSocket(OkHttpClient.java:435)
		at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.runWatch(WatchConnectionManager.java:153)
		at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.access$1200(WatchConnectionManager.java:48)
		at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$2$1.execute(WatchConnectionManager.java:316)
		at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$NamedRunnable.run(WatchConnectionManager.java:405)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
		at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
		at java.base/java.lang.Thread.run(Thread.java:834)
	Caused by: java.util.concurrent.RejectedExecutionException: Task okhttp3.RealCall$AsyncCall@40b9daa5 rejected from java.util.concurrent.ThreadPoolExecutor@41225385[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 96]
		at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055)
		at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825)
		at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355)
		at okhttp3.RealCall$AsyncCall.executeOn(RealCall.java:182)
		... 15 more

~~~

We suggested to the customer to use the plugin versions described in 'https://github.com/openshift/jenkins/blob/release-4.4/2/contrib/openshift/base-plugins.txt', but the same exception is thrown.

Attaching Jobs list, configured jenkins plugin list and jenkins pod's log.



Version-Release number of selected component (if applicable):

Openshift 4.4

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Adam Kaplan 2021-06-21 18:21:47 UTC
OpenShift 4.5 reached end of full support in November 2020 [1]. As this issue does not have priority set and severity is "medium", it is not a candidate for a maintenance support fix.

OpenShift 4.6 is an Extended Update Support release, with extended maintenance support through May 2022. Customers should update to 4.6.z as soon as possible.

[1] https://access.redhat.com/support/policy/updates/openshift


Note You need to log in before you can comment on or make changes to this bug.