Bug 1925524 - openshift-jenkins-sync plugin does not scale on OCP 4
Summary: openshift-jenkins-sync plugin does not scale on OCP 4
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Jenkins
Version: 4.7
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
: 4.9.0
Assignee: Akram Ben Aissi
QA Contact: Jitendar Singh
URL:
Whiteboard:
: 1934564 1968623 (view as bug list)
Depends On:
Blocks: 1925610
TreeView+ depends on / blocked
 
Reported: 2021-02-05 12:47 UTC by Akram Ben Aissi
Modified: 2023-09-15 01:00 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1925610 (view as bug list)
Environment:
Last Closed: 2021-10-18 17:29:20 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift jenkins-sync-plugin pull 386 0 None closed Bug 1925524: Migrate to shared informers and watchers removed 2021-07-21 21:52:12 UTC
Github openshift jenkins pull 1297 0 None closed Bug 1925524: bump k8s plugin to 1.30 to enable 1.0.48 release of openshift sync plugin 2021-07-21 21:52:12 UTC
Github openshift origin pull 26336 0 None closed Bug 1925524: tweak jenkins imagestreamtag podtemplate test to label imagestream; jenkins multi namespace test has been m... 2021-07-21 21:52:15 UTC
Red Hat Product Errata RHSA-2021:3759 0 None None None 2021-10-18 17:29:39 UTC

Description Akram Ben Aissi 2021-02-05 12:47:27 UTC
This bug was initially created as a copy of Bug #1902692

I am copying this bug because: 



Description of problem:

Customer has deployed Jenkins on Openshift 4.4.16, almost identical setup on Openshift 3 worked without issues:


	There are 20 namespaces/projects that jenkins scans every 5 secs and each namespace got couple of jobs so it would be around 100 jobs configured

	These jobs are regular pipelines doing the deployment in openshift.

	Customer starts the configure-* job from the openshift console and it calls the other job usually deploy-* which in turn calls the other jobs. 

	The issue is when we start the configure-* job sometimes it doesn't start and stays in new/pending status until we either restart jenkins or going to manage jenkins->configure system ->apply.

We can see the following exception in Jenkins pod's log:

~~~
	2020-11-25 20:57:34 WARNING io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1 onFailure Exec Failure
	java.io.InterruptedIOException: executor rejected
		at okhttp3.RealCall$AsyncCall.executeOn(RealCall.java:185)
		at okhttp3.Dispatcher.promoteAndExecute(Dispatcher.java:186)
		at okhttp3.Dispatcher.enqueue(Dispatcher.java:137)
		at okhttp3.RealCall.enqueue(RealCall.java:126)
		at okhttp3.internal.ws.RealWebSocket.connect(RealWebSocket.java:193)
		at okhttp3.OkHttpClient.newWebSocket(OkHttpClient.java:435)
		at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.runWatch(WatchConnectionManager.java:153)
		at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.access$1200(WatchConnectionManager.java:48)
		at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$2$1.execute(WatchConnectionManager.java:316)
		at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$NamedRunnable.run(WatchConnectionManager.java:405)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
		at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
		at java.base/java.lang.Thread.run(Thread.java:834)
	Caused by: java.util.concurrent.RejectedExecutionException: Task okhttp3.RealCall$AsyncCall@40b9daa5 rejected from java.util.concurrent.ThreadPoolExecutor@41225385[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 96]
		at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055)
		at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825)
		at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355)
		at okhttp3.RealCall$AsyncCall.executeOn(RealCall.java:182)
		... 15 more

~~~

We suggested to the customer to use the plugin versions described in 'https://github.com/openshift/jenkins/blob/release-4.4/2/contrib/openshift/base-plugins.txt', but the same exception is thrown.

Attaching Jobs list, configured jenkins plugin list and jenkins pod's log.



Version-Release number of selected component (if applicable):

Openshift 4.4

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 6 Adam Kaplan 2021-06-30 20:26:30 UTC
*** Bug 1968623 has been marked as a duplicate of this bug. ***

Comment 14 Akram Ben Aissi 2021-07-19 14:20:32 UTC
*** Bug 1934564 has been marked as a duplicate of this bug. ***

Comment 26 Jitendar Singh 2021-09-06 07:46:29 UTC
VERFIED

Comment 29 errata-xmlrpc 2021-10-18 17:29:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759

Comment 30 Red Hat Bugzilla 2023-09-15 01:00:34 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days


Note You need to log in before you can comment on or make changes to this bug.