I think the stack trace is 1.0.52 is due to the dependencies of the sync plugin not being upgraded. Given the challenges of managing dependencies in the plugin manager, we tend to prefer that folks wait for our official image to be updated.

We believe this issue will be addressed once the 1.0.52 update merges, but will verify this before closing.

I've also noticed that, similarly, OpenShift Secrets that were previously synced are also getting periodically removed by Jenkins. Snippet of Jenkins log below: 

[...]
2022-01-17 19:01:32 INFO    io.fabric8.jenkins.openshiftsync.SecretInformer onUpdate Secret informer received update event for: artifactory-registry
2022-01-17 19:01:32 INFO    io.fabric8.jenkins.openshiftsync.SecretManager updateCredential Modifying Secret with Uid 9b57ca32-1c81-4e4b-bfd8-99936e7f1a87 with Name artifactory-registry
2022-01-17 19:01:32 INFO    io.fabric8.jenkins.openshiftsync.SecretManager validSecret Validating Secret with Uid 9b57ca32-1c81-4e4b-bfd8-99936e7f1a87 with Name artifactory-registry
2022-01-17 19:01:32 INFO    io.fabric8.jenkins.openshiftsync.SecretInformer onUpdate Secret informer received update event for: gitlab-scg
2022-01-17 19:01:32 INFO    io.fabric8.jenkins.openshiftsync.SecretManager updateCredential Modifying Secret with Uid e989c866-c9de-4e32-b7c3-b3ab55ddcdd0 with Name gitlab-scg
2022-01-17 19:01:32 INFO    io.fabric8.jenkins.openshiftsync.SecretManager validSecret Validating Secret with Uid e989c866-c9de-4e32-b7c3-b3ab55ddcdd0 with Name gitlab-scg
2022-01-17 19:02:12 WARNING org.jenkinsci.plugins.prometheus.DiskUsageCollector collect Cannot get disk usage data. Install CloudBees Disk Usage Simple plugin to enable
2022-01-17 19:03:25 WARNING io.fabric8.kubernetes.client.informers.cache.ReflectorWatcher onClose Watch closing with exception
io.fabric8.kubernetes.client.WatcherException: too old resource version: 119855318 (119900627)
	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$TypedWatcherWebSocketListener.onMessage(WatchConnectionManager.java:103)
	at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:323)
	at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219)
	at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105)
	at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: too old resource version: 119855318 (119900627)
	... 11 more

2022-01-17 19:03:25 INFO    io.fabric8.jenkins.openshiftsync.SecretInformer onDelete Secret informer received delete event for: artifactory-registry
2022-01-17 19:03:25 INFO    io.fabric8.jenkins.openshiftsync.CredentialsUtils deleteCredential Deleted credential carrier-int-artifactory-registry from Secret NamespaceName{carrier-int:artifactory-registry} with revision: 119855318
2022-01-17 19:03:25 INFO    io.fabric8.jenkins.openshiftsync.SecretInformer onDelete Secret informer received delete event for: gitlab-scg
2022-01-17 19:03:25 INFO    io.fabric8.jenkins.openshiftsync.CredentialsUtils deleteCredential About to delete credential carrier-int-gitlab-scgwhich is referenced by jobs: test-cluster-pipeline 
2022-01-17 19:03:25 INFO    io.fabric8.jenkins.openshiftsync.CredentialsUtils deleteCredential Deleted credential carrier-int-gitlab-scg from Secret NamespaceName{carrier-int:gitlab-scg} with revision: 119854874
[...]

The java.lang.NoSuchMethodError has also been reported in upstream Jenkins: https://issues.jenkins.io/browse/JENKINS-66468?focusedCommentId=420796&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-420796

I could observe this behaviour with latest origin Jenkins images (+ most recent okhttp-api plugin) and this exception blocked the integration of Jenkins into the OC Cluster completely.
The following patch gets the openshift-sync and the Cluster integration plugin back to work, but before filing a PR to the openshift/jenkins-sync-plugin I would like you to review the changes, if they are correct. Thank you!


Index: src/main/java/io/fabric8/jenkins/openshiftsync/OpenShiftUtils.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/src/main/java/io/fabric8/jenkins/openshiftsync/OpenShiftUtils.java b/src/main/java/io/fabric8/jenkins/openshiftsync/OpenShiftUtils.java
--- a/src/main/java/io/fabric8/jenkins/openshiftsync/OpenShiftUtils.java	(revision 9330934fe76831a5838e51c6be6c49f20c6c138f)
+++ b/src/main/java/io/fabric8/jenkins/openshiftsync/OpenShiftUtils.java	(date 1642698701726)
@@ -130,7 +130,7 @@
      *
      * @param serverUrl the optional URL of where the OpenShift cluster API server
      *                  is running
-     * @param maxConnections2 
+     * @param maxConnections
      */
     public synchronized static void initializeOpenShiftClient(String serverUrl, int maxConnections) {
         if (openShiftClient != null) {
@@ -146,14 +146,12 @@
         
         String version = JENKINS_INSTANCE.getPluginManager().getPlugin("openshift-sync").getVersion();
         config.setUserAgent("openshift-sync-plugin-" + version + "/fabric8-" + Version.clientVersion());
+        if (maxConnections > 0) {
+            config.setMaxConcurrentRequestsPerHost(maxConnections);
+            config.setMaxConcurrentRequests(maxConnections);
+        }
         openShiftClient = new DefaultOpenShiftClient(config);
-        logger.log(INFO, "New OpenShift client initialized: " + openShiftClient);
-
-        DefaultOpenShiftClient defClient = (DefaultOpenShiftClient) openShiftClient;
-        Dispatcher dispatcher = defClient.getHttpClient().dispatcher();
-//        int maxConnections = 100;//GlobalPluginConfiguration.get().getMaxConnections();
-        dispatcher.setMaxRequestsPerHost(maxConnections);
-        dispatcher.setMaxRequests(maxConnections);
+        logger.log(INFO, "New OpenShift client initialized: " + OpenShiftUtils.openShiftClient);
     }
 
     public synchronized static OpenShiftClient getOpenShiftClient() {

Apparently this issue is already fixed by openshift-sync 1.0.52. I have been running jenkins instance built from master for a whole day, and podtemplate is still there. As for the `NoSuchMethodError: 'okhttp3.OkHttpClient io.fabric8.openshift.client.DefaultOpenShiftClient.getHttpClient()'` error, it happens when `openshift-sync` is updated, but `kubernetes` and `kubernetes client api` plugins (on which `openshift-sync` depends) are not. They end up being built against different versions of underlying dependencies, which leads to this error.

Sorry Alice, but we keep seeing the 'getHttpClient' with the latest version of every plugin:

Openshift 4.8
Jenkins 2.303.3
Opensift syn plugin: 1.0.52
Kubernetes plugin: 1.31.3
Kubernetes Client Api Plugin: 5.11.2-182

We're running Jenkins from the last and official certified (and supported) image in RH Catalog, not built from master, that's a difference with your test environment.

`openshift-sync` 1.0.52 depends on earlier versions of Kubernetes plugins, as seen from https://github.com/openshift/jenkins-sync-plugin/blob/539b071fad9b7e2225ec6a9cee0b8a22c01effb1/pom.xml
Problem is, these different versions of plugins all point towards different versions of Jenkins's BOM (which lists versions of libraries provided by jenkins), and eventually these plugins are compiled against different versions of underlying http client library. It seems that some code might have been deprecated and later removed, which is affecting some of the plugins in some situations.
However, please feel free to create PR to the openshift/jenkins-sync-plugin, and if it all works, we'll get it merged.

These are versions which current master branch of our jenkins image gets bundled with:
    <dependency>
      <groupId>org.csanchez.jenkins.plugins</groupId>
      <artifactId>kubernetes</artifactId>
      <version>1.31.0</version>
    </dependency>
    <dependency>
      <groupId>org.jenkins-ci.plugins</groupId>
      <artifactId>kubernetes-client-api</artifactId>
      <version>5.10.1-171.vaa0774fb8c20</version>
    </dependency>

Thanks, Alice,

I see there's a little mess with plugin versions. Just to clarify... do you think that downgrading the kubernetes and kubernetes-client-api to those previous versions will work? Or does it's needed a full build of Jenkins so the proper dependent libraries get bundled?

About the PR, don't know what you mean, I don't have a solution for this bug, so I can't create a PR. Maybe you refer to the user 'emschu' proposal above? I think he/she is waiting for your review before creating the PR.

Regards,
Raúl

Hi again Alice,

Following your suggestion, just tested with these versions:

Openshift 4.8
Jenkins 2.303.3
Opensift syn plugin: 1.0.52
Kubernetes plugin: 1.31.0
Kubernetes Client Api Plugin: 5.10.1-171.vaa0774fb8c20

Just downgraded versions of Kubernetes and Kubernetes-client-api, ant it worked. httpClient error is gone and secrets and other objects are in sync now.

Thank you!

Yes, Raúl, I'm sorry, I meant to tell the PR thing to emschu.
I am going to close the bug.

Hi Alice,

As you wish, there's a workaround if you know the exact combination of plugin versions, but, from my humble perspective, this is still a bug because Jenkins doesn't work properly in Openshift with the latest supported versions of those plugins.

Hi everyone!


We are experiencing the same problem here... 
I have an Openshift cluster created using ROSA (AWS) and after create Jenkins using RedHat templates, I'm unable to use it consistently because every day I lose my credentials, history of pipelines and the pipelines as well... 
To make Jenkins functional again, I need to delete the templates and import them again.
Monitoring Jenkins logs, I see same behavior reported by jamieclinton 

My current settings are:

Openshift 4.8.23
Jenkins 2.303.3
Openshift sync plugin: 1.0.50
Kubernetes plugin: 1.30.1
Kubernetes Client Api Plugin: 5.4.1

I agree with Raul, this seems to be a bug that needs to have some attention.

I agree. I still don't know how to fix this issue. The Jenkins Plugin Manager doesn't seem to allow me to upgrade/downgrade to the recommended combination of plugins, which apparently is the following:

Openshift 4.8.2x
Jenkins 2.303.3
Opensift sync plugin: 1.0.52
Kubernetes plugin: 1.31.0
Kubernetes Client Api Plugin: 5.10.1-171.vaa0774fb8c20

Is there some other way I can install these specific versions of the plugins?

Thanks,
  Jamie

Hi Alice,

Can you please reopen this? Unless I'm missing something, it still seems to be a bug that is blocking us (and, apparently, others) from running Jenkins pipelines reliably on OpenShift 4.8

Thanks,
  Jamie

Hi Jamie,

In the plugin manager in Jenkins, you can find an "Advanced" tab. There you can upload and install specific versions of plugins. Just download manually the plugin version from the Releases link in jenkins.io in hpi format( i.e: from kubernetes plugins -> https://updates.jenkins.io/download/plugins/kubernetes/1.30.1/kubernetes.hpi ), and upload and install. Then restart and voilà. That worked for me in order to apply this workaround. 

Anyway, I agree, this should be considered a bug because we're locked to a set of versions that we cant change/upgrade without breaking Jenkins. The bug will show again in automatic process upgrades for Jenkins, or Jenkins plugins.

Due to the nature of the Jenkins plugin ecosystem, we must pin the plugin versions of our dependencies. Upgrading Jenkins plugins independently is not supported outside of exceptional circumstances.

We do provide backports of plugin upgrades through the Jenkins image in the OpenShift payload, based on bug severity and OCP version support level. The default Jenkins templates ensure that Jenkins is upgraded when the cluster is upgraded.

Thank you Adam.

Just to clarify, even if Jenkins shows recommendations to update plugins in the Plugin Manager (in theory, compatible with the Jenkins version), if Jenkins is provided as an image stream in Openshift, your recommendation is to don't upgrade and stick to the version of the plugin provided in the base image. Right?

Hi Raúl,

Thanks for your instructions on how to install the recommended plugin versions from the Plugin Manager Advanced tab. It works! The workaround appears to have fixed the OpenShift Sync problems using this combination:

Openshift 4.8.22
Jenkins 2.303.3
Opensift sync plugin: 1.0.52
Kubernetes plugin: 1.31.0
Kubernetes Client Api Plugin: 5.10.1-171.vaa0774fb8c20

*** Bug 2043084 has been marked as a duplicate of this bug. ***