Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2043084

Summary: Jenkins losing some information in the Jenkins Dashboard
Product: OpenShift Container Platform Reporter: Nelson Lombo Paez <npaez>
Component: JenkinsAssignee: Alice Rum <irum>
Status: CLOSED DUPLICATE QA Contact: Jitendar Singh <jitsingh>
Severity: medium Docs Contact:
Priority: low    
Version: 4.8CC: adam.kaplan, irum, pbhattac, spandura
Target Milestone: ---   
Target Release: 4.11.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-05-17 08:08:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Jenkins Master logs none

Description Nelson Lombo Paez 2022-01-20 15:32:38 UTC 
Created attachment 1852213 [details]
Jenkins Master logs

Description of problem:

The customer is deploying Jenkins by using the permanent template:


~~~
oc new-app jenkins-persistent -p VOLUME_CAPACITY=30Gi --namespace=$PROJECT_NAME_DE
~~~

The current Jenkins version is 1.0.50.

Some resources are disappearing from time to time in their Jenkins dashboard, for example, we have created some credentials two days ago via the Jenkins dashboard but there are not in the cluster anymore. Some builds disappear from the Jenkins Dashboard even if they are still present as a BuildConfig in the OpenShift Console. As mentioned on the following Slack Thread[1] this could be due some prune configuration but we were not able to find anything related to this.

Attached are the full logs from the master pod at the time of the strange behaviour on the console, the most interesting part I've found is the following:

~~~
2022-01-19 02:38:14 INFO    io.fabric8.jenkins.openshiftsync.SecretInformer onDelete Secret informer received delete event for: jenkins-git
2022-01-19 02:38:14 INFO    io.fabric8.jenkins.openshiftsync.CredentialsUtils deleteCredential About to delete credential dev99-jenkins-gitwhich is referenced by jobs: dev99/dev99-aws-cloudformation-deploy-v2 
2022-01-19 02:38:14 INFO    io.fabric8.jenkins.openshiftsync.CredentialsUtils 
~~~

It seems that something is removing the credentials, not sure if that is the root cause here.

If they trigger a BuildConfig in the OCP console, it appears again on the Jenkins Dashboard, so the pipelines are still present on the OCP. 

I've tried to troubleshoot as much as possible but I don't know how to continue or if this is some kind of bug. I'd like to help/contribute as much as possible on this particular case, so if someone needs something from my side or the customer side, please, let us know.

For example, currently there is a build created via OpenShift template is still there but the history is gone(this could be related to the prune?) here is the log at the moment of the deletion:

~~~
2022-01-20 10:19:10 INFO    io.fabric8.jenkins.openshiftsync.JobProcessor createOrUpdateJob Updated job dev99-aws-cloudformation-deploy-v2 from BuildConfig NamespaceName{dev99:aws-cloudformation-deploy-v2} with revision: 201170875
2022-01-20 10:19:27 INFO    io.fabric8.jenkins.openshiftsync.BuildInformer onUpdate Build informer received update event for: {} to: {}200799529 200799529
2022-01-20 10:19:27 INFO    io.fabric8.jenkins.openshiftsync.BuildInformer onUpdate Build informer received update event for: {} to: {}201174424 201174424
2022-01-20 10:19:51 WARNING org.jenkinsci.plugins.prometheus.DiskUsageCollector collect Cannot get disk usage data. Install CloudBees Disk Usage Simple plugin to enable
2022-01-20 10:21:23 WARNING io.fabric8.kubernetes.client.informers.cache.ReflectorWatcher onClose Watch closing with exception
io.fabric8.kubernetes.client.WatcherException: too old resource version: 201174424 (201248022)
    at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$TypedWatcherWebSocketListener.onMessage(WatchConnectionManager.java:103)
    at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:323)
    at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219)
    at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105)
    at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
    at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
    at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
    at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: too old resource version: 201174424 (201248022)
    ... 11 more

2022-01-20 10:21:23 INFO    io.fabric8.jenkins.openshiftsync.BuildInformer onDelete Build informer received delete event for: {}Build(apiVersion=build.openshift.io/v1, kind=Build, metadata=ObjectMeta(annotations={openshift.io/jenkins-blueocean-log-url=/https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/blue/organizations/jenkins/dev99%2Fdev99-aws-cloudformation-deploy-v2/detail/dev99-aws-cloudformation-deploy-v2/2/, openshift.io/jenkins-build-uri=https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/2/, openshift.io/jenkins-console-log-url=https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/2/console, openshift.io/jenkins-log-url=https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/2/consoleText, openshift.io/jenkins-status-json={"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/2/wfapi/describe"}},"id":"2","name":"#2","status":"FAILED","startTimeMillis":1642608012654,"endTimeMillis":1642608041799,"durationMillis":29145,"queueDurationMillis":23,"pauseDurationMillis":0,"stages":[{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/2/execution/node/16/wfapi/describe"}},"id":"16","name":"Input Parameters","execNode":"","status":"FAILED","error":{"message":"Error cloning remote repo 'origin'","type":"hudson.AbortException"},"startTimeMillis":1642608036006,"durationMillis":5291,"pauseDurationMillis":0,"stageFlowNodes":[{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/2/execution/node/17/wfapi/describe"}},"id":"17","name":"Internal utility function for OpenShift DSL","execNode":"","status":"SUCCESS","startTimeMillis":1642608036034,"durationMillis":26,"pauseDurationMillis":0,"parentNodes":["16"]},{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/2/execution/node/18/wfapi/describe"}},"id":"18","name":"Internal utility function for OpenShift DSL","execNode":"","status":"SUCCESS","startTimeMillis":1642608036060,"durationMillis":58,"pauseDurationMillis":0,"parentNodes":["17"]},{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/2/execution/node/19/wfapi/describe"}},"id":"19","name":"Git","execNode":"","status":"FAILED","error":{"message":"Error cloning remote repo 'origin'","type":"hudson.AbortException"},"startTimeMillis":1642608036118,"durationMillis":5152,"pauseDurationMillis":0,"parentNodes":["18"]}]}]}}, clusterName=null, creationTimestamp=2022-01-19T16:00:12Z, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=1, labels={buildconfig=aws-cloudformation-deploy-v2, openshift.io/build-config.name=aws-cloudformation-deploy-v2, openshift.io/build.start-policy=Serial, type=jenkins-pipeline}, managedFields=[], name=aws-cloudformation-deploy-v2-2, namespace=dev99, ownerReferences=[OwnerReference(apiVersion=build.openshift.io/v1, blockOwnerDeletion=null, controller=true, kind=BuildConfig, name=aws-cloudformation-deploy-v2, uid=a8ea6145-711a-40d7-9f53-8ac96b00d6a2, additionalProperties={})], resourceVersion=200799529, selfLink=null, uid=00839659-dde3-4b22-bffe-897641620d35, additionalProperties={}), spec=BuildSpec(completionDeadlineSeconds=null, nodeSelector=null, output=BuildOutput(imageLabels=[], pushSecret=null, to=null, additionalProperties={}), postCommit=BuildPostCommitSpec(args=[], command=[], script=null, additionalProperties={}), resources=ResourceRequirements(limits=null, requests=null, additionalProperties={}), revision=null, serviceAccount=builder, source=BuildSource(binary=null, configMaps=[], contextDir=null, dockerfile=null, git=null, images=[], secrets=[], sourceSecret=null, type=None, additionalProperties={}), strategy=BuildStrategy(customStrategy=null, dockerStrategy=null, jenkinsPipelineStrategy=JenkinsPipelineBuildStrategy(env=[], jenkinsfile=properties([[$class: 'BuildConfigProjectProperty', name: '', namespace: '', resourceVersion: '', uid: ''],
    buildDiscarder(logRotator(artifactDaysToKeepStr: '', artifactNumToKeepStr: '',
        daysToKeepStr: '', numToKeepStr: '50'))]) ............

2022-01-20 10:21:23 INFO    io.fabric8.jenkins.openshiftsync.JenkinsUtils deleteRun Deleting run: dev99/dev99-aws-cloudformation-deploy-v2 #2
2022-01-20 10:21:23 INFO    io.fabric8.jenkins.openshiftsync.BuildSyncRunListener upsertBuild Setting build status values to: aws-cloudformation-deploy-v2-2:[ Failed ]: 2022-01-19T16:00:12Z->2022-01-19T16:00:41Z
2022-01-20 10:21:23 INFO    io.fabric8.jenkins.openshiftsync.BuildSyncRunListener onDeleted onDeleted job/dev99/job/dev99-aws-cloudformation-deploy-v2/2/
2022-01-20 10:21:23 INFO    io.fabric8.jenkins.openshiftsync.BuildInformer onDelete Build informer received delete event for: {}Build(apiVersion=build.openshift.io/v1, kind=Build, metadata=ObjectMeta(annotations={openshift.io/jenkins-blueocean-log-url=/https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/blue/organizations/jenkins/dev99%2Fdev99-aws-cloudformation-deploy-v2/detail/dev99-aws-cloudformation-deploy-v2/3/, openshift.io/jenkins-build-uri=https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/, openshift.io/jenkins-console-log-url=https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/console, openshift.io/jenkins-log-url=https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/consoleText, openshift.io/jenkins-status-json={"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/wfapi/describe"}},"id":"3","name":"#3","status":"FAILED","startTimeMillis":1642626534848,"endTimeMillis":1642626703364,"durationMillis":168516,"queueDurationMillis":3,"pauseDurationMillis":0,"stages":[{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/execution/node/16/wfapi/describe"}},"id":"16","name":"Input Parameters","execNode":"","status":"FAILED","error":{"message":"Cannot get property 'branch' on null object","type":"java.lang.NullPointerException"},"startTimeMillis":1642626565474,"durationMillis":137776,"pauseDurationMillis":117465,"stageFlowNodes":[{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/execution/node/17/wfapi/describe"}},"id":"17","name":"Internal utility function for OpenShift DSL","execNode":"","status":"SUCCESS","startTimeMillis":1642626565500,"durationMillis":23,"pauseDurationMillis":0,"parentNodes":["16"]},{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/execution/node/18/wfapi/describe"}},"id":"18","name":"Internal utility function for OpenShift DSL","execNode":"","status":"SUCCESS","startTimeMillis":1642626565523,"durationMillis":24,"pauseDurationMillis":0,"parentNodes":["17"]},{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/execution/node/19/wfapi/describe"}},"id":"19","name":"Git","execNode":"","status":"SUCCESS","startTimeMillis":1642626565547,"durationMillis":17971,"pauseDurationMillis":0,"parentNodes":["18"]},{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/execution/node/20/wfapi/describe"}},"id":"20","name":"Shell Script","execNode":"","status":"SUCCESS","parameterDescription":"oc get project --template='{{range .items}}{{.metadata.name}} {{end}}' > projectList","startTimeMillis":1642626583518,"durationMillis":1828,"pauseDurationMillis":0,"parentNodes":["19"]},{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/execution/node/21/wfapi/describe"}},"id":"21","name":"Read file from workspace","execNode":"","status":"SUCCESS","parameterDescription":"projectList","startTimeMillis":1642626585346,"durationMillis":49,"pauseDurationMillis":0,"parentNodes":["20"]},{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/execution/node/22/wfapi/describe"}},"id":"22","name":"Shell Script","execNode":"","status":"SUCCESS","parameterDescription":"\n                        git branch -r > projectListTmp\n                        sed 's/origin\\///g' projectListTmp > projectList\n                    ","startTimeMillis":1642626585395,"durationMillis":316,"pauseDurationMillis":0,"parentNodes":["21"]},{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/execution/node/23/wfapi/describe"}},"id":"23","name":"Read file from workspace","execNode":"","status":"SUCCESS","parameterDescription":"projectList","startTimeMillis":1642626585711,"durationMillis":22,"pauseDurationMillis":0,"parentNodes":["22"]},{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/execution/node/24/wfapi/describe"}},"id":"24","name":"Wait for interactive input","execNode":"","status":"SUCCESS","startTimeMillis":1642626585733,"durationMillis":117499,"pauseDurationMillis":117465,"parentNodes":["23"]}]}]}}, clusterName=null, creationTimestamp=2022-01-19T21:08:54Z, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=1, labels={buildconfig=aws-cloudformation-deploy-v2, openshift.io/build-config.name=aws-cloudformation-deploy-v2, openshift.io/build.start-policy=Serial, type=jenkins-pipeline}, managedFields=[], name=aws-cloudformation-deploy-v2-3, namespace=dev99, ownerReferences=[OwnerReference(apiVersion=build.openshift.io/v1, blockOwnerDeletion=null, controller=true, kind=BuildConfig, name=aws-cloudformation-deploy-v2, uid=a8ea6145-711a-40d7-9f53-8ac96b00d6a2, additionalProperties={})], resourceVersion=201174424, selfLink=null, uid=103bb0fd-b553-44c1-9d8c-7ac3f6467143, additionalProperties={}), spec=BuildSpec(completionDeadlineSeconds=null, nodeSelector=null, output=BuildOutput(imageLabels=[], pushSecret=null, to=null, additionalProperties={}), postCommit=BuildPostCommitSpec(args=[], command=[], script=null, additionalProperties={}), resources=ResourceRequirements(limits=null, requests=null, additionalProperties={}), revision=null, serviceAccount=builder, source=BuildSource(binary=null, configMaps=[], contextDir=null, dockerfile=null, git=null, images=[], secrets=[], sourceSecret=null, type=None, additionalProperties={}), strategy=BuildStrategy(customStrategy=null, dockerStrategy=null, jenkinsPipelineStrategy=JenkinsPipelineBuildStrategy(env=[], jenkinsfile=properties([[$class: 'BuildConfigProjectProperty', name: '', namespace: '', resourceVersion: '', uid: ''],
    buildDiscarder(logRotator(artifactDaysToKeepStr: '', artifactNumToKeepStr: '',
        daysToKeepStr: '', numToKeepStr: '50'))]).............

2022-01-20 10:21:23 INFO    io.fabric8.jenkins.openshiftsync.JenkinsUtils deleteRun Deleting run: dev99/dev99-aws-cloudformation-deploy-v2 #3
2022-01-20 10:21:23 INFO    io.fabric8.jenkins.openshiftsync.BuildSyncRunListener upsertBuild Setting build status values to: aws-cloudformation-deploy-v2-3:[ Failed ]: 2022-01-19T21:08:54Z->2022-01-19T21:11:43Z
2022-01-20 10:21:23 INFO    io.fabric8.jenkins.openshiftsync.BuildSyncRunListener onDeleted onDeleted job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/
2022-01-20 10:21:51 WARNING org.jenkinsci.plugins.prometheus.DiskUsageCollector collect Cannot get disk usage data. Install CloudBees Disk Usage Simple plugin to enable
2022-01-20 10:23:51 WARNING org.jenkinsci.plugins.prometheus.DiskUsageCollector collect Cannot get disk usage data. Install CloudBees Disk Usage Simple plugin to enable
2022-01-20 10:24:10 INFO    io.fabric8.jenkins.openshiftsync.BuildConfigInformer onUpdate BuildConfig informer received update event for: {} to: {}201170875 201170875
2022-01-20 10:24:10 INFO    io.fabric8.jenkins.openshiftsync.JobProcessor createOrUpdateJob Updated job dev99-aws-cloudformation-deploy-v2 from BuildConfig NamespaceName{dev99:aws-cloudformation-deploy-v2} with revision: 201170875
2022-01-20 10:25:51 WARNING org.jenkinsci.plugins.prometheus.DiskUsageCollector collect Cannot get disk usage data. Install CloudBees Disk Usage Simple plugin to enable
2022-01-20 10:27:51 WARNING org.jenkinsci.plugins.prometheus.DiskUsageCollector collect Cannot get disk usage data. Install CloudBees Disk Usage Simple plugin to enable
2022-01-20 10:29:10 INFO    io.fabric8.jenkins.openshiftsync.BuildConfigInformer onUpdate BuildConfig informer received update event for: {} to: {}201170875 201170875
2022-01-20 10:29:10 INFO    io.fabric8.jenkins.openshiftsync.JobProcessor createOrUpdateJob Updated job dev99-aws-cloudformation-deploy-v2 from BuildConfig NamespaceName{dev99:aws-cloudformation-deploy-v2} with revision: 201170875
2022-01-20 10:29:51 WARNING org.jenkinsci.plugins.prometheus.DiskUsageCollector collect Cannot get disk usage data. Install CloudBees Disk Usage Simple plugin to enable
2022-01-20 10:31:51 WARNING org.jenkinsci.plugins.prometheus.DiskUsageCollector collect Cannot get disk usage data. Install CloudBees Disk Usage Simple plugin to enable
2022-01-20 10:33:51 WARNING org.jenkinsci.plugins.prometheus.DiskUsageCollector collect Cannot get disk usage data. Install CloudBees Disk Usage Simple plugin to enable
2022-01-20 10:34:10 INFO    io.fabric8.jenkins.openshiftsync.BuildConfigInformer onUpdate BuildConfig informer received update event for: {} to: {}201170875 201170875
~~~

Customer is importing the pipelines from another Jenkins Instance that they have in OCP, I recommended them to create some pipelines manually in the Jenkins Dashboard instead of the imported one, the one created manually seems to be still there. They are importing the pipelines by running the following:

~~~
# Setup firstStepSetup job
curl "XYZrepos/openshift_jenkins/raw/FirstStepSetup.yml" -o /tmp/openshift/jenkins/FirstStepSetup.yml
oc apply -f /tmp/openshift/jenkins/FirstStepSetup.yml --namespace=$PROJECT_NAME_DEV

# Setup git secret
curl "XYZrepos/openshift_jenkins/raw/secrets/dev/jenkins-git.yml" -o /tmp/openshift/jenkins/git.yml
oc apply -f /tmp/openshift/jenkins/git.yml --namespace=$PROJECT_NAME_DEV

oc start-build first-step-setup --namespace=$PROJECT_NAME_DEV
~~~

Hope that information is enough, but please, feel free to ping me if needed.

Thank you very much! 

**
[1] https://coreos.slack.com/archives/CNU6JC8GJ/p1642524138011800

SalesForce case number --> 03094659


Additional info:
Comment 1 Adam Kaplan 2022-01-26 19:50:42 UTC 
This is potentially related to Bug #2040810.
Comment 6 Adam Kaplan 2022-04-26 12:52:05 UTC 
This potentially duplicates https://bugzilla.redhat.com/show_bug.cgi?id=2041888 - @Alice should be able to confirm.
Comment 8 Alice Rum 2022-05-17 08:08:07 UTC 
Seems to be duplicate of 2040810

This issue is fixed by openshift-sync 1.0.52, which is released as part of jenkins image in the latest OpenShift.

*** This bug has been marked as a duplicate of bug 2040810 ***