Bug 2043084

Created attachment 1852213 [details]
Jenkins Master logs

Description of problem:

The customer is deploying Jenkins by using the permanent template:


~~~
oc new-app jenkins-persistent -p VOLUME_CAPACITY=30Gi --namespace=$PROJECT_NAME_DE
~~~

The current Jenkins version is 1.0.50.

Some resources are disappearing from time to time in their Jenkins dashboard, for example, we have created some credentials two days ago via the Jenkins dashboard but there are not in the cluster anymore. Some builds disappear from the Jenkins Dashboard even if they are still present as a BuildConfig in the OpenShift Console. As mentioned on the following Slack Thread[1] this could be due some prune configuration but we were not able to find anything related to this.

Attached are the full logs from the master pod at the time of the strange behaviour on the console, the most interesting part I've found is the following:

~~~
2022-01-19 02:38:14 INFO    io.fabric8.jenkins.openshiftsync.SecretInformer onDelete Secret informer received delete event for: jenkins-git
2022-01-19 02:38:14 INFO    io.fabric8.jenkins.openshiftsync.CredentialsUtils deleteCredential About to delete credential dev99-jenkins-gitwhich is referenced by jobs: dev99/dev99-aws-cloudformation-deploy-v2 
2022-01-19 02:38:14 INFO    io.fabric8.jenkins.openshiftsync.CredentialsUtils 
~~~

It seems that something is removing the credentials, not sure if that is the root cause here.

If they trigger a BuildConfig in the OCP console, it appears again on the Jenkins Dashboard, so the pipelines are still present on the OCP. 

I've tried to troubleshoot as much as possible but I don't know how to continue or if this is some kind of bug. I'd like to help/contribute as much as possible on this particular case, so if someone needs something from my side or the customer side, please, let us know.

For example, currently there is a build created via OpenShift template is still there but the history is gone(this could be related to the prune?) here is the log at the moment of the deletion:

~~~
2022-01-20 10:19:10 INFO    io.fabric8.jenkins.openshiftsync.JobProcessor createOrUpdateJob Updated job dev99-aws-cloudformation-deploy-v2 from BuildConfig NamespaceName{dev99:aws-cloudformation-deploy-v2} with revision: 201170875
2022-01-20 10:19:27 INFO    io.fabric8.jenkins.openshiftsync.BuildInformer onUpdate Build informer received update event for: {} to: {}200799529 200799529
2022-01-20 10:19:27 INFO    io.fabric8.jenkins.openshiftsync.BuildInformer onUpdate Build informer received update event for: {} to: {}201174424 201174424
2022-01-20 10:19:51 WARNING org.jenkinsci.plugins.prometheus.DiskUsageCollector collect Cannot get disk usage data. Install CloudBees Disk Usage Simple plugin to enable
2022-01-20 10:21:23 WARNING io.fabric8.kubernetes.client.informers.cache.ReflectorWatcher onClose Watch closing with exception
io.fabric8.kubernetes.client.WatcherException: too old resource version: 201174424 (201248022)
    at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$TypedWatcherWebSocketListener.onMessage(WatchConnectionManager.java:103)
    at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:323)
    at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219)
    at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105)
    at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
    at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
    at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
    at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: too old resource version: 201174424 (201248022)
    ... 11 more

2022-01-20 10:21:23 INFO    io.fabric8.jenkins.openshiftsync.BuildInformer onDelete Build informer received delete event for: {}Build(apiVersion=build.openshift.io/v1, kind=Build, metadata=ObjectMeta(annotations={openshift.io/jenkins-blueocean-log-url=/https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/blue/organizations/jenkins/dev99%2Fdev99-aws-cloudformation-deploy-v2/detail/dev99-aws-cloudformation-deploy-v2/2/, openshift.io/jenkins-build-uri=https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/2/, openshift.io/jenkins-console-log-url=https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/2/console, openshift.io/jenkins-log-url=https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/2/consoleText, openshift.io/jenkins-status-json={"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/2/wfapi/describe"}},"id":"2","name":"#2","status":"FAILED","startTimeMillis":1642608012654,"endTimeMillis":1642608041799,"durationMillis":29145,"queueDurationMillis":23,"pauseDurationMillis":0,"stages":[{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/2/execution/node/16/wfapi/describe"}},"id":"16","name":"Input Parameters","execNode":"","status":"FAILED","error":{"message":"Error cloning remote repo 'origin'","type":"hudson.AbortException"},"startTimeMillis":1642608036006,"durationMillis":5291,"pauseDurationMillis":0,"stageFlowNodes":[{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/2/execution/node/17/wfapi/describe"}},"id":"17","name":"Internal utility function for OpenShift DSL","execNode":"","status":"SUCCESS","startTimeMillis":1642608036034,"durationMillis":26,"pauseDurationMillis":0,"parentNodes":["16"]},{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/2/execution/node/18/wfapi/describe"}},"id":"18","name":"Internal utility function for OpenShift DSL","execNode":"","status":"SUCCESS","startTimeMillis":1642608036060,"durationMillis":58,"pauseDurationMillis":0,"parentNodes":["17"]},{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/2/execution/node/19/wfapi/describe"}},"id":"19","name":"Git","execNode":"","status":"FAILED","error":{"message":"Error cloning remote repo 'origin'","type":"hudson.AbortException"},"startTimeMillis":1642608036118,"durationMillis":5152,"pauseDurationMillis":0,"parentNodes":["18"]}]}]}}, clusterName=null, creationTimestamp=2022-01-19T16:00:12Z, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=1, labels={buildconfig=aws-cloudformation-deploy-v2, openshift.io/build-config.name=aws-cloudformation-deploy-v2, openshift.io/build.start-policy=Serial, type=jenkins-pipeline}, managedFields=[], name=aws-cloudformation-deploy-v2-2, namespace=dev99, ownerReferences=[OwnerReference(apiVersion=build.openshift.io/v1, blockOwnerDeletion=null, controller=true, kind=BuildConfig, name=aws-cloudformation-deploy-v2, uid=a8ea6145-711a-40d7-9f53-8ac96b00d6a2, additionalProperties={})], resourceVersion=200799529, selfLink=null, uid=00839659-dde3-4b22-bffe-897641620d35, additionalProperties={}), spec=BuildSpec(completionDeadlineSeconds=null, nodeSelector=null, output=BuildOutput(imageLabels=[], pushSecret=null, to=null, additionalProperties={}), postCommit=BuildPostCommitSpec(args=[], command=[], script=null, additionalProperties={}), resources=ResourceRequirements(limits=null, requests=null, additionalProperties={}), revision=null, serviceAccount=builder, source=BuildSource(binary=null, configMaps=[], contextDir=null, dockerfile=null, git=null, images=[], secrets=[], sourceSecret=null, type=None, additionalProperties={}), strategy=BuildStrategy(customStrategy=null, dockerStrategy=null, jenkinsPipelineStrategy=JenkinsPipelineBuildStrategy(env=[], jenkinsfile=properties([[$class: 'BuildConfigProjectProperty', name: '', namespace: '', resourceVersion: '', uid: ''],
    buildDiscarder(logRotator(artifactDaysToKeepStr: '', artifactNumToKeepStr: '',
        daysToKeepStr: '', numToKeepStr: '50'))]) ............

2022-01-20 10:21:23 INFO    io.fabric8.jenkins.openshiftsync.JenkinsUtils deleteRun Deleting run: dev99/dev99-aws-cloudformation-deploy-v2 #2
2022-01-20 10:21:23 INFO    io.fabric8.jenkins.openshiftsync.BuildSyncRunListener upsertBuild Setting build status values to: aws-cloudformation-deploy-v2-2:[ Failed ]: 2022-01-19T16:00:12Z->2022-01-19T16:00:41Z
2022-01-20 10:21:23 INFO    io.fabric8.jenkins.openshiftsync.BuildSyncRunListener onDeleted onDeleted job/dev99/job/dev99-aws-cloudformation-deploy-v2/2/
2022-01-20 10:21:23 INFO    io.fabric8.jenkins.openshiftsync.BuildInformer onDelete Build informer received delete event for: {}Build(apiVersion=build.openshift.io/v1, kind=Build, metadata=ObjectMeta(annotations={openshift.io/jenkins-blueocean-log-url=/https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/blue/organizations/jenkins/dev99%2Fdev99-aws-cloudformation-deploy-v2/detail/dev99-aws-cloudformation-deploy-v2/3/, openshift.io/jenkins-build-uri=https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/, openshift.io/jenkins-console-log-url=https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/console, openshift.io/jenkins-log-url=https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/consoleText, openshift.io/jenkins-status-json={"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/wfapi/describe"}},"id":"3","name":"#3","status":"FAILED","startTimeMillis":1642626534848,"endTimeMillis":1642626703364,"durationMillis":168516,"queueDurationMillis":3,"pauseDurationMillis":0,"stages":[{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/execution/node/16/wfapi/describe"}},"id":"16","name":"Input Parameters","execNode":"","status":"FAILED","error":{"message":"Cannot get property 'branch' on null object","type":"java.lang.NullPointerException"},"startTimeMillis":1642626565474,"durationMillis":137776,"pauseDurationMillis":117465,"stageFlowNodes":[{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/execution/node/17/wfapi/describe"}},"id":"17","name":"Internal utility function for OpenShift DSL","execNode":"","status":"SUCCESS","startTimeMillis":1642626565500,"durationMillis":23,"pauseDurationMillis":0,"parentNodes":["16"]},{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/execution/node/18/wfapi/describe"}},"id":"18","name":"Internal utility function for OpenShift DSL","execNode":"","status":"SUCCESS","startTimeMillis":1642626565523,"durationMillis":24,"pauseDurationMillis":0,"parentNodes":["17"]},{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/execution/node/19/wfapi/describe"}},"id":"19","name":"Git","execNode":"","status":"SUCCESS","startTimeMillis":1642626565547,"durationMillis":17971,"pauseDurationMillis":0,"parentNodes":["18"]},{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/execution/node/20/wfapi/describe"}},"id":"20","name":"Shell Script","execNode":"","status":"SUCCESS","parameterDescription":"oc get project --template='{{range .items}}{{.metadata.name}} {{end}}' > projectList","startTimeMillis":1642626583518,"durationMillis":1828,"pauseDurationMillis":0,"parentNodes":["19"]},{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/execution/node/21/wfapi/describe"}},"id":"21","name":"Read file from workspace","execNode":"","status":"SUCCESS","parameterDescription":"projectList","startTimeMillis":1642626585346,"durationMillis":49,"pauseDurationMillis":0,"parentNodes":["20"]},{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/execution/node/22/wfapi/describe"}},"id":"22","name":"Shell Script","execNode":"","status":"SUCCESS","parameterDescription":"\n                        git branch -r > projectListTmp\n                        sed 's/origin\\///g' projectListTmp > projectList\n                    ","startTimeMillis":1642626585395,"durationMillis":316,"pauseDurationMillis":0,"parentNodes":["21"]},{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/execution/node/23/wfapi/describe"}},"id":"23","name":"Read file from workspace","execNode":"","status":"SUCCESS","parameterDescription":"projectList","startTimeMillis":1642626585711,"durationMillis":22,"pauseDurationMillis":0,"parentNodes":["22"]},{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/execution/node/24/wfapi/describe"}},"id":"24","name":"Wait for interactive input","execNode":"","status":"SUCCESS","startTimeMillis":1642626585733,"durationMillis":117499,"pauseDurationMillis":117465,"parentNodes":["23"]}]}]}}, clusterName=null, creationTimestamp=2022-01-19T21:08:54Z, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=1, labels={buildconfig=aws-cloudformation-deploy-v2, openshift.io/build-config.name=aws-cloudformation-deploy-v2, openshift.io/build.start-policy=Serial, type=jenkins-pipeline}, managedFields=[], name=aws-cloudformation-deploy-v2-3, namespace=dev99, ownerReferences=[OwnerReference(apiVersion=build.openshift.io/v1, blockOwnerDeletion=null, controller=true, kind=BuildConfig, name=aws-cloudformation-deploy-v2, uid=a8ea6145-711a-40d7-9f53-8ac96b00d6a2, additionalProperties={})], resourceVersion=201174424, selfLink=null, uid=103bb0fd-b553-44c1-9d8c-7ac3f6467143, additionalProperties={}), spec=BuildSpec(completionDeadlineSeconds=null, nodeSelector=null, output=BuildOutput(imageLabels=[], pushSecret=null, to=null, additionalProperties={}), postCommit=BuildPostCommitSpec(args=[], command=[], script=null, additionalProperties={}), resources=ResourceRequirements(limits=null, requests=null, additionalProperties={}), revision=null, serviceAccount=builder, source=BuildSource(binary=null, configMaps=[], contextDir=null, dockerfile=null, git=null, images=[], secrets=[], sourceSecret=null, type=None, additionalProperties={}), strategy=BuildStrategy(customStrategy=null, dockerStrategy=null, jenkinsPipelineStrategy=JenkinsPipelineBuildStrategy(env=[], jenkinsfile=properties([[$class: 'BuildConfigProjectProperty', name: '', namespace: '', resourceVersion: '', uid: ''],
    buildDiscarder(logRotator(artifactDaysToKeepStr: '', artifactNumToKeepStr: '',
        daysToKeepStr: '', numToKeepStr: '50'))]).............

2022-01-20 10:21:23 INFO    io.fabric8.jenkins.openshiftsync.JenkinsUtils deleteRun Deleting run: dev99/dev99-aws-cloudformation-deploy-v2 #3
2022-01-20 10:21:23 INFO    io.fabric8.jenkins.openshiftsync.BuildSyncRunListener upsertBuild Setting build status values to: aws-cloudformation-deploy-v2-3:[ Failed ]: 2022-01-19T21:08:54Z->2022-01-19T21:11:43Z
2022-01-20 10:21:23 INFO    io.fabric8.jenkins.openshiftsync.BuildSyncRunListener onDeleted onDeleted job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/
2022-01-20 10:21:51 WARNING org.jenkinsci.plugins.prometheus.DiskUsageCollector collect Cannot get disk usage data. Install CloudBees Disk Usage Simple plugin to enable
2022-01-20 10:23:51 WARNING org.jenkinsci.plugins.prometheus.DiskUsageCollector collect Cannot get disk usage data. Install CloudBees Disk Usage Simple plugin to enable
2022-01-20 10:24:10 INFO    io.fabric8.jenkins.openshiftsync.BuildConfigInformer onUpdate BuildConfig informer received update event for: {} to: {}201170875 201170875
2022-01-20 10:24:10 INFO    io.fabric8.jenkins.openshiftsync.JobProcessor createOrUpdateJob Updated job dev99-aws-cloudformation-deploy-v2 from BuildConfig NamespaceName{dev99:aws-cloudformation-deploy-v2} with revision: 201170875
2022-01-20 10:25:51 WARNING org.jenkinsci.plugins.prometheus.DiskUsageCollector collect Cannot get disk usage data. Install CloudBees Disk Usage Simple plugin to enable
2022-01-20 10:27:51 WARNING org.jenkinsci.plugins.prometheus.DiskUsageCollector collect Cannot get disk usage data. Install CloudBees Disk Usage Simple plugin to enable
2022-01-20 10:29:10 INFO    io.fabric8.jenkins.openshiftsync.BuildConfigInformer onUpdate BuildConfig informer received update event for: {} to: {}201170875 201170875
2022-01-20 10:29:10 INFO    io.fabric8.jenkins.openshiftsync.JobProcessor createOrUpdateJob Updated job dev99-aws-cloudformation-deploy-v2 from BuildConfig NamespaceName{dev99:aws-cloudformation-deploy-v2} with revision: 201170875
2022-01-20 10:29:51 WARNING org.jenkinsci.plugins.prometheus.DiskUsageCollector collect Cannot get disk usage data. Install CloudBees Disk Usage Simple plugin to enable
2022-01-20 10:31:51 WARNING org.jenkinsci.plugins.prometheus.DiskUsageCollector collect Cannot get disk usage data. Install CloudBees Disk Usage Simple plugin to enable
2022-01-20 10:33:51 WARNING org.jenkinsci.plugins.prometheus.DiskUsageCollector collect Cannot get disk usage data. Install CloudBees Disk Usage Simple plugin to enable
2022-01-20 10:34:10 INFO    io.fabric8.jenkins.openshiftsync.BuildConfigInformer onUpdate BuildConfig informer received update event for: {} to: {}201170875 201170875
~~~

Customer is importing the pipelines from another Jenkins Instance that they have in OCP, I recommended them to create some pipelines manually in the Jenkins Dashboard instead of the imported one, the one created manually seems to be still there. They are importing the pipelines by running the following:

~~~
# Setup firstStepSetup job
curl "XYZrepos/openshift_jenkins/raw/FirstStepSetup.yml" -o /tmp/openshift/jenkins/FirstStepSetup.yml
oc apply -f /tmp/openshift/jenkins/FirstStepSetup.yml --namespace=$PROJECT_NAME_DEV

# Setup git secret
curl "XYZrepos/openshift_jenkins/raw/secrets/dev/jenkins-git.yml" -o /tmp/openshift/jenkins/git.yml
oc apply -f /tmp/openshift/jenkins/git.yml --namespace=$PROJECT_NAME_DEV

oc start-build first-step-setup --namespace=$PROJECT_NAME_DEV
~~~

Hope that information is enough, but please, feel free to ping me if needed.

Thank you very much! 

**
[1] https://coreos.slack.com/archives/CNU6JC8GJ/p1642524138011800

SalesForce case number --> 03094659


Additional info: