Created attachment 1852213 [details] Jenkins Master logs Description of problem: The customer is deploying Jenkins by using the permanent template: ~~~ oc new-app jenkins-persistent -p VOLUME_CAPACITY=30Gi --namespace=$PROJECT_NAME_DE ~~~ The current Jenkins version is 1.0.50. Some resources are disappearing from time to time in their Jenkins dashboard, for example, we have created some credentials two days ago via the Jenkins dashboard but there are not in the cluster anymore. Some builds disappear from the Jenkins Dashboard even if they are still present as a BuildConfig in the OpenShift Console. As mentioned on the following Slack Thread[1] this could be due some prune configuration but we were not able to find anything related to this. Attached are the full logs from the master pod at the time of the strange behaviour on the console, the most interesting part I've found is the following: ~~~ 2022-01-19 02:38:14 INFO io.fabric8.jenkins.openshiftsync.SecretInformer onDelete Secret informer received delete event for: jenkins-git 2022-01-19 02:38:14 INFO io.fabric8.jenkins.openshiftsync.CredentialsUtils deleteCredential About to delete credential dev99-jenkins-gitwhich is referenced by jobs: dev99/dev99-aws-cloudformation-deploy-v2 2022-01-19 02:38:14 INFO io.fabric8.jenkins.openshiftsync.CredentialsUtils ~~~ It seems that something is removing the credentials, not sure if that is the root cause here. If they trigger a BuildConfig in the OCP console, it appears again on the Jenkins Dashboard, so the pipelines are still present on the OCP. I've tried to troubleshoot as much as possible but I don't know how to continue or if this is some kind of bug. I'd like to help/contribute as much as possible on this particular case, so if someone needs something from my side or the customer side, please, let us know. For example, currently there is a build created via OpenShift template is still there but the history is gone(this could be related to the prune?) here is the log at the moment of the deletion: ~~~ 2022-01-20 10:19:10 INFO io.fabric8.jenkins.openshiftsync.JobProcessor createOrUpdateJob Updated job dev99-aws-cloudformation-deploy-v2 from BuildConfig NamespaceName{dev99:aws-cloudformation-deploy-v2} with revision: 201170875 2022-01-20 10:19:27 INFO io.fabric8.jenkins.openshiftsync.BuildInformer onUpdate Build informer received update event for: {} to: {}200799529 200799529 2022-01-20 10:19:27 INFO io.fabric8.jenkins.openshiftsync.BuildInformer onUpdate Build informer received update event for: {} to: {}201174424 201174424 2022-01-20 10:19:51 WARNING org.jenkinsci.plugins.prometheus.DiskUsageCollector collect Cannot get disk usage data. Install CloudBees Disk Usage Simple plugin to enable 2022-01-20 10:21:23 WARNING io.fabric8.kubernetes.client.informers.cache.ReflectorWatcher onClose Watch closing with exception io.fabric8.kubernetes.client.WatcherException: too old resource version: 201174424 (201248022) at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$TypedWatcherWebSocketListener.onMessage(WatchConnectionManager.java:103) at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:323) at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219) at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105) at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274) at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214) at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: io.fabric8.kubernetes.client.KubernetesClientException: too old resource version: 201174424 (201248022) ... 11 more 2022-01-20 10:21:23 INFO io.fabric8.jenkins.openshiftsync.BuildInformer onDelete Build informer received delete event for: {}Build(apiVersion=build.openshift.io/v1, kind=Build, metadata=ObjectMeta(annotations={openshift.io/jenkins-blueocean-log-url=/https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/blue/organizations/jenkins/dev99%2Fdev99-aws-cloudformation-deploy-v2/detail/dev99-aws-cloudformation-deploy-v2/2/, openshift.io/jenkins-build-uri=https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/2/, openshift.io/jenkins-console-log-url=https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/2/console, openshift.io/jenkins-log-url=https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/2/consoleText, openshift.io/jenkins-status-json={"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/2/wfapi/describe"}},"id":"2","name":"#2","status":"FAILED","startTimeMillis":1642608012654,"endTimeMillis":1642608041799,"durationMillis":29145,"queueDurationMillis":23,"pauseDurationMillis":0,"stages":[{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/2/execution/node/16/wfapi/describe"}},"id":"16","name":"Input Parameters","execNode":"","status":"FAILED","error":{"message":"Error cloning remote repo 'origin'","type":"hudson.AbortException"},"startTimeMillis":1642608036006,"durationMillis":5291,"pauseDurationMillis":0,"stageFlowNodes":[{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/2/execution/node/17/wfapi/describe"}},"id":"17","name":"Internal utility function for OpenShift DSL","execNode":"","status":"SUCCESS","startTimeMillis":1642608036034,"durationMillis":26,"pauseDurationMillis":0,"parentNodes":["16"]},{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/2/execution/node/18/wfapi/describe"}},"id":"18","name":"Internal utility function for OpenShift DSL","execNode":"","status":"SUCCESS","startTimeMillis":1642608036060,"durationMillis":58,"pauseDurationMillis":0,"parentNodes":["17"]},{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/2/execution/node/19/wfapi/describe"}},"id":"19","name":"Git","execNode":"","status":"FAILED","error":{"message":"Error cloning remote repo 'origin'","type":"hudson.AbortException"},"startTimeMillis":1642608036118,"durationMillis":5152,"pauseDurationMillis":0,"parentNodes":["18"]}]}]}}, clusterName=null, creationTimestamp=2022-01-19T16:00:12Z, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=1, labels={buildconfig=aws-cloudformation-deploy-v2, openshift.io/build-config.name=aws-cloudformation-deploy-v2, openshift.io/build.start-policy=Serial, type=jenkins-pipeline}, managedFields=[], name=aws-cloudformation-deploy-v2-2, namespace=dev99, ownerReferences=[OwnerReference(apiVersion=build.openshift.io/v1, blockOwnerDeletion=null, controller=true, kind=BuildConfig, name=aws-cloudformation-deploy-v2, uid=a8ea6145-711a-40d7-9f53-8ac96b00d6a2, additionalProperties={})], resourceVersion=200799529, selfLink=null, uid=00839659-dde3-4b22-bffe-897641620d35, additionalProperties={}), spec=BuildSpec(completionDeadlineSeconds=null, nodeSelector=null, output=BuildOutput(imageLabels=[], pushSecret=null, to=null, additionalProperties={}), postCommit=BuildPostCommitSpec(args=[], command=[], script=null, additionalProperties={}), resources=ResourceRequirements(limits=null, requests=null, additionalProperties={}), revision=null, serviceAccount=builder, source=BuildSource(binary=null, configMaps=[], contextDir=null, dockerfile=null, git=null, images=[], secrets=[], sourceSecret=null, type=None, additionalProperties={}), strategy=BuildStrategy(customStrategy=null, dockerStrategy=null, jenkinsPipelineStrategy=JenkinsPipelineBuildStrategy(env=[], jenkinsfile=properties([[$class: 'BuildConfigProjectProperty', name: '', namespace: '', resourceVersion: '', uid: ''], buildDiscarder(logRotator(artifactDaysToKeepStr: '', artifactNumToKeepStr: '', daysToKeepStr: '', numToKeepStr: '50'))]) ............ 2022-01-20 10:21:23 INFO io.fabric8.jenkins.openshiftsync.JenkinsUtils deleteRun Deleting run: dev99/dev99-aws-cloudformation-deploy-v2 #2 2022-01-20 10:21:23 INFO io.fabric8.jenkins.openshiftsync.BuildSyncRunListener upsertBuild Setting build status values to: aws-cloudformation-deploy-v2-2:[ Failed ]: 2022-01-19T16:00:12Z->2022-01-19T16:00:41Z 2022-01-20 10:21:23 INFO io.fabric8.jenkins.openshiftsync.BuildSyncRunListener onDeleted onDeleted job/dev99/job/dev99-aws-cloudformation-deploy-v2/2/ 2022-01-20 10:21:23 INFO io.fabric8.jenkins.openshiftsync.BuildInformer onDelete Build informer received delete event for: {}Build(apiVersion=build.openshift.io/v1, kind=Build, metadata=ObjectMeta(annotations={openshift.io/jenkins-blueocean-log-url=/https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/blue/organizations/jenkins/dev99%2Fdev99-aws-cloudformation-deploy-v2/detail/dev99-aws-cloudformation-deploy-v2/3/, openshift.io/jenkins-build-uri=https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/, openshift.io/jenkins-console-log-url=https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/console, openshift.io/jenkins-log-url=https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/consoleText, openshift.io/jenkins-status-json={"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/wfapi/describe"}},"id":"3","name":"#3","status":"FAILED","startTimeMillis":1642626534848,"endTimeMillis":1642626703364,"durationMillis":168516,"queueDurationMillis":3,"pauseDurationMillis":0,"stages":[{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/execution/node/16/wfapi/describe"}},"id":"16","name":"Input Parameters","execNode":"","status":"FAILED","error":{"message":"Cannot get property 'branch' on null object","type":"java.lang.NullPointerException"},"startTimeMillis":1642626565474,"durationMillis":137776,"pauseDurationMillis":117465,"stageFlowNodes":[{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/execution/node/17/wfapi/describe"}},"id":"17","name":"Internal utility function for OpenShift DSL","execNode":"","status":"SUCCESS","startTimeMillis":1642626565500,"durationMillis":23,"pauseDurationMillis":0,"parentNodes":["16"]},{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/execution/node/18/wfapi/describe"}},"id":"18","name":"Internal utility function for OpenShift DSL","execNode":"","status":"SUCCESS","startTimeMillis":1642626565523,"durationMillis":24,"pauseDurationMillis":0,"parentNodes":["17"]},{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/execution/node/19/wfapi/describe"}},"id":"19","name":"Git","execNode":"","status":"SUCCESS","startTimeMillis":1642626565547,"durationMillis":17971,"pauseDurationMillis":0,"parentNodes":["18"]},{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/execution/node/20/wfapi/describe"}},"id":"20","name":"Shell Script","execNode":"","status":"SUCCESS","parameterDescription":"oc get project --template='{{range .items}}{{.metadata.name}} {{end}}' > projectList","startTimeMillis":1642626583518,"durationMillis":1828,"pauseDurationMillis":0,"parentNodes":["19"]},{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/execution/node/21/wfapi/describe"}},"id":"21","name":"Read file from workspace","execNode":"","status":"SUCCESS","parameterDescription":"projectList","startTimeMillis":1642626585346,"durationMillis":49,"pauseDurationMillis":0,"parentNodes":["20"]},{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/execution/node/22/wfapi/describe"}},"id":"22","name":"Shell Script","execNode":"","status":"SUCCESS","parameterDescription":"\n git branch -r > projectListTmp\n sed 's/origin\\///g' projectListTmp > projectList\n ","startTimeMillis":1642626585395,"durationMillis":316,"pauseDurationMillis":0,"parentNodes":["21"]},{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/execution/node/23/wfapi/describe"}},"id":"23","name":"Read file from workspace","execNode":"","status":"SUCCESS","parameterDescription":"projectList","startTimeMillis":1642626585711,"durationMillis":22,"pauseDurationMillis":0,"parentNodes":["22"]},{"_links":{"self":{"href":"https://jenkins-dev99.apps.lfs-dev.bi8s.p1.openshiftapps.com/job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/execution/node/24/wfapi/describe"}},"id":"24","name":"Wait for interactive input","execNode":"","status":"SUCCESS","startTimeMillis":1642626585733,"durationMillis":117499,"pauseDurationMillis":117465,"parentNodes":["23"]}]}]}}, clusterName=null, creationTimestamp=2022-01-19T21:08:54Z, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=1, labels={buildconfig=aws-cloudformation-deploy-v2, openshift.io/build-config.name=aws-cloudformation-deploy-v2, openshift.io/build.start-policy=Serial, type=jenkins-pipeline}, managedFields=[], name=aws-cloudformation-deploy-v2-3, namespace=dev99, ownerReferences=[OwnerReference(apiVersion=build.openshift.io/v1, blockOwnerDeletion=null, controller=true, kind=BuildConfig, name=aws-cloudformation-deploy-v2, uid=a8ea6145-711a-40d7-9f53-8ac96b00d6a2, additionalProperties={})], resourceVersion=201174424, selfLink=null, uid=103bb0fd-b553-44c1-9d8c-7ac3f6467143, additionalProperties={}), spec=BuildSpec(completionDeadlineSeconds=null, nodeSelector=null, output=BuildOutput(imageLabels=[], pushSecret=null, to=null, additionalProperties={}), postCommit=BuildPostCommitSpec(args=[], command=[], script=null, additionalProperties={}), resources=ResourceRequirements(limits=null, requests=null, additionalProperties={}), revision=null, serviceAccount=builder, source=BuildSource(binary=null, configMaps=[], contextDir=null, dockerfile=null, git=null, images=[], secrets=[], sourceSecret=null, type=None, additionalProperties={}), strategy=BuildStrategy(customStrategy=null, dockerStrategy=null, jenkinsPipelineStrategy=JenkinsPipelineBuildStrategy(env=[], jenkinsfile=properties([[$class: 'BuildConfigProjectProperty', name: '', namespace: '', resourceVersion: '', uid: ''], buildDiscarder(logRotator(artifactDaysToKeepStr: '', artifactNumToKeepStr: '', daysToKeepStr: '', numToKeepStr: '50'))])............. 2022-01-20 10:21:23 INFO io.fabric8.jenkins.openshiftsync.JenkinsUtils deleteRun Deleting run: dev99/dev99-aws-cloudformation-deploy-v2 #3 2022-01-20 10:21:23 INFO io.fabric8.jenkins.openshiftsync.BuildSyncRunListener upsertBuild Setting build status values to: aws-cloudformation-deploy-v2-3:[ Failed ]: 2022-01-19T21:08:54Z->2022-01-19T21:11:43Z 2022-01-20 10:21:23 INFO io.fabric8.jenkins.openshiftsync.BuildSyncRunListener onDeleted onDeleted job/dev99/job/dev99-aws-cloudformation-deploy-v2/3/ 2022-01-20 10:21:51 WARNING org.jenkinsci.plugins.prometheus.DiskUsageCollector collect Cannot get disk usage data. Install CloudBees Disk Usage Simple plugin to enable 2022-01-20 10:23:51 WARNING org.jenkinsci.plugins.prometheus.DiskUsageCollector collect Cannot get disk usage data. Install CloudBees Disk Usage Simple plugin to enable 2022-01-20 10:24:10 INFO io.fabric8.jenkins.openshiftsync.BuildConfigInformer onUpdate BuildConfig informer received update event for: {} to: {}201170875 201170875 2022-01-20 10:24:10 INFO io.fabric8.jenkins.openshiftsync.JobProcessor createOrUpdateJob Updated job dev99-aws-cloudformation-deploy-v2 from BuildConfig NamespaceName{dev99:aws-cloudformation-deploy-v2} with revision: 201170875 2022-01-20 10:25:51 WARNING org.jenkinsci.plugins.prometheus.DiskUsageCollector collect Cannot get disk usage data. Install CloudBees Disk Usage Simple plugin to enable 2022-01-20 10:27:51 WARNING org.jenkinsci.plugins.prometheus.DiskUsageCollector collect Cannot get disk usage data. Install CloudBees Disk Usage Simple plugin to enable 2022-01-20 10:29:10 INFO io.fabric8.jenkins.openshiftsync.BuildConfigInformer onUpdate BuildConfig informer received update event for: {} to: {}201170875 201170875 2022-01-20 10:29:10 INFO io.fabric8.jenkins.openshiftsync.JobProcessor createOrUpdateJob Updated job dev99-aws-cloudformation-deploy-v2 from BuildConfig NamespaceName{dev99:aws-cloudformation-deploy-v2} with revision: 201170875 2022-01-20 10:29:51 WARNING org.jenkinsci.plugins.prometheus.DiskUsageCollector collect Cannot get disk usage data. Install CloudBees Disk Usage Simple plugin to enable 2022-01-20 10:31:51 WARNING org.jenkinsci.plugins.prometheus.DiskUsageCollector collect Cannot get disk usage data. Install CloudBees Disk Usage Simple plugin to enable 2022-01-20 10:33:51 WARNING org.jenkinsci.plugins.prometheus.DiskUsageCollector collect Cannot get disk usage data. Install CloudBees Disk Usage Simple plugin to enable 2022-01-20 10:34:10 INFO io.fabric8.jenkins.openshiftsync.BuildConfigInformer onUpdate BuildConfig informer received update event for: {} to: {}201170875 201170875 ~~~ Customer is importing the pipelines from another Jenkins Instance that they have in OCP, I recommended them to create some pipelines manually in the Jenkins Dashboard instead of the imported one, the one created manually seems to be still there. They are importing the pipelines by running the following: ~~~ # Setup firstStepSetup job curl "XYZrepos/openshift_jenkins/raw/FirstStepSetup.yml" -o /tmp/openshift/jenkins/FirstStepSetup.yml oc apply -f /tmp/openshift/jenkins/FirstStepSetup.yml --namespace=$PROJECT_NAME_DEV # Setup git secret curl "XYZrepos/openshift_jenkins/raw/secrets/dev/jenkins-git.yml" -o /tmp/openshift/jenkins/git.yml oc apply -f /tmp/openshift/jenkins/git.yml --namespace=$PROJECT_NAME_DEV oc start-build first-step-setup --namespace=$PROJECT_NAME_DEV ~~~ Hope that information is enough, but please, feel free to ping me if needed. Thank you very much! ** [1] https://coreos.slack.com/archives/CNU6JC8GJ/p1642524138011800 SalesForce case number --> 03094659 Additional info:
This is potentially related to Bug #2040810.
This potentially duplicates https://bugzilla.redhat.com/show_bug.cgi?id=2041888 - @Alice should be able to confirm.
Seems to be duplicate of 2040810 This issue is fixed by openshift-sync 1.0.52, which is released as part of jenkins image in the latest OpenShift. *** This bug has been marked as a duplicate of bug 2040810 ***