Bug 1543916
Summary: | Pipeline builds do not get pruned correctly | ||
---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Corey Daley <cdaley> |
Component: | RFE | Assignee: | Ben Parees <bparees> |
Status: | CLOSED CURRENTRELEASE | QA Contact: | Wenjing Zheng <wzheng> |
Severity: | medium | Docs Contact: | |
Priority: | medium | ||
Version: | 3.9.0 | CC: | aos-bugs, bparees, cdaley, jokerman, knakayam, mmccomas, wsun |
Target Milestone: | --- | ||
Target Release: | 3.11.0 | ||
Hardware: | x86_64 | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2018-12-21 15:16:27 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Corey Daley
2018-02-09 14:57:42 UTC
It looks like the builds never get pruned due to the following line(s): https://github.com/openshift/origin/blob/master/pkg/build/controller/build/build_controller.go#L329 https://github.com/openshift/origin/blob/master/pkg/build/controller/build/build_controller.go#L378-L381 Basically if the build strategy is JenkinsPipelineStrategy we are relying on Jenkins to do all creating/updating/deletion of the job. I believe the fix here is to update the openshift-client plugin to at least 3.0.0, and let Jenkins handle deleting the old builds based on the successfulBuildsHistoryLimit and failedBuildsHistoryLimit options in the BuildConfig, just to keep things consistent. I don't think the "shouldIgnore" logic should apply to the build pruning logic. We should prune pipeline builds. The shouldignore was really intended to say "ignore this build in that we aren't going to create a build pod for it and monitor the pod state." notes to self: as i recall the hard problem here is that if the controller on the openshift side deletes the build, then the jenkins sync plugin may say "hey i have this jenkins job run and there's no corresponding build object, let me create one". The solutions to that are either: 1) As corey suggested, have the sync plugin responsible for pruning pipeline builds 2) have the sync plugin be smart enough not to create a build object for job runs that are already completed (it may already be that smart, but that may also not be enough to close all the potential timing windows for getting this right..) definitely having only a single entity responsible for creating/deleting pipeline builds is "safer" though possibly harder to implement and results in us having code in two places) Corey if you have any other recollections around this, please add them. Ben, Since we have the sync plugin deleting job runs if the associated openshift build is deleted, I don't think that we have the issue with jenkins recreating the builds, so it seems like it would be safe to have OpenShift prune pipeline builds and then the Sync plugin would clean up the Jenkins jobs. Of course some tests should/would be created around this scenario. > Since we have the sync plugin deleting job runs if the associated openshift build is deleted, I don't think that we have the issue with jenkins recreating the builds
my fear is timing between the sync plugin seeing the delete event, and the sync plugin seeing the build is missing. I can (because i'm a pessimist) envision a case where a build is pruned, then a relist happens, the build is not in the list, the sync plugin starts a build for it, and then we see the delete event for the build and delete the job run.
We also need to update to the openshift-client 3.x which is being held up by https://github.com/fabric8io/kubernetes-client/issues/1046 The OpenShift client has now been updated to openshift-client 3.x This is actually implemented right Corey? Delivered in 3.11? Ben, Yes, I was just tracking down the commit for it to post here. This bug is fixed by https://github.com/openshift/origin/commit/37de5d244bf82e61e9d4d10bc913dbe44794a855 I'm going to throw this straight into ON_QA since i'm confident it's in a build at this point. Yes, pipeline build can be pruned now. Verified with below version: registry.dev.redhat.io/openshift3/jenkins-2-rhel7@sha256:8b9cc096eaa54eafe905c79ad0b7b43a31137c73a51e91e3ee8e10e6f22734a1 Closing bugs that were verified and targeted for GA but for some reason were not picked up by errata. This bug fix should be present in current 3.11 release content. |