Bug 1685322
| Summary: | Builds delayed with multiple occurrences of the error "build_controller.go:1289] Giving up retrying <build name>: invalid phase transition <build name> (Running) -> Pending" | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Venkata Tadimarri <ktadimar> | ||||||||||||
| Component: | Build | Assignee: | Gabe Montero <gmontero> | ||||||||||||
| Status: | CLOSED ERRATA | QA Contact: | wewang <wewang> | ||||||||||||
| Severity: | medium | Docs Contact: | |||||||||||||
| Priority: | medium | ||||||||||||||
| Version: | 3.10.0 | CC: | aos-bugs, bparees, dsquirre, gmontero, ktadimar, mirollin, nstielau, pweil, rhowe, rkshirsa, wzheng | ||||||||||||
| Target Milestone: | --- | ||||||||||||||
| Target Release: | 3.10.z | ||||||||||||||
| Hardware: | All | ||||||||||||||
| OS: | Linux | ||||||||||||||
| Whiteboard: | |||||||||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||||||||
| Doc Text: |
Cause: slow progression through a builds init container steps could result in builds getting marked "Running" and then the build controller subsequently attempts to mark the build back to "Pending"
Consequence: the build will still eventually complete, but undo churn in build controller processing occurs and unneeded warning messages are presented to the user
Fix: the build controller now better prevents these erroneous transitions, and logs an more useful diagnostic event for edge/unexpected cases
Result: clean presentation to the user
|
Story Points: | --- | ||||||||||||
| Clone Of: | Environment: | ||||||||||||||
| Last Closed: | 2019-06-27 16:41:12 UTC | Type: | Bug | ||||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||||
| Documentation: | --- | CRM: | |||||||||||||
| Verified Versions: | Category: | --- | |||||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||
| Embargoed: | |||||||||||||||
| Attachments: |
|
||||||||||||||
|
Description
Venkata Tadimarri
2019-03-05 00:22:31 UTC
@Adam - attaching logs as requested. @Mitchell @Venkata based on the logs builds are able to complete. However, at some point there is a build whose state isn't able to be transitioned, and eventually the build controller gives up reporting state. This is causes the next build for the given BuildConfig to be delayed. As a work-around, the customer can cancel the individual builds that are generating the invalid state transition messages. If that fails to cancel, they can force the build to be deleted via `oc delete build <build-config>-<build-number>`. We will need additional time to investigate why the build is trying to transition from Running to Pending in the first place. PR https://github.com/openshift/origin/pull/22585 is marked for merging in the master/4.x branch, and cherry picks have been requested for 3.11 and 3.10. Will report back when the PRs for those cherrypick requests are up. Now tested in 3.10 3.11 and 4.1, error "invalid phase transition" already disappeared. $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.1.0-0.nightly-2019-05-06-011159 True False 20h Cluster version is 4.1.0-0.nightly-2019-05-06-011159 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1607 Created attachment 1594861 [details]
master-logs-controllers_02279152
Created attachment 1594862 [details]
master-logs_api.txt
Created attachment 1594863 [details]
journalctl_atomic-openshift.txt
Created attachment 1594864 [details]
master-logs_controllers.tx
Created attachment 1594865 [details]
events.txt
Yeah build_controller.go:1354 does not line up with the 3.11 version that has the fix. That log comes from line 1111 in the 3.11 version. And line 1058 in the 3.10 veersion. So the log from https://bugzilla.redhat.com/show_bug.cgi?id=1685322#c30 does not come from a system with the fix. |