Bug 1701334
| Summary: | Nova's setting and weighing of failed_builds is highly problematic | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | vivek koul <vkoul> |
| Component: | openstack-nova | Assignee: | OSP DFG:Compute <osp-dfg-compute> |
| Status: | CLOSED NOTABUG | QA Contact: | OSP DFG:Compute <osp-dfg-compute> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 13.0 (Queens) | CC: | astupnik, dasmith, ebarrera, eglynn, fpalin, gkadam, jhakimra, kchamart, lyarwood, msecaur, mwitt, rsafrono, sbauza, sgordon, vromanso |
| Target Milestone: | --- | Keywords: | Triaged |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-03-10 23:03:00 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
vivek koul
2019-04-18 16:21:53 UTC
*** Bug 1703110 has been marked as a duplicate of this bug. *** Apparently on master we're filtering some failed_build causes upstream, but it sounds like we should look at extending it. (In reply to Matthew Booth from comment #14) > Apparently on master we're filtering some failed_build causes upstream, but > it sounds like we should look at extending it. Turns out I was mistaken about this -- there is no such filtering on master either. I was thinking of a patch attempt from the past [1] that ended up abandoned because of the complexity and maintainability concerns about having such a whitelist. The recommended way to handle this issue in an affected environment is to disable the BuildFailureWeigher in config by setting the option: [filter_scheduler]build_failure_weight_multiplier = 0. [1] https://review.opendev.org/568953 (In reply to vivek koul from comment #16) > Hello, > > After restarting services my Cu is again seeing failed_builds for there > compute nodes. > I have suggested the Cu to disable the BuildFailureWeigher, but they want to > know the reason why they are experiencing build fails on hosts. > > So for that, I did some tests on my test env. > > I resized one of my test instances and it failed to get resize(I did that > intentionally). That particular instance got migrated to different compute > and it went into an error state. > So my question is should there be any failed_build for that compute node? For the scenario you describe, (intentionally cause a failed resize), it is expected there will be a failed_build for the related compute node. This is why the BuildFailureWeigher can be problematic, because it does not differentiate between user-caused build failures vs compute node-related build failures. Any situation where a request goes to a compute node and fails to build the instance (even a reschedule) will cause a failed_build to be tracked by the BuildFailureWeigher. The failed_build counter is reset (cleared out) for a compute node when any successful build occurs on that compute node. So, it does do some self-healing, but will still result in inconsistent instance placement if any build failures occur. If the customer environment requires a consistent placement of instances on compute nodes, it is best to disable the BuildFailureWeigher by setting [filter_scheduler]build_failure_weight_multiplier = 0. *** Bug 1705930 has been marked as a duplicate of this bug. *** *** Bug 1722201 has been marked as a duplicate of this bug. *** *** Bug 1728335 has been marked as a duplicate of this bug. *** The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days |