Bug 1841358
| Summary: | MCO does not update kubelet.conf if KubeletConfig created before MCP | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Denys Shchedrivyi <dshchedr> |
| Component: | Node | Assignee: | Ryan Phillips <rphillips> |
| Status: | CLOSED ERRATA | QA Contact: | Sunil Choudhary <schoudha> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 4.5 | CC: | amurdaca, aos-bugs, fromani, jokerman, minmli |
| Target Milestone: | --- | Keywords: | UpcomingSprint |
| Target Release: | 4.6.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-10-27 16:01:56 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1847562 | ||
I'm pretty sure the KubeletConfig retries after a minute or so:
func (ctrl *Controller) handleErr(err error, key interface{}) {
if err == nil {
ctrl.queue.Forget(key)
return
}
if _, ok := err.(*forgetError); ok {
ctrl.queue.Forget(key)
return
}
if ctrl.queue.NumRequeues(key) < maxRetries {
glog.V(2).Infof("Error syncing kubeletconfig %v: %v", key, err)
ctrl.queue.AddRateLimited(key)
return
}
utilruntime.HandleError(err)
glog.V(2).Infof("Dropping kubeletconfig %q out of the queue: %v", key, err)
ctrl.queue.Forget(key)
ctrl.queue.AddAfter(key, 1*time.Minute)
}
I'll spin up a cluster to confirm and setting to 4.6 if this behavior was already there.
Oh, damn, so yeah, this is exactly the behavior:
var errCouldNotFindMCPSet = newForgetError(errors.New("could not find any MachineConfigPool set for KubeletConfig"))
So that error is a forget error and when you get it, the KubeletConfig controller doesn't retry. I'm leaving it to Ryan to assess why that's the case for a transient error but this definitely is the case with <4.5 as well so leaving 4.6 as the target.
Nice find. Yeah, we should probably retry in that instance. I'll PR the change. verified with version : 4.6.0-0.nightly-2020-06-20-011219 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196 |
Description of problem: I created KubeletConfig pointed to non-existing MCP - and it showed expected condition: Machine Config Pool Selector: Match Labels: machineconfiguration.openshift.io/role: worker-cnf Status: Conditions: Last Transition Time: 2020-05-28T19:24:03Z Message: Error: could not find any MachineConfigPool set for KubeletConfig Status: False Type: Failure After that I created needed MCP and all data from MC was pushed to the node, but from KubeletConfig - was not. Version-Release number of selected component (if applicable): # oc version Client Version: 4.5.0-0.nightly-2020-05-26-063751 Server Version: 4.5.0-0.nightly-2020-05-26-063751 How reproducible: 100% Steps to Reproduce: 1. create KubeletConfig with non-existing "Machine Config Pool Selector" 2. create MCP for this KubeletConfig Actual results: Data from KubeletConfig was not pushed to the node, kubelet.conf on the node was not updated Expected results: MCO updates kubelet.conf on the node as soon as right MCP is created. Additional info: After re-creating/updating KubeletConfig MCO starts updating nodes.