|
Doc Text:
|
Cause:
Systems under load can run into a situation where pod or container creation requests from the Kubelet to CRI-O can take longer than the Kubelet expects. This causes the request to timeout. The Kubelet will attempt to re-request that resource (even though CRI-O is still working on creating that resource), causing new pod/container creation requests to fail with a message "name is reserved". Once CRI-O finishes the original request, it notices the request timed out, and cleans up the failed pod/container, starting the process over.
Consequence:
Pod and Container creation can stall for a long time, and many "name is reserved" errors are reported by the Kubelet. This also causes an already overloaded node to be further overloaded.
Fix:
CRI-O now saves progress on container/pod creations that timeout due to system load. It also stalls new requests from the Kubelet so there are fewer "name is reserved" errors reported.
Result:
Now, when systems are under load, CRI-O does everything it can to slow down the Kubelet and reduce load on the system. Before, when this situation came up, there would be a "name is reserved" error posted every 2 seconds. They're now stalled to every couple of minutes. Also, because CRI-O saves its work, and doesn't cleanup and retry, the overall load on the node is reduced and Kubelet and CRI-O should reconcile faster. Note: this does not solve the underlying problem of the node being resource starved, but it does mitigate the affect of resource starvation.
|