We've encountered a scenario on OCP 3.11 on Azure where the kubelet blocks for 15 minutes attaching a disk to a node. Details in https://github.com/openshift/openshift-azure/issues/1362. The problem happens when the Azure API is throttling API calls due to load, returning 429s. https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_openshift-azure/1430/pull-ci-openshift-openshift-azure-master-e2e-azure/3415/artifacts/internal-e2e/compute-testapp-goroutine-dump is an example goroutine dump of the kubelet when the issue happens. Goroutine 219 is cloudResourceSyncManager -> NodeAddresses -> getVmssVM. It has been stuck for 10 minutes because of 429(s?) returned by Azure. Goroutine 3792 is GetDiskLun. It is stuck on Get because 219 holds the mutex. We have also seen similar issues with the controller blocking for 15 minutes. I don't have a goroutine dump but suspect the same problem. We are enabling UseInstanceMetadata which will decrease API load, but would also like https://github.com/kubernetes/kubernetes/pull/70002 to be backported to at least OpenShift 3.11+. This removes an unnecessary API call in this code path on the storage side, which will improve robustness. Qualitatively, we have seen improvements by running with this patch.
PR submitted: https://github.com/openshift/origin/pull/22563
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2352