Bug 1696901

Summary:	Request backport of https://github.com/kubernetes/kubernetes/pull/70002
Product:	OpenShift Container Platform	Reporter:	Jim Minter <jminter>
Component:	Storage	Assignee:	Fabio Bertinatto <fbertina>
Status:	CLOSED ERRATA	QA Contact:	Liang Xia <lxia>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	3.11.0	CC:	aos-bugs, aos-storage-staff, bchilds, esimard, fbertina, jsafrane, mgahagan, mifiedle
Target Milestone:	---
Target Release:	3.11.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-08-13 14:09:19 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Jim Minter 2019-04-06 00:12:23 UTC

We've encountered a scenario on OCP 3.11 on Azure where the kubelet blocks for 15 minutes attaching a disk to a node.  Details in https://github.com/openshift/openshift-azure/issues/1362.

The problem happens when the Azure API is throttling API calls due to load, returning 429s.

https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_openshift-azure/1430/pull-ci-openshift-openshift-azure-master-e2e-azure/3415/artifacts/internal-e2e/compute-testapp-goroutine-dump is an example goroutine dump of the kubelet when the issue happens.

Goroutine 219 is cloudResourceSyncManager -> NodeAddresses -> getVmssVM. It has been stuck for 10 minutes because of 429(s?) returned by Azure.

Goroutine 3792 is GetDiskLun. It is stuck on Get because 219 holds the mutex.

We have also seen similar issues with the controller blocking for 15 minutes.  I don't have a goroutine dump but suspect the same problem.

We are enabling UseInstanceMetadata which will decrease API load, but would also like https://github.com/kubernetes/kubernetes/pull/70002 to be backported to at least OpenShift 3.11+.  This removes an unnecessary API call in this code path on the storage side, which will improve robustness.

Qualitatively, we have seen improvements by running with this patch.

Comment 5 Fabio Bertinatto 2019-04-12 15:17:39 UTC

PR submitted: https://github.com/openshift/origin/pull/22563

Comment 12 errata-xmlrpc 2019-08-13 14:09:19 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2352