Red Hat Bugzilla – Bug 1523354
[starter-us-west-1] AWS API invocations from storage contributing to API call limit
Last modified: 2018-03-28 10:15:00 EDT
Description of problem: Over the course of about 4 hours, the following API invocation were recorded. DescribeVolumes and AttachVolumes are major contributors to hitting the AWS API limits for our clusters. cloud_provider: DescribeSubnets 1904 DescribeInstances 110747 CreateTags 247 CreateVolume 2765 DescribeVolumes 137380 <--- DetachVolume 3459 AttachVolume 159765 <--- DeleteVolume 274 DescribeLoadBalancers 16622 (~4700 PV's on the cluster. ~4800 PVCs) When these limits are hit, Ops cannot perform vital management operations on the clusters. Version-Release number of selected component (if applicable): v3.7.9 How reproducible: 100% Steps to Reproduce: 1. Monitor AWS API usage on large cluster Actual results: AWS API limits being hit and interfering with critical Operations to manage the clusters. Amazon will not raise these limits. Expected results: AWS API invocations must be well under the API limits imposed by Amazon.
We have fix majority of DescribeVolume and AttachVolume calls in https://github.com/openshift/ose/pull/981 PR. Specifically - https://github.com/openshift/ose/pull/981/commits/13570756be7d2d9119760f70465198716e6b83a5 commit ensures that we do not make calls for AttachVolume if volume is not in attachable state. Also because starter-us-east-2 is running old version of 3.6, it does not have multiattach fix we made (https://github.com/openshift/ose/pull/877). In a nutshell https://github.com/openshift/ose/pull/877 and https://github.com/openshift/ose/pull/981 will reduce frequency of these calls within a very reasonable limit.
This should be fixed in 3.9
It is passed on oc v3.9.0-0.31.0 kubernetes v1.9.1+a0ce1bc657 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://ip-172-18-1-24.ec2.internal:443 openshift v3.9.0-0.31.0 kubernetes v1.9.1+a0ce1bc657 The describevolumes and attachvolumes calls is reduced in V3.9
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0489