Bug 1523354
| Summary: | [starter-us-west-1] AWS API invocations from storage contributing to API call limit | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Justin Pierce <jupierce> | |
| Component: | Storage | Assignee: | Hemant Kumar <hekumar> | |
| Status: | CLOSED ERRATA | QA Contact: | Chao Yang <chaoyang> | |
| Severity: | unspecified | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 3.7.0 | CC: | aos-bugs, aos-storage-staff, sten | |
| Target Milestone: | --- | |||
| Target Release: | 3.9.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1523359 (view as bug list) | Environment: | ||
| Last Closed: | 2018-03-28 14:14:24 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1523359 | |||
We have fix majority of DescribeVolume and AttachVolume calls in https://github.com/openshift/ose/pull/981 PR. Specifically - https://github.com/openshift/ose/pull/981/commits/13570756be7d2d9119760f70465198716e6b83a5 commit ensures that we do not make calls for AttachVolume if volume is not in attachable state. Also because starter-us-east-2 is running old version of 3.6, it does not have multiattach fix we made (https://github.com/openshift/ose/pull/877). In a nutshell https://github.com/openshift/ose/pull/877 and https://github.com/openshift/ose/pull/981 will reduce frequency of these calls within a very reasonable limit. This should be fixed in 3.9 It is passed on oc v3.9.0-0.31.0 kubernetes v1.9.1+a0ce1bc657 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://ip-172-18-1-24.ec2.internal:443 openshift v3.9.0-0.31.0 kubernetes v1.9.1+a0ce1bc657 The describevolumes and attachvolumes calls is reduced in V3.9 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0489 |
Description of problem: Over the course of about 4 hours, the following API invocation were recorded. DescribeVolumes and AttachVolumes are major contributors to hitting the AWS API limits for our clusters. cloud_provider: DescribeSubnets 1904 DescribeInstances 110747 CreateTags 247 CreateVolume 2765 DescribeVolumes 137380 <--- DetachVolume 3459 AttachVolume 159765 <--- DeleteVolume 274 DescribeLoadBalancers 16622 (~4700 PV's on the cluster. ~4800 PVCs) When these limits are hit, Ops cannot perform vital management operations on the clusters. Version-Release number of selected component (if applicable): v3.7.9 How reproducible: 100% Steps to Reproduce: 1. Monitor AWS API usage on large cluster Actual results: AWS API limits being hit and interfering with critical Operations to manage the clusters. Amazon will not raise these limits. Expected results: AWS API invocations must be well under the API limits imposed by Amazon.