Bug 1779312 - AWS installer flake: fetching availability zones: RequestLimitExceeded
Summary: AWS installer flake: fetching availability zones: RequestLimitExceeded
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 4.5.0
Assignee: Abhinav Dahiya
QA Contact: Johnny Liu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-12-03 18:05 UTC by W. Trevor King
Modified: 2020-07-13 17:12 UTC (History)
0 users

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Rate limiting from AWS would cause failure to fetch zones for the cluster Consequence: Rate limiting from AWS would cause failure to install the cluster Fix: Use exponential back-off to allow for longer wait timeout. Result: Less failures due to Rate limiting
Clone Of:
Environment:
Last Closed: 2020-07-13 17:12:18 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 3159 0 None closed Bug 1779312: pkg/asset/installconfig/aws/session.go: bump the retries to 25 for aws sdk 2020-10-28 15:57:58 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:12:47 UTC

Description W. Trevor King 2019-12-03 18:05:16 UTC
We've had a few earlier bugs about various AWS throttling flakes, but I didn't see one for this particular flake yet, so here's a new bug.  Seen in [1]

Installing from release registry.svc.ci.openshift.org/ocp/release:4.3.0-0.nightly-2019-12-02-195545 level=fatal msg="failed to fetch Terraform Variables: failed to load asset \"Install Config\": [controlPlane.platform.aws: Internal error: creating AWS session: fetching availability zones: RequestLimitExceeded: Request limit exceeded.\n\tstatus code: 503, request id: ca5b3f84-aa5e-45b0-924e-6ed30a2cc280, compute[0].platform.aws: Internal error: creating AWS session: fetching availability zones: RequestLimitExceeded: Request limit exceeded.\n\tstatus code: 503, request id: db922eb8-eb1c-45c2-acfc-60d32b47fd2c]"

Flake rate will depend heavily on cluster throughput in the CI account, but currently this failure was the only such failure in the past 24 hours [2].

[1]: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.3/402
[2]: https://search.svc.ci.openshift.org/chart?name=aws&search=failed%20to%20fetch%20Terraform%20Variables.*fetching%20availability%20zones:%20RequestLimitExceeded

Comment 3 Johnny Liu 2020-03-23 10:25:11 UTC
Searched the past 14 days ci job record, did not such issues, so move this bug to VERIFIED.

https://search.svc.ci.openshift.org/?search=RequestLimitExceeded%3A+Request+limit+exceeded&maxAge=336h&context=15&type=all

Comment 5 errata-xmlrpc 2020-07-13 17:12:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.