Bug 1779312

Summary:	AWS installer flake: fetching availability zones: RequestLimitExceeded
Product:	OpenShift Container Platform	Reporter:	W. Trevor King <wking>
Component:	Installer	Assignee:	Abhinav Dahiya <adahiya>
Installer sub component:	openshift-installer	QA Contact:	Johnny Liu <jialiu>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	low
Priority:	low
Version:	4.3.0
Target Milestone:	---
Target Release:	4.5.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: Rate limiting from AWS would cause failure to fetch zones for the cluster Consequence: Rate limiting from AWS would cause failure to install the cluster Fix: Use exponential back-off to allow for longer wait timeout. Result: Less failures due to Rate limiting	Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-07-13 17:12:18 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description W. Trevor King 2019-12-03 18:05:16 UTC

We've had a few earlier bugs about various AWS throttling flakes, but I didn't see one for this particular flake yet, so here's a new bug.  Seen in [1]

Installing from release registry.svc.ci.openshift.org/ocp/release:4.3.0-0.nightly-2019-12-02-195545 level=fatal msg="failed to fetch Terraform Variables: failed to load asset \"Install Config\": [controlPlane.platform.aws: Internal error: creating AWS session: fetching availability zones: RequestLimitExceeded: Request limit exceeded.\n\tstatus code: 503, request id: ca5b3f84-aa5e-45b0-924e-6ed30a2cc280, compute[0].platform.aws: Internal error: creating AWS session: fetching availability zones: RequestLimitExceeded: Request limit exceeded.\n\tstatus code: 503, request id: db922eb8-eb1c-45c2-acfc-60d32b47fd2c]"

Flake rate will depend heavily on cluster throughput in the CI account, but currently this failure was the only such failure in the past 24 hours [2].

[1]: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.3/402
[2]: https://search.svc.ci.openshift.org/chart?name=aws&search=failed%20to%20fetch%20Terraform%20Variables.*fetching%20availability%20zones:%20RequestLimitExceeded

Comment 3 Johnny Liu 2020-03-23 10:25:11 UTC

Searched the past 14 days ci job record, did not such issues, so move this bug to VERIFIED.

https://search.svc.ci.openshift.org/?search=RequestLimitExceeded%3A+Request+limit+exceeded&maxAge=336h&context=15&type=all

Comment 5 errata-xmlrpc 2020-07-13 17:12:18 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409