Bug 1861088

Summary: AWS GovCloud blocked by Service Quota
Product: OpenShift Container Platform Reporter: Patrick Dillon <padillon>
Component: InstallerAssignee: Abhinav Dahiya <adahiya>
Installer sub component: openshift-installer QA Contact: Yunfei Jiang <yunjiang>
Status: CLOSED ERRATA Docs Contact:
Severity: urgent    
Priority: high CC: adahiya
Version: 4.6   
Target Milestone: ---   
Target Release: 4.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-10-27 16:17:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Patrick Dillon 2020-07-27 18:53:45 UTC
Description of problem:
Service quota endpoint is not available in GovCloud. See list of endpoints here:
https://docs.aws.amazon.com/general/latest/gr/servicequotas.html

Steps to Reproduce:
Attempt install in govcloud environment.

Actual results:
In the govcloud emulator environment, install will stall for a long period. The emulator analytics shows repeated calls to the bad endpoint.

Expected results:
Installer should skip the quota check in govcloud regions.

Comment 1 Abhinav Dahiya 2020-07-27 19:01:16 UTC
Can you include the error message from the failed run?

Comment 2 Patrick Dillon 2020-07-27 19:13:09 UTC
The most noticeable behavior from the installer is that it will just hang on the Platform Quota Check:

time="2020-07-27T19:06:20Z" level=debug msg="    Reusing previously-fetched Install Config"
time="2020-07-27T19:06:20Z" level=debug msg="  Generating Platform Provisioning Check..."
time="2020-07-27T19:06:20Z" level=debug msg="  Fetching Platform Quota Check..."
time="2020-07-27T19:06:20Z" level=debug msg="    Fetching Install Config..."
time="2020-07-27T19:06:20Z" level=debug msg="    Reusing previously-fetched Install Config"
time="2020-07-27T19:06:20Z" level=debug msg="    Fetching Master Machines..."
time="2020-07-27T19:06:20Z" level=debug msg="    Reusing previously-fetched Master Machines"
time="2020-07-27T19:06:20Z" level=debug msg="    Fetching Worker Machines..."
time="2020-07-27T19:06:20Z" level=debug msg="    Reusing previously-fetched Worker Machines"
time="2020-07-27T19:06:20Z" level=debug msg="  Generating Platform Quota Check..."


I am not sure if an error message ever gets populated back to the installer, as the aws client constantly retries the endpoint. I will let the installer run and see if an error message eventually pops up eventually. I will update this BZ.

The error message in the emulator says:

Validation Type	Service_Not_Available
Severity	SEVERE
Validation Target	servicequotas.us-gov-west-1.amazonaws.com.ec2.internal
Error Message	The service you are attempting to use is not available in the emulated region
Remedy	Unfortunately there is no remedy for this issue, you must refactor your system to not use this service to work in the emulated region

Comment 3 Patrick Dillon 2020-07-28 03:18:07 UTC
An hour later, the error returned is:
time="2020-07-27T20:07:59Z" level=fatal msg="failed to fetch Cluster: failed to fetch dependency of \"Cluster\": failed to generate asset \"Platform Quota Check\": failed to load Quota for services: ec2, vpc: failed to load limits for servicequotas: failed to list default serviceqquotas for ec2: RequestError: send request failed\ncaused by: Post \"https://servicequotas.us-gov-west-1.amazonaws.com/\": dial tcp: lookup servicequotas.us-gov-west-1.amazonaws.com on 10.85.2.94:53: read udp 10.119.0.58:55953->10.85.2.94:53: i/o timeout"

Comment 8 Yunfei Jiang 2020-08-17 07:36:59 UTC
verified. PASS.

version: 4.6.0-0.nightly-2020-08-16-072105

level=debug msg="us-gov-west-1 does not support API for checking quotas, therefore skipping."

Comment 10 errata-xmlrpc 2020-10-27 16:17:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196

Comment 11 errata-xmlrpc 2020-10-27 16:20:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196