Bug 1945467 - openshift-install AWS SDK is missing endpoints for the ap-northeast-3 region
Summary: openshift-install AWS SDK is missing endpoints for the ap-northeast-3 region
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.6
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.7.z
Assignee: Matthew Staebler
QA Contact: Yunfei Jiang
URL:
Whiteboard:
Depends On: 1944268 1948923 1977159
Blocks: 1981929
TreeView+ depends on / blocked
 
Reported: 2021-04-01 01:22 UTC by Matthew Staebler
Modified: 2021-07-26 17:35 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1944268
: 1981929 (view as bug list)
Environment:
Last Closed: 2021-07-26 17:35:21 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 4807 0 None closed Bug 1945467: aws: allow use of unknown regions in known partitions 2021-07-14 18:49:49 UTC
Red Hat Product Errata RHBA-2021:2762 0 None None None 2021-07-26 17:35:37 UTC

Description Matthew Staebler 2021-04-01 01:22:40 UTC
+++ This bug was initially created as a clone of Bug #1944268 +++

Version:

$ openshift-install version
./openshift-install 4.6.23
built from commit 9c86c823fff234c104f574eaf25953485edfe4b1
release image quay.io/openshift-release-dev/ocp-release@sha256:5c8ab6c4a863f9ac077bc743579d9387bb7cd311a36c3197e609f8be63d17981

Platform:
AWS

Please specify:
IPI 

What happened?

% ./openshift-install create cluster --dir katherine
FATAL failed to fetch Metadata: failed to load asset "Install Config": platform.aws.serviceEndpoints: Invalid value: []aws.ServiceEndpoint(nil): [failed to find endpoint for service "ec2": UnknownEndpointError: could not resolve endpoint
	partition: "all partitions", service: "ec2", region: "ap-northeast-3", failed to find endpoint for service "elasticloadbalancing": UnknownEndpointError: could not resolve endpoint
	partition: "all partitions", service: "elasticloadbalancing", region: "ap-northeast-3", failed to find endpoint for service "iam": UnknownEndpointError: could not resolve endpoint
	partition: "all partitions", service: "iam", region: "ap-northeast-3", failed to find endpoint for service "route53": UnknownEndpointError: could not resolve endpoint
	partition: "all partitions", service: "route53", region: "ap-northeast-3", failed to find endpoint for service "s3": UnknownEndpointError: could not resolve endpoint
	partition: "all partitions", service: "s3", region: "ap-northeast-3", failed to find endpoint for service "sts": UnknownEndpointError: could not resolve endpoint
	partition: "all partitions", service: "sts", region: "ap-northeast-3", failed to find endpoint for service "tagging": UnknownEndpointError: could not resolve endpoint
	partition: "all partitions", service: "tagging", region: "ap-northeast-3"]

What did you expect to happen?

Successful installation.

How to reproduce it (as minimally and precisely as possible)?

install-config.yaml: 

platform:
  aws:
    region: ap-northeast-3
    amiID: ami-0310bc3d6eec49e56

Anything else we need to know?

Trying to enable support for AWS ap-northeast-3, but the AWS SDK used for the installer doesn't know the necessary endpoints for this region.

--- Additional comment from Matthew Staebler on 2021-03-29 16:16:07 UTC ---

The ap-northeast-3 endpoint was added to the ASK SDK in v1.37.24 [1]. The 4.6 installer is using v1.32.3.

As a workaround, the user could add the endpoints manually to the install config.

[1] https://github.com/aws/aws-sdk-go/commit/6a71e1594856a4350bed5c29ba63724b66372591

--- Additional comment from Katherine Dubé on 2021-03-29 19:58:12 UTC ---

Manually defining AWS service endpoints for the ap-northeast-3 region doesn't appear to work.

Excerpt from install-config.yaml:

platform:
  aws:
    amiID: ami-0310bc3d6eec49e56
    region: ap-northeast-3
    serviceEndpoints:
    - name: ec2
      url: https://ec2.ap-northeast-3.amazonaws.com
    - name: elasticloadbalancing
      url: https://elasticloadbalancing.ap-northeast-3.amazonaws.com
    - name: iam
      url: https://iam.amazonaws.com
    - name: route53
      url: https://route53.amazonaws.com
    - name: s3
      url: https://s3.ap-northeast-3.amazonaws.com
    - name: sts
      url: https://sts.ap-northeast-3.amazonaws.com
    - name: tagging
      url: https://tagging.ap-northeast-3.amazonaws.com


% ./openshift-install create cluster --dir katherine
INFO Consuming Install Config from target directory
INFO Credentials loaded from the "openshift-dev" profile in file "/Users/katherine/.aws/credentials"
WARNING Failed to find information on quotas ec2/L-0263D0A3, ec2/L-1216C47A
INFO Creating infrastructure resources...
ERROR
ERROR Error: Error creating IAM Role katherine-2kjw8-bootstrap-role: SignatureDoesNotMatch: Credential should be scoped to a valid region, not 'ap-northeast-3'.
ERROR 	status code: 403, request id: 862e5492-15b9-4fa5-926a-c31518a9984a
ERROR
ERROR   on ../../../../private/var/folders/2m/t4ltl17174s0x9v935kr59v40000gn/T/openshift-install-702836295/bootstrap/main.tf line 51, in resource "aws_iam_role" "bootstrap":
ERROR   51: resource "aws_iam_role" "bootstrap" {
ERROR
ERROR
ERROR
ERROR Error: Error creating IAM Role katherine-2kjw8-worker-role: SignatureDoesNotMatch: Credential should be scoped to a valid region, not 'ap-northeast-3'.
ERROR 	status code: 403, request id: 8b4f0f0b-8299-4297-bf9b-c5f4f180dacb
ERROR
ERROR   on ../../../../private/var/folders/2m/t4ltl17174s0x9v935kr59v40000gn/T/openshift-install-702836295/iam/main.tf line 13, in resource "aws_iam_role" "worker_role":
ERROR   13: resource "aws_iam_role" "worker_role" {
ERROR
ERROR
ERROR
ERROR Error: Error creating IAM Role katherine-2kjw8-master-role: SignatureDoesNotMatch: Credential should be scoped to a valid region, not 'ap-northeast-3'.
ERROR 	status code: 403, request id: 6fd96bfa-636a-4df9-bd25-22e0384238ab
ERROR
ERROR   on ../../../../private/var/folders/2m/t4ltl17174s0x9v935kr59v40000gn/T/openshift-install-702836295/master/main.tf line 17, in resource "aws_iam_role" "master_role":
ERROR   17: resource "aws_iam_role" "master_role" {
ERROR
ERROR
FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change

Additionally, if I omit the service endpoints for IAM and Route 53 (since they're not region specific), then the installer throws an error:

% ./openshift-install create cluster --dir katherine
FATAL failed to fetch Metadata: failed to load asset "Install Config": platform.aws.serviceEndpoints: Invalid value: []aws.ServiceEndpoint{aws.ServiceEndpoint{Name:"ec2", URL:"https://ec2.ap-northeast-3.amazonaws.com"}, aws.ServiceEndpoint{Name:"elasticloadbalancing", URL:"https://elasticloadbalancing.ap-northeast-3.amazonaws.com"}, aws.ServiceEndpoint{Name:"s3", URL:"https://s3.ap-northeast-3.amazonaws.com"}, aws.ServiceEndpoint{Name:"sts", URL:"https://sts.ap-northeast-3.amazonaws.com"}, aws.ServiceEndpoint{Name:"tagging", URL:"https://tagging.ap-northeast-3.amazonaws.com"}}: [failed to find endpoint for service "iam": UnknownEndpointError: could not resolve endpoint
	partition: "all partitions", service: "iam", region: "ap-northeast-3", failed to find endpoint for service "route53": UnknownEndpointError: could not resolve endpoint
	partition: "all partitions", service: "route53", region: "ap-northeast-3"]

--- Additional comment from Matthew Staebler on 2021-03-29 21:31:12 UTC ---

The installer does not need to be so restrictive when validation the service endpoints. The installer should accept a region [1] that matches the regex for a partition, even if the SDK does not know the region.

[1] https://github.com/openshift/installer/blob/6363f3ab700e3976e8655ba0e826843593c7c98f/pkg/asset/installconfig/aws/validation.go#L255-L264

Comment 2 Yunfei Jiang 2021-06-18 06:59:06 UTC
verify failed, worker nodes were not created successfully.
OCP version: 4.7.0-0.nightly-2021-06-17-224843

Comment 4 Matthew Staebler 2021-06-18 15:28:47 UTC
(In reply to Yunfei Jiang from comment #2)
> verify failed, worker nodes were not created successfully.
> OCP version: 4.7.0-0.nightly-2021-06-17-224843

The worker nodes not created needs to be handled by a backport of https://bugzilla.redhat.com/show_bug.cgi?id=1949626.
The image registry operator failing needs to be handled by a backport of https://bugzilla.redhat.com/show_bug.cgi?id=1949626.

Comment 5 Yunfei Jiang 2021-06-21 01:41:06 UTC
Thanks Matthew, I will verify again after the backport of bug 1949626 get merged.

Comment 9 Yunfei Jiang 2021-07-16 02:04:02 UTC
verified. pass.

OCP version: 4.7.0-0.nightly-2021-07-14-184406

Comment 12 errata-xmlrpc 2021-07-26 17:35:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.7.21 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2762


Note You need to log in before you can comment on or make changes to this bug.