Version: $ openshift-install version ./openshift-install 4.6.23 built from commit 9c86c823fff234c104f574eaf25953485edfe4b1 release image quay.io/openshift-release-dev/ocp-release@sha256:5c8ab6c4a863f9ac077bc743579d9387bb7cd311a36c3197e609f8be63d17981 Platform: AWS Please specify: IPI What happened? % ./openshift-install create cluster --dir katherine FATAL failed to fetch Metadata: failed to load asset "Install Config": platform.aws.serviceEndpoints: Invalid value: []aws.ServiceEndpoint(nil): [failed to find endpoint for service "ec2": UnknownEndpointError: could not resolve endpoint partition: "all partitions", service: "ec2", region: "ap-northeast-3", failed to find endpoint for service "elasticloadbalancing": UnknownEndpointError: could not resolve endpoint partition: "all partitions", service: "elasticloadbalancing", region: "ap-northeast-3", failed to find endpoint for service "iam": UnknownEndpointError: could not resolve endpoint partition: "all partitions", service: "iam", region: "ap-northeast-3", failed to find endpoint for service "route53": UnknownEndpointError: could not resolve endpoint partition: "all partitions", service: "route53", region: "ap-northeast-3", failed to find endpoint for service "s3": UnknownEndpointError: could not resolve endpoint partition: "all partitions", service: "s3", region: "ap-northeast-3", failed to find endpoint for service "sts": UnknownEndpointError: could not resolve endpoint partition: "all partitions", service: "sts", region: "ap-northeast-3", failed to find endpoint for service "tagging": UnknownEndpointError: could not resolve endpoint partition: "all partitions", service: "tagging", region: "ap-northeast-3"] What did you expect to happen? Successful installation. How to reproduce it (as minimally and precisely as possible)? install-config.yaml: platform: aws: region: ap-northeast-3 amiID: ami-0310bc3d6eec49e56 Anything else we need to know? Trying to enable support for AWS ap-northeast-3, but the AWS SDK used for the installer doesn't know the necessary endpoints for this region.
The ap-northeast-3 endpoint was added to the ASK SDK in v1.37.24 [1]. The 4.6 installer is using v1.32.3. As a workaround, the user could add the endpoints manually to the install config. [1] https://github.com/aws/aws-sdk-go/commit/6a71e1594856a4350bed5c29ba63724b66372591
Manually defining AWS service endpoints for the ap-northeast-3 region doesn't appear to work. Excerpt from install-config.yaml: platform: aws: amiID: ami-0310bc3d6eec49e56 region: ap-northeast-3 serviceEndpoints: - name: ec2 url: https://ec2.ap-northeast-3.amazonaws.com - name: elasticloadbalancing url: https://elasticloadbalancing.ap-northeast-3.amazonaws.com - name: iam url: https://iam.amazonaws.com - name: route53 url: https://route53.amazonaws.com - name: s3 url: https://s3.ap-northeast-3.amazonaws.com - name: sts url: https://sts.ap-northeast-3.amazonaws.com - name: tagging url: https://tagging.ap-northeast-3.amazonaws.com % ./openshift-install create cluster --dir katherine INFO Consuming Install Config from target directory INFO Credentials loaded from the "openshift-dev" profile in file "/Users/katherine/.aws/credentials" WARNING Failed to find information on quotas ec2/L-0263D0A3, ec2/L-1216C47A INFO Creating infrastructure resources... ERROR ERROR Error: Error creating IAM Role katherine-2kjw8-bootstrap-role: SignatureDoesNotMatch: Credential should be scoped to a valid region, not 'ap-northeast-3'. ERROR status code: 403, request id: 862e5492-15b9-4fa5-926a-c31518a9984a ERROR ERROR on ../../../../private/var/folders/2m/t4ltl17174s0x9v935kr59v40000gn/T/openshift-install-702836295/bootstrap/main.tf line 51, in resource "aws_iam_role" "bootstrap": ERROR 51: resource "aws_iam_role" "bootstrap" { ERROR ERROR ERROR ERROR Error: Error creating IAM Role katherine-2kjw8-worker-role: SignatureDoesNotMatch: Credential should be scoped to a valid region, not 'ap-northeast-3'. ERROR status code: 403, request id: 8b4f0f0b-8299-4297-bf9b-c5f4f180dacb ERROR ERROR on ../../../../private/var/folders/2m/t4ltl17174s0x9v935kr59v40000gn/T/openshift-install-702836295/iam/main.tf line 13, in resource "aws_iam_role" "worker_role": ERROR 13: resource "aws_iam_role" "worker_role" { ERROR ERROR ERROR ERROR Error: Error creating IAM Role katherine-2kjw8-master-role: SignatureDoesNotMatch: Credential should be scoped to a valid region, not 'ap-northeast-3'. ERROR status code: 403, request id: 6fd96bfa-636a-4df9-bd25-22e0384238ab ERROR ERROR on ../../../../private/var/folders/2m/t4ltl17174s0x9v935kr59v40000gn/T/openshift-install-702836295/master/main.tf line 17, in resource "aws_iam_role" "master_role": ERROR 17: resource "aws_iam_role" "master_role" { ERROR ERROR FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change Additionally, if I omit the service endpoints for IAM and Route 53 (since they're not region specific), then the installer throws an error: % ./openshift-install create cluster --dir katherine FATAL failed to fetch Metadata: failed to load asset "Install Config": platform.aws.serviceEndpoints: Invalid value: []aws.ServiceEndpoint{aws.ServiceEndpoint{Name:"ec2", URL:"https://ec2.ap-northeast-3.amazonaws.com"}, aws.ServiceEndpoint{Name:"elasticloadbalancing", URL:"https://elasticloadbalancing.ap-northeast-3.amazonaws.com"}, aws.ServiceEndpoint{Name:"s3", URL:"https://s3.ap-northeast-3.amazonaws.com"}, aws.ServiceEndpoint{Name:"sts", URL:"https://sts.ap-northeast-3.amazonaws.com"}, aws.ServiceEndpoint{Name:"tagging", URL:"https://tagging.ap-northeast-3.amazonaws.com"}}: [failed to find endpoint for service "iam": UnknownEndpointError: could not resolve endpoint partition: "all partitions", service: "iam", region: "ap-northeast-3", failed to find endpoint for service "route53": UnknownEndpointError: could not resolve endpoint partition: "all partitions", service: "route53", region: "ap-northeast-3"]
The installer does not need to be so restrictive when validation the service endpoints. The installer should accept a region [1] that matches the regex for a partition, even if the SDK does not know the region. [1] https://github.com/openshift/installer/blob/6363f3ab700e3976e8655ba0e826843593c7c98f/pkg/asset/installconfig/aws/validation.go#L255-L264
verified. FAILED. OCP version: 4.8.0-0.nightly-2021-04-29-222100 only the master nodes were created but no workers: Status: Conditions: Last Transition Time: 2021-04-30T07:55:54Z Message: Failed to check if machine exists: yunjiang-ap3-g5s68-worker-ap-northeast-3c-9lpgn: failed to create scope for machine: failed to create aws client: region "ap-northeast-3" not resolved: UnknownEndpointError: could not resolve endpoint partition: "all partitions", service: "ec2", region: "ap-northeast-3" Reason: ErrorCheckingProvider Status: Unknown Type: InstanceExists Last Updated: 2021-04-30T07:55:54Z Phase:
This BZ just addresses installer support for the region. The in-cluster operators that failed when using the new region have been fixed in separate BZs. If there are other operators that fail, we should create additional BZs for those.
verified. PASS. OCP version: 4.8.0-0.nightly-2021-06-08-005718
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438