+++ This bug was initially created as a clone of Bug #1944268 +++ Version: $ openshift-install version ./openshift-install 4.6.23 built from commit 9c86c823fff234c104f574eaf25953485edfe4b1 release image quay.io/openshift-release-dev/ocp-release@sha256:5c8ab6c4a863f9ac077bc743579d9387bb7cd311a36c3197e609f8be63d17981 Platform: AWS Please specify: IPI What happened? % ./openshift-install create cluster --dir katherine FATAL failed to fetch Metadata: failed to load asset "Install Config": platform.aws.serviceEndpoints: Invalid value: []aws.ServiceEndpoint(nil): [failed to find endpoint for service "ec2": UnknownEndpointError: could not resolve endpoint partition: "all partitions", service: "ec2", region: "ap-northeast-3", failed to find endpoint for service "elasticloadbalancing": UnknownEndpointError: could not resolve endpoint partition: "all partitions", service: "elasticloadbalancing", region: "ap-northeast-3", failed to find endpoint for service "iam": UnknownEndpointError: could not resolve endpoint partition: "all partitions", service: "iam", region: "ap-northeast-3", failed to find endpoint for service "route53": UnknownEndpointError: could not resolve endpoint partition: "all partitions", service: "route53", region: "ap-northeast-3", failed to find endpoint for service "s3": UnknownEndpointError: could not resolve endpoint partition: "all partitions", service: "s3", region: "ap-northeast-3", failed to find endpoint for service "sts": UnknownEndpointError: could not resolve endpoint partition: "all partitions", service: "sts", region: "ap-northeast-3", failed to find endpoint for service "tagging": UnknownEndpointError: could not resolve endpoint partition: "all partitions", service: "tagging", region: "ap-northeast-3"] What did you expect to happen? Successful installation. How to reproduce it (as minimally and precisely as possible)? install-config.yaml: platform: aws: region: ap-northeast-3 amiID: ami-0310bc3d6eec49e56 Anything else we need to know? Trying to enable support for AWS ap-northeast-3, but the AWS SDK used for the installer doesn't know the necessary endpoints for this region. --- Additional comment from Matthew Staebler on 2021-03-29 16:16:07 UTC --- The ap-northeast-3 endpoint was added to the ASK SDK in v1.37.24 [1]. The 4.6 installer is using v1.32.3. As a workaround, the user could add the endpoints manually to the install config. [1] https://github.com/aws/aws-sdk-go/commit/6a71e1594856a4350bed5c29ba63724b66372591 --- Additional comment from Katherine Dubé on 2021-03-29 19:58:12 UTC --- Manually defining AWS service endpoints for the ap-northeast-3 region doesn't appear to work. Excerpt from install-config.yaml: platform: aws: amiID: ami-0310bc3d6eec49e56 region: ap-northeast-3 serviceEndpoints: - name: ec2 url: https://ec2.ap-northeast-3.amazonaws.com - name: elasticloadbalancing url: https://elasticloadbalancing.ap-northeast-3.amazonaws.com - name: iam url: https://iam.amazonaws.com - name: route53 url: https://route53.amazonaws.com - name: s3 url: https://s3.ap-northeast-3.amazonaws.com - name: sts url: https://sts.ap-northeast-3.amazonaws.com - name: tagging url: https://tagging.ap-northeast-3.amazonaws.com % ./openshift-install create cluster --dir katherine INFO Consuming Install Config from target directory INFO Credentials loaded from the "openshift-dev" profile in file "/Users/katherine/.aws/credentials" WARNING Failed to find information on quotas ec2/L-0263D0A3, ec2/L-1216C47A INFO Creating infrastructure resources... ERROR ERROR Error: Error creating IAM Role katherine-2kjw8-bootstrap-role: SignatureDoesNotMatch: Credential should be scoped to a valid region, not 'ap-northeast-3'. ERROR status code: 403, request id: 862e5492-15b9-4fa5-926a-c31518a9984a ERROR ERROR on ../../../../private/var/folders/2m/t4ltl17174s0x9v935kr59v40000gn/T/openshift-install-702836295/bootstrap/main.tf line 51, in resource "aws_iam_role" "bootstrap": ERROR 51: resource "aws_iam_role" "bootstrap" { ERROR ERROR ERROR ERROR Error: Error creating IAM Role katherine-2kjw8-worker-role: SignatureDoesNotMatch: Credential should be scoped to a valid region, not 'ap-northeast-3'. ERROR status code: 403, request id: 8b4f0f0b-8299-4297-bf9b-c5f4f180dacb ERROR ERROR on ../../../../private/var/folders/2m/t4ltl17174s0x9v935kr59v40000gn/T/openshift-install-702836295/iam/main.tf line 13, in resource "aws_iam_role" "worker_role": ERROR 13: resource "aws_iam_role" "worker_role" { ERROR ERROR ERROR ERROR Error: Error creating IAM Role katherine-2kjw8-master-role: SignatureDoesNotMatch: Credential should be scoped to a valid region, not 'ap-northeast-3'. ERROR status code: 403, request id: 6fd96bfa-636a-4df9-bd25-22e0384238ab ERROR ERROR on ../../../../private/var/folders/2m/t4ltl17174s0x9v935kr59v40000gn/T/openshift-install-702836295/master/main.tf line 17, in resource "aws_iam_role" "master_role": ERROR 17: resource "aws_iam_role" "master_role" { ERROR ERROR FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change Additionally, if I omit the service endpoints for IAM and Route 53 (since they're not region specific), then the installer throws an error: % ./openshift-install create cluster --dir katherine FATAL failed to fetch Metadata: failed to load asset "Install Config": platform.aws.serviceEndpoints: Invalid value: []aws.ServiceEndpoint{aws.ServiceEndpoint{Name:"ec2", URL:"https://ec2.ap-northeast-3.amazonaws.com"}, aws.ServiceEndpoint{Name:"elasticloadbalancing", URL:"https://elasticloadbalancing.ap-northeast-3.amazonaws.com"}, aws.ServiceEndpoint{Name:"s3", URL:"https://s3.ap-northeast-3.amazonaws.com"}, aws.ServiceEndpoint{Name:"sts", URL:"https://sts.ap-northeast-3.amazonaws.com"}, aws.ServiceEndpoint{Name:"tagging", URL:"https://tagging.ap-northeast-3.amazonaws.com"}}: [failed to find endpoint for service "iam": UnknownEndpointError: could not resolve endpoint partition: "all partitions", service: "iam", region: "ap-northeast-3", failed to find endpoint for service "route53": UnknownEndpointError: could not resolve endpoint partition: "all partitions", service: "route53", region: "ap-northeast-3"] --- Additional comment from Matthew Staebler on 2021-03-29 21:31:12 UTC --- The installer does not need to be so restrictive when validation the service endpoints. The installer should accept a region [1] that matches the regex for a partition, even if the SDK does not know the region. [1] https://github.com/openshift/installer/blob/6363f3ab700e3976e8655ba0e826843593c7c98f/pkg/asset/installconfig/aws/validation.go#L255-L264
verify failed, worker nodes were not created successfully. OCP version: 4.7.0-0.nightly-2021-06-17-224843
(In reply to Yunfei Jiang from comment #2) > verify failed, worker nodes were not created successfully. > OCP version: 4.7.0-0.nightly-2021-06-17-224843 The worker nodes not created needs to be handled by a backport of https://bugzilla.redhat.com/show_bug.cgi?id=1949626. The image registry operator failing needs to be handled by a backport of https://bugzilla.redhat.com/show_bug.cgi?id=1949626.
Thanks Matthew, I will verify again after the backport of bug 1949626 get merged.
verified. pass. OCP version: 4.7.0-0.nightly-2021-07-14-184406
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.7.21 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:2762