Created attachment 1866188 [details]
AP-Southeast-3 Must Gather

Version:

$ openshift-install version
v4.9.0

Platform:
AWS/OSD

Please specify:
* IPI

What happened?

When attempting to create a cluster in the new ap-southeast-3 region, the installer gets past the bootstrap phase and fails to create worker nodes.

Must gather is attached, I forgot to get the install logs specifically.

When I was digging into the cluster, I saw the following errors in the machine-api-operator machine-controller pod:

I0315 17:09:29.768070       1 controller.go:174] kirk-aps3-nqj8z-worker-ap-southeast-3a-lhfpv: reconciling Machine
I0315 17:09:29.768081       1 actuator.go:104] kirk-aps3-nqj8z-worker-ap-southeast-3a-lhfpv: actuator checking if machine exists
E0315 17:09:29.768650       1 controller.go:281] kirk-aps3-nqj8z-worker-ap-southeast-3a-lhfpv: failed to check if machine exists: kirk-aps3-nqj8z-worker-ap-southeast-3a-lhfpv: failed to create scope for machine: failed to create aws client: region "ap-southeast-3" not resolved: UnknownEndpointError: could not resolve endpoint
        partition: "all partitions", service: "ec2", region: "ap-southeast-3"
E0315 17:09:29.776716       1 controller.go:304] controller-runtime/manager/controller/machine_controller "msg"="Reconciler error" "error"="kirk-aps3-nqj8z-worker-ap-southeast-3a-lhfpv: failed to create scope for machine: failed to create aws client: region \"ap-southeast-3\" not resolved: UnknownEndpointError: could not resolve endpoint\n\tpartition: \"all partitions\", service: \"ec2\", region: \"ap-southeast-3\"" "name"="kirk-aps3-nqj8z-worker-ap-southeast-3a-lhfpv" "namespace"="openshift-machine-api"

What did you expect to happen?

Worker nodes to be spun up and installation finished.

How to reproduce it (as minimally and precisely as possible)?

Create a cluster in OCM with the following json and then wait for install to fail

$ cat << EOF > ocm.json
{
    "name": "[CLUSTER_NAME]",
    "ccs": {
        "enabled": true
    },
    "region": {
        "id": "ap-southeast-3"
    },
    "version": {
    "id": "openshift-v4.9.21",
    "channel_group": "stable"
     },
    "aws": {
        "account_id": "[ACCOUNT ID]",
        "access_key_id": "[ACCESS_KEY_ID]",
        "secret_access_key": "[SECRET_ACCESS_KEY]"
    }
}
EOF
$ ocm post /api/clusters_mgmt/v1/clusters --body ocm.json
...
wait for install to fail


Anything else we need to know?