Description of problem: Installer isn't showing all the regions made available by cloud providers (AWS/GCP). How reproducible: Launch the installer and go through the list of regions available to you and compare that with the regions made available by the cloud provider. Steps to Reproduce: 1. Confirm the regions you can see for your account/projects. For GCP, run cloud compute regions list --project openshift-qe 2. Launch the installer and scroll through the list of regions available (using the same GCP credential as your cli). Compare the two and you will find that a few regions are not listed by the installer: Missing GCP regions: asia-northeast3, asia-southeast-2, us-west3, us-west4 Missing aws: af-south-1, ap-east-1, eu-south-1 Actual results: Some regions are missing Expected results: All regions available are shown by the installer Additional info: 1. Check the cloud provider's page and you can see that those missing regions are indeed available (some are recently made available in 2020) AWS: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-available-regions GCP: https://cloud.google.com/compute/docs/regions-zones 2. Looks like the installer module doesn't include those missing regions: https://github.com/openshift/installer/blob/master/pkg/types/gcp/validation/platform.go var ( // Regions is a map of known GCP regions. The key of the map is // the short name of the region. The value of the map is the long // name of the region. Regions = map[string]string{ // List from: https://cloud.google.com/compute/docs/regions-zones/ "asia-east1": "Changhua County, Taiwan", "asia-east2": "Hong Kong", "asia-northeast1": "Tokyo, Japan", "asia-northeast2": "Osaka, Japan", "asia-south1": "Mumbai, India", "asia-southeast1": "Jurong West, Singapore", "australia-southeast1": "Sydney, Australia", "europe-north1": "Hamina, Finland", "europe-west1": "St. Ghislain, Belgium", "europe-west2": "London, England, UK", "europe-west3": "Frankfurt, Germany", "europe-west4": "Eemshaven, Netherlands", "europe-west6": "Zürich, Switzerland", "northamerica-northeast1": "Montréal, Québec, Canada", "southamerica-east1": "São Paulo, Brazil", "us-central1": "Council Bluffs, Iowa, USA", "us-east1": "Moncks Corner, South Carolina, USA", "us-east4": "Ashburn, Northern Virginia, USA", "us-west1": "The Dalles, Oregon, USA", "us-west2": "Los Angeles, California, USA", } 3. Using Flexy to install a build while specifying region to be "us-west4" will fail with this Generate manifests files....... + ./openshift-install create manifests --dir '/home/jenkins/workspace/Launch Environment Flexy/workdir/install-dir' level=fatal msg="failed to fetch Master Machines: failed to load asset \"Install Config\": invalid \"install-config.yaml\" file: platform.gcp.region: Unsupported value: \"us-west4\": supported values: \"asia-east1\", \"asia-east2\", \"asia-northeast1\", \"asia-northeast2\", \"asia-south1\", \"asia-southeast1\", \"australia-southeast1\", \"europe-north1\", \"europe-west1\", \"europe-west2\", \"europe-west3\", \"europe-west4\", \"europe-west6\", \"northamerica-northeast1\", \"southamerica-east1\", \"us-central1\", \"us-east1\", \"us-east4\", \"us-west1\", \"us-west2\"" 4. I have this AWS version installed aws --version aws-cli/2.0.17 Python/3.7.3 Linux/4.18.0-144.el8.x86_64 botocore/2.0.0dev21 5. AWS console does show those missing region as available to my account. https://console.aws.amazon.com/ec2/
If and when the list of regions made available by the installer is updated, the doc should also be updated to show the newly added regions as supported: https://docs.openshift.com/container-platform/4.4/installing/installing_gcp/installing-restricted-networks-gcp.html#installation-gcp-regions_installing-restricted-networks-gcp https://docs.openshift.com/container-platform/4.4/installing/installing_aws/installing-aws-account.html#installation-aws-regions_installing-aws-account https://docs.openshift.com/container-platform/4.4/installing/installing_azure/installing-azure-account.html
New bugzilla will need to be created to update the doc when new regions are supported but not before we are sure we can handle the new regions.
The AWS regions can be used modifying the install-config.yaml. They will not show up in the IPI survey until AMIs have been published to them.
jstuever could you please clarify what changes were made to the installer? Was any new region now available for AWS? How about GCP? Azure? Thanks.
Will do. Did know the platforms are so separate.
The installer from linux-4.6.0-0.nightly-2020-07-17-043219 is showing the mentioned new regions for GCP. I am not able to create a cluster in those new regions yet due to: FATAL failed to fetch Cluster: failed to fetch dependency of "Cluster": failed to generate asset "Platform Quota Check": error(MissingQuota): could not find any quota information for compute.googleapis.com/cpus, could not find any quota information for compute.googleapis.com/regional_static_addresses Checking with team lead to see what needs to be changed in our project to proceed. As mentioned, AWS will have new region showed up after AMI is published (still trying to find out who can do that). Azure will be handled separately.
Edited the title to remove AWS so that this ticket is for code changes for GCP only. For AWS, the understanding is that when AMI is published, the installer will pick up the new region. Not sure who will be publishing it - will work on figuring it out separately. Tested linux-4.6.0-0.nightly-2020-07-17-043219, the new regions don't work with this: FATAL failed to fetch Cluster: failed to fetch dependency of "Cluster": failed to generate asset "Platform Quota Check": error(MissingQuota): could not find any quota information for compute.googleapis.com/cpus, could not find any quota information for compute.googleapis.com/regional_static_addresses Here is a PR to unblock new regions (at minimum, it is believed to be why asia-southeast2 doesn't work): https://github.com/openshift/installer/pull/3916 Will retest.
Tried 07-18-135114 with us-west-4 and got this: ERROR ERROR Error: Error waiting for instance to create: Quota 'SSD_TOTAL_GB' exceeded. Limit: 500.0 in region us-west4. ERROR ERROR ERROR on ../../tmp/openshift-install-818994188/master/main.tf line 31, in resource "google_compute_instance" "master": ERROR 31: resource "google_compute_instance" "master" { ERROR ERROR FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change Looks like there is a 500GB ssd_total_gb quota for us-west-4 in our project. Checking to see if we can get it increased too allow us to successfully create a cluster. How much disk is required in total in a standard IPI install?
Found the answer: 896GB during install, 128 after bootstrap is removed.
Quota has been increased and we got pass the last point where we failed. But install still fail with this: DEBUG module.bootstrap.google_compute_instance_group.bootstrap[0]: Creation complete after 13s [id=projects/openshift-qe/zones/us-west4-a/instanceGroups/tszegcp72220a-9dskn-bootstrap] ERROR ERROR Error: Error creating service account: Post "https://iam.googleapis.com/v1/projects/openshift-qe/serviceAccounts?alt=json&prettyPrint=false": oauth2: cannot fetch token: Post "https://oauth2.googleapis.com/token": dial tcp 172.217.164.170:443: connect: no route to host ERROR ERROR on ../../../../../tmp/openshift-install-702665375/iam/main.tf line 1, in resource "google_service_account" "worker-node-sa": ERROR 1: resource "google_service_account" "worker-node-sa" { ERROR ERROR ERROR ERROR Error: Error creating service account: Post "https://iam.googleapis.com/v1/projects/openshift-qe/serviceAccounts?alt=json&prettyPrint=false": dial tcp 172.217.164.170:443: connect: no route to host ERROR ERROR on ../../../../../tmp/openshift-install-702665375/master/main.tf line 1, in resource "google_service_account" "master-node-sa": ERROR 1: resource "google_service_account" "master-node-sa" { ERROR ERROR FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change Is there some other changes needed to allow us to install to those new regions? (this one is us-west4)
That looks like a basic network routing problem between the host running the installer and oauth2.googleapis.com (172.217.164.170 at the time).
I retried and managed to bring up a cluster in us-west4. Closing this one as fixed. Will continue to try installing in other new regions. Thank you!
Defect to update 4.6 doc to include new regions: https://bugzilla.redhat.com/show_bug.cgi?id=1868788
Actually, this page https://docs.openshift.com/container-platform/4.5/installing/installing_gcp/installing-gcp-account.html#installation-gcp-regions_installing-gcp-account lists supported regions and don't match what we allow now.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196