Description of problem: In e2e-agnostic-upgrade (https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_machine-config-operator/2973/pull-ci-openshift-machine-config-operator-master-e2e-agnostic-upgrade/1499485185937248256) seeing an azure infra issue: msg=Error: creating/updating Load Balancer: (Name "ci-op-kkp8ky8k-57c36-2nkkg-internal" / Resource Group "ci-op-kkp8ky8k-57c36-2nkkg-rg"): network.LoadBalancersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="SubscriptionDoesNotSupportZone" Message="Cannot create/move resource /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-kkp8ky8k-57c36-2nkkg-rg/providers/Microsoft.Network/loadBalancers/ci-op-kkp8ky8k-57c36-2nkkg-internal/frontendIPConfigurations/internal-lb-ip-v4 which uses zone 1 since subscription d38f1e38-4bed-438e-b227-833f997adf6a does not support zone 1 in location westus." Details=[] level=error level=error msg= with azurerm_lb.internal, level=error msg= on internal-lb.tf line 6, in resource "azurerm_lb" "internal": level=error msg= 6: resource "azurerm_lb" "internal" { level=error level=error level=error msg=Error: creating/updating Public Ip Address: (Name "ci-op-kkp8ky8k-57c36-2nkkg-pip-v4" / Resource Group "ci-op-kkp8ky8k-57c36-2nkkg-rg"): network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="LocationNotSupportAvailabilityZones" Message="The resource 'Microsoft.Network/publicIPAddresses/ci-op-kkp8ky8k-57c36-2nkkg-pip-v4' does not support availability zones at location 'westus'." level=error level=error msg= with azurerm_public_ip.cluster_public_ip_v4[0], level=error msg= on public-lb.tf line 14, in resource "azurerm_public_ip" "cluster_public_ip_v4": level=error msg= 14: resource "azurerm_public_ip" "cluster_public_ip_v4" { level=error level=fatal msg=failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: exit status 1 level=fatal level=fatal msg=Error: creating/updating Load Balancer: (Name "ci-op-kkp8ky8k-57c36-2nkkg-internal" / Resource Group "ci-op-kkp8ky8k-57c36-2nkkg-rg"): network.LoadBalancersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="SubscriptionDoesNotSupportZone" Message="Cannot create/move resource /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-kkp8ky8k-57c36-2nkkg-rg/providers/Microsoft.Network/loadBalancers/ci-op-kkp8ky8k-57c36-2nkkg-internal/frontendIPConfigurations/internal-lb-ip-v4 which uses zone 1 since subscription d38f1e38-4bed-438e-b227-833f997adf6a does not support zone 1 in location westus." Details=[] level=fatal level=fatal msg= with azurerm_lb.internal, level=fatal msg= on internal-lb.tf line 6, in resource "azurerm_lb" "internal": level=fatal msg= 6: resource "azurerm_lb" "internal" { level=fatal level=fatal level=fatal msg=Error: creating/updating Public Ip Address: (Name "ci-op-kkp8ky8k-57c36-2nkkg-pip-v4" / Resource Group "ci-op-kkp8ky8k-57c36-2nkkg-rg"): network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="LocationNotSupportAvailabilityZones" Message="The resource 'Microsoft.Network/publicIPAddresses/ci-op-kkp8ky8k-57c36-2nkkg-pip-v4' does not support availability zones at location 'westus'." level=fatal level=fatal msg= with azurerm_public_ip.cluster_public_ip_v4[0], level=fatal msg= on public-lb.tf line 14, in resource "azurerm_public_ip" "cluster_public_ip_v4": level=fatal msg= 14: resource "azurerm_public_ip" "cluster_public_ip_v4" { level=fatal level=fatal Impact: $ w3m -dump -cols 200 'https://search.ci.openshift.org/?maxAge=96h&type=build-log&search=SubscriptionDoesNotSupportZone' | grep 'failures match' | sort periodic-ci-openshift-release-master-ci-4.11-e2e-azure (all) - 4 runs, 75% failed, 67% of failures match = 50% impact periodic-ci-openshift-release-master-ci-4.11-e2e-azure-ovn-upgrade (all) - 23 runs, 74% failed, 6% of failures match = 4% impact periodic-ci-openshift-release-master-nightly-4.11-e2e-azure (all) - 4 runs, 25% failed, 100% of failures match = 25% impact pull-ci-kubevirt-hyperconverged-cluster-operator-main-hco-e2e-image-index-azure (all) - 23 runs, 39% failed, 11% of failures match = 4% impact pull-ci-kubevirt-hyperconverged-cluster-operator-main-hco-e2e-image-index-sno-azure (all) - 23 runs, 30% failed, 43% of failures match = 13% impact pull-ci-kubevirt-hyperconverged-cluster-operator-main-hco-e2e-kv-smoke-azure (all) - 24 runs, 38% failed, 22% of failures match = 8% impact pull-ci-kubevirt-hyperconverged-cluster-operator-main-hco-e2e-upgrade-index-azure (all) - 28 runs, 61% failed, 12% of failures match = 7% impact ...
With the latest version of the azurerm terraform provider, it may be that the availabilityZones field on the frontend_ip_configuration needs to be explicitly set to "No-Zone" when installing to a non-zonal region. See https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/lb#availability_zone
See https://github.com/hashicorp/terraform-provider-azurerm/issues/12756 for details of this issue in the azurerm terraform provider.
Pre-Merge Test done install on westus region succeed with 4.11.0-0.ci.test-2022-03-24-033534-ci-ln-bskvb2t-latest
verified on registry.ci.openshift.org/ocp/release:4.11.0-0.nightly-2022-03-27-195539 install cluster on westus succeed.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5069