Bug 2060687

Summary:	Azure Ci: SubscriptionDoesNotSupportZone - does not support availability zones at location 'westus'
Product:	OpenShift Container Platform	Reporter:	Kirsten Garrison <kgarriso>
Component:	Installer	Assignee:	Nobody <nobody>
Installer sub component:	openshift-installer	QA Contact:	MayXu <maxu>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	medium	CC:	akaris, lxia, wking
Version:	4.11	Keywords:	Regression
Target Milestone:	---
Target Release:	4.11.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-08-10 10:52:11 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Kirsten Garrison 2022-03-04 01:01:35 UTC

Description of problem:
In e2e-agnostic-upgrade (https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_machine-config-operator/2973/pull-ci-openshift-machine-config-operator-master-e2e-agnostic-upgrade/1499485185937248256) seeing an azure infra issue:

msg=Error: creating/updating Load Balancer: (Name "ci-op-kkp8ky8k-57c36-2nkkg-internal" / Resource Group "ci-op-kkp8ky8k-57c36-2nkkg-rg"): network.LoadBalancersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="SubscriptionDoesNotSupportZone" Message="Cannot create/move resource /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-kkp8ky8k-57c36-2nkkg-rg/providers/Microsoft.Network/loadBalancers/ci-op-kkp8ky8k-57c36-2nkkg-internal/frontendIPConfigurations/internal-lb-ip-v4 which uses zone 1 since subscription d38f1e38-4bed-438e-b227-833f997adf6a does not support zone 1 in location westus." Details=[]
level=error
level=error msg=  with azurerm_lb.internal,
level=error msg=  on internal-lb.tf line 6, in resource "azurerm_lb" "internal":
level=error msg=   6: resource "azurerm_lb" "internal" {
level=error
level=error
level=error msg=Error: creating/updating Public Ip Address: (Name "ci-op-kkp8ky8k-57c36-2nkkg-pip-v4" / Resource Group "ci-op-kkp8ky8k-57c36-2nkkg-rg"): network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="LocationNotSupportAvailabilityZones" Message="The resource 'Microsoft.Network/publicIPAddresses/ci-op-kkp8ky8k-57c36-2nkkg-pip-v4' does not support availability zones at location 'westus'."
level=error
level=error msg=  with azurerm_public_ip.cluster_public_ip_v4[0],
level=error msg=  on public-lb.tf line 14, in resource "azurerm_public_ip" "cluster_public_ip_v4":
level=error msg=  14: resource "azurerm_public_ip" "cluster_public_ip_v4" {
level=error
level=fatal msg=failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: exit status 1
level=fatal
level=fatal msg=Error: creating/updating Load Balancer: (Name "ci-op-kkp8ky8k-57c36-2nkkg-internal" / Resource Group "ci-op-kkp8ky8k-57c36-2nkkg-rg"): network.LoadBalancersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="SubscriptionDoesNotSupportZone" Message="Cannot create/move resource /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-kkp8ky8k-57c36-2nkkg-rg/providers/Microsoft.Network/loadBalancers/ci-op-kkp8ky8k-57c36-2nkkg-internal/frontendIPConfigurations/internal-lb-ip-v4 which uses zone 1 since subscription d38f1e38-4bed-438e-b227-833f997adf6a does not support zone 1 in location westus." Details=[]
level=fatal
level=fatal msg=  with azurerm_lb.internal,
level=fatal msg=  on internal-lb.tf line 6, in resource "azurerm_lb" "internal":
level=fatal msg=   6: resource "azurerm_lb" "internal" {
level=fatal
level=fatal
level=fatal msg=Error: creating/updating Public Ip Address: (Name "ci-op-kkp8ky8k-57c36-2nkkg-pip-v4" / Resource Group "ci-op-kkp8ky8k-57c36-2nkkg-rg"): network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="LocationNotSupportAvailabilityZones" Message="The resource 'Microsoft.Network/publicIPAddresses/ci-op-kkp8ky8k-57c36-2nkkg-pip-v4' does not support availability zones at location 'westus'."
level=fatal
level=fatal msg=  with azurerm_public_ip.cluster_public_ip_v4[0],
level=fatal msg=  on public-lb.tf line 14, in resource "azurerm_public_ip" "cluster_public_ip_v4":
level=fatal msg=  14: resource "azurerm_public_ip" "cluster_public_ip_v4" {
level=fatal
level=fatal

Impact:
$ w3m -dump -cols 200 'https://search.ci.openshift.org/?maxAge=96h&type=build-log&search=SubscriptionDoesNotSupportZone' | grep 'failures match' | sort
periodic-ci-openshift-release-master-ci-4.11-e2e-azure (all) - 4 runs, 75% failed, 67% of failures match = 50% impact
periodic-ci-openshift-release-master-ci-4.11-e2e-azure-ovn-upgrade (all) - 23 runs, 74% failed, 6% of failures match = 4% impact
periodic-ci-openshift-release-master-nightly-4.11-e2e-azure (all) - 4 runs, 25% failed, 100% of failures match = 25% impact
pull-ci-kubevirt-hyperconverged-cluster-operator-main-hco-e2e-image-index-azure (all) - 23 runs, 39% failed, 11% of failures match = 4% impact
pull-ci-kubevirt-hyperconverged-cluster-operator-main-hco-e2e-image-index-sno-azure (all) - 23 runs, 30% failed, 43% of failures match = 13% impact
pull-ci-kubevirt-hyperconverged-cluster-operator-main-hco-e2e-kv-smoke-azure (all) - 24 runs, 38% failed, 22% of failures match = 8% impact
pull-ci-kubevirt-hyperconverged-cluster-operator-main-hco-e2e-upgrade-index-azure (all) - 28 runs, 61% failed, 12% of failures match = 7% impact
...

Comment 2 Matthew Staebler 2022-03-04 01:13:35 UTC

With the latest version of the azurerm terraform provider, it may be that the availabilityZones field on the frontend_ip_configuration needs to be explicitly set to "No-Zone" when installing to a non-zonal region.

See https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/lb#availability_zone

Comment 3 Matthew Staebler 2022-03-04 21:36:04 UTC

See https://github.com/hashicorp/terraform-provider-azurerm/issues/12756 for details of this issue in the azurerm terraform provider.

Comment 4 MayXu 2022-03-24 06:57:18 UTC

Pre-Merge Test done 
install on westus region succeed with 4.11.0-0.ci.test-2022-03-24-033534-ci-ln-bskvb2t-latest

Comment 6 MayXu 2022-03-28 02:53:14 UTC

verified on registry.ci.openshift.org/ocp/release:4.11.0-0.nightly-2022-03-27-195539
install cluster on westus succeed.

Comment 14 errata-xmlrpc 2022-08-10 10:52:11 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069