Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2100746

Summary: [IPI on Alibabacloud] unexpected "User not authorized" on RemoveBackendServers from slb during destroying bootstrap resources
Product: OpenShift Container Platform Reporter: Jianli Wei <jiwei>
Component: InstallerAssignee: OCP Installer <ocp-installer>
Installer sub component: openshift-installer QA Contact: Jianli Wei <jiwei>
Status: CLOSED DEFERRED Docs Contact:
Severity: high    
Priority: high CC: beth.white, gpei, howang, padillon
Version: 4.11   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
Cause: With resourceGroupID specified in install-config.yaml, and using a RAM user who has the minimum required permissions. Consequence: Error during destroying bootstrap resources. Workaround (if any): Do not specify resourceGroupID in install-config.yaml. Result: OCP installation would fail due to the error.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2022-10-14 15:10:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jianli Wei 2022-06-24 07:44:51 UTC 
Version:
$ openshift-install version
openshift-install 4.11.0-0.nightly-2022-06-23-092832
built from commit 7cdf85d8df9a454c4de2297c5b5d4ae7b06fe96e
release image registry.ci.openshift.org/ocp/release@sha256:a901f4e94f74af13a5227130c7b8d2e4b71ee35753ead592a475e88c36eff3d5
release architecture amd64

Platform: alibabacloud

Please specify: IPI

What happened?
With resourceGroupID specified in install-config.yaml, and using a RAM user who has the minimum required permissions (see the custom policy JSON file in https://docs.openshift.com/container-platform/4.10/installing/installing_alibaba/manually-creating-alibaba-ram.html#manually-creating-alibaba-ram-user_manually-creating-alibaba-ram) for OCP installation (of "Alibaba Cloud Account" scope), the intsallation would fail during destroying bootstrap resources with error "User not authorized to operate on the specified resource" on RemoveBackendServers from slb.

What did you expect to happen?
The installation should succeed.

How to reproduce it (as minimally and precisely as possible)?
Always.

Anything else we need to know?
>FYI if without specifiying a resource group, using the same RAM user could get successful OCP installation.

$ openshift-install create install-config --dir work3
? SSH Public Key /home/fedora/.ssh/openshift-qe.pub
? Platform alibabacloud
? Region us-east-1
? Base Domain alicloud-cn.devcluster.openshift.com
? Cluster Name jiwei-0624-04
? Pull Secret [? for help] *****
INFO Install-Config created in: work3
$ vim work3/install-config.yaml 
$ yq-3.3.0 r work3/install-config.yaml platform
alibabacloud:
  region: us-east-1
  resourceGroupID: rg-aekzg4dlbv6dajq
$ yq-3.3.0 r work3/install-config.yaml credentialsMode
Manual
$ yq-3.3.0 r work3/install-config.yaml metadata
creationTimestamp: null
name: jiwei-0624-04
$ yq-3.3.0 r work3/install-config.yaml baseDomain
alicloud-cn.devcluster.openshift.com
$ openshift-install create manifests --dir work3
INFO Consuming Install Config from target directory 
INFO Manifests created in: work3/manifests and work3/openshift 
$ 
$ yq-3.3.0 r work3/install-config.yaml platform
alibabacloud:
  region: us-east-1
  resourceGroupID: rg-aekzg4dlbv6dajq
$ yq-3.3.0 r work3/install-config.yaml credentialsMode
Manual
$ yq-3.3.0 r work3/install-config.yaml metadata
creationTimestamp: null
name: jiwei-0624-04
$ yq-3.3.0 r work3/install-config.yaml baseDomain
alicloud-cn.devcluster.openshift.com
$ openshift-install create manifests --dir work3
INFO Consuming Install Config from target directory
INFO Manifests created in: work3/manifests and work3/openshift
$
>Run 'ccoctl' to create the required RAM users...
$ ls -l work3/manifests/*credentials.yaml
-rw-------. 1 fedora fedora 292 Jun 24 06:18 work3/manifests/openshift-cluster-csi-drivers-alibaba-disk-credentials-credentials.yaml
-rw-------. 1 fedora fedora 290 Jun 24 06:18 work3/manifests/openshift-image-registry-installer-cloud-credentials-credentials.yaml
-rw-------. 1 fedora fedora 282 Jun 24 06:18 work3/manifests/openshift-ingress-operator-cloud-credentials-credentials.yaml
-rw-------. 1 fedora fedora 284 Jun 24 06:18 work3/manifests/openshift-machine-api-alibabacloud-credentials-credentials.yaml
$
$ openshift-install create cluster --dir work3
INFO Consuming OpenShift Install (Manifests) from target directory
INFO Consuming Worker Machines from target directory
INFO Consuming Openshift Manifests from target directory
INFO Consuming Common Manifests from target directory
INFO Consuming Master Machines from target directory
INFO Creating infrastructure resources...
INFO Waiting up to 20m0s (until 6:42AM) for the Kubernetes API at https://api.jiwei-0624-04.alicloud-cn.devcluster.openshift.com:6443... 
INFO API v1.24.0+284d62a up
INFO Waiting up to 30m0s (until 6:54AM) for bootstrapping to complete...
INFO Destroying the bootstrap resources...
ERROR
ERROR Error: [ERROR] terraform-provider-alicloud/alicloud/resource_alicloud_slb_backend_server.go:173: Resource lb-0xixuyh3e8qxihh5e3lef RemoveBackendServers Failed!!! [SDK alibaba-cloud-sdk-go ERROR]:
ERROR SDK.ServerError
ERROR ErrorCode: Forbidden
ERROR Recommend: https://next.api.aliyun.com/troubleshoot?q=Forbidden&product=Slb
ERROR RequestId: 5076EA29-52C8-546B-8FA0-CA486C47D766
ERROR Message: User not authorized to operate on the specified resource.
ERROR
ERROR   with alicloud_slb_backend_server.slb_attach_controlplane[1],
ERROR   on main.tf line 13, in resource "alicloud_slb_backend_server" "slb_attach_controlplane":
ERROR   13: resource "alicloud_slb_backend_server" "slb_attach_controlplane" {
ERROR
ERROR
ERROR Error: [ERROR] terraform-provider-alicloud/alicloud/resource_alicloud_slb_backend_server.go:173: Resource lb-0ximpzxo80cf8trgqcsn2 RemoveBackendServers Failed!!! [SDK alibaba-cloud-sdk-go ERROR]:
ERROR SDK.ServerError
ERROR ErrorCode: Forbidden 
ERROR Recommend: https://next.api.aliyun.com/troubleshoot?q=Forbidden&product=Slb
ERROR RequestId: EA9E7529-F577-55E0-88E5-D602907285A5
ERROR Message: User not authorized to operate on the specified resource.
ERROR
ERROR   with alicloud_slb_backend_server.slb_attach_controlplane[0],
ERROR   on main.tf line 13, in resource "alicloud_slb_backend_server" "slb_attach_controlplane":
ERROR   13: resource "alicloud_slb_backend_server" "slb_attach_controlplane" {
ERROR
FATAL failed disabling bootstrap load balancing: failed to apply Terraform: exit status 1
FATAL
FATAL Error: [ERROR] terraform-provider-alicloud/alicloud/resource_alicloud_slb_backend_server.go:173: Resource lb-0xixuyh3e8qxihh5e3lef RemoveBackendServers Failed!!! [SDK alibaba-cloud-sdk-go ERROR]:
FATAL SDK.ServerError
FATAL ErrorCode: Forbidden 
FATAL Recommend: https://next.api.aliyun.com/troubleshoot?q=Forbidden&product=Slb
FATAL RequestId: 5076EA29-52C8-546B-8FA0-CA486C47D766
FATAL Message: User not authorized to operate on the specified resource.
FATAL
FATAL   with alicloud_slb_backend_server.slb_attach_controlplane[1],
FATAL   on main.tf line 13, in resource "alicloud_slb_backend_server" "slb_attach_controlplane":
FATAL   13: resource "alicloud_slb_backend_server" "slb_attach_controlplane" {
FATAL
FATAL
FATAL Error: [ERROR] terraform-provider-alicloud/alicloud/resource_alicloud_slb_backend_server.go:173: Resource lb-0ximpzxo80cf8trgqcsn2 RemoveBackendServers Failed!!! [SDK alibaba-cloud-sdk-go ERROR]:
FATAL SDK.ServerError
FATAL ErrorCode: Forbidden 
FATAL Recommend: https://next.api.aliyun.com/troubleshoot?q=Forbidden&product=Slb
FATAL RequestId: EA9E7529-F577-55E0-88E5-D602907285A5
FATAL Message: User not authorized to operate on the specified resource.
FATAL
FATAL   with alicloud_slb_backend_server.slb_attach_controlplane[0],
FATAL   on main.tf line 13, in resource "alicloud_slb_backend_server" "slb_attach_controlplane":
FATAL   13: resource "alicloud_slb_backend_server" "slb_attach_controlplane" {
FATAL
FATAL
$
Comment 1 Beth White 2022-10-14 15:10:04 UTC 
Cloned to Jira project https://issues.redhat.com/browse/OCPBUGS-2376