Bug 2096692 - [IPI on Alibabacloud] some resources (eni, security group, slb, oss bucket) are not put into the specified resource group
Summary: [IPI on Alibabacloud] some resources (eni, security group, slb, oss bucket) a...
Keywords:
Status: CLOSED DEFERRED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.11
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: OCP Installer
QA Contact: Jianli Wei
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-06-14 08:39 UTC by Jianli Wei
Modified: 2023-03-09 01:21 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
Cause: Some Alibabacloud services not working very well in terms of resource group support. Consequence: Some resources created by OCP installer, including the ENI and the security group of the Internet NAT Gateway, the OSS bucket of image-registry, and Ingress SLB, are put into the default resource group. Workaround (if any): n/a Result:
Clone Of:
Environment:
Last Closed: 2023-03-09 01:21:28 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Jianli Wei 2022-06-14 08:39:28 UTC
Version:
./openshift-install 4.11.0-0.nightly-2022-06-11-054027
built from commit e7724f20e2384e50e73bf8d1dce2a5eb52c1fdab
release image registry.ci.openshift.org/ocp/release@sha256:158f720226a184361c116b7bfa6c14637c283098459db3f131634ce05d7195e7
release architecture amd64

Platform: alibabacloud

Please specify: IPI

What happened?
With "resourceGroupID" specified (see below example snippet) in install-config.yaml, some resources, including 1 eni and 1 security group (for a nat gateway), 1 oss bucket and 1 slb, are put into the default resource group, rather than the specified one. 

platform:
  alibabacloud:
    region: cn-zhangjiakou
    resourceGroupID: rg-aek2wky7lxk4f5y

Note that the OSS bucket issue has a bug while its https://bugzilla.redhat.com/show_bug.cgi?id=2039304#c3 says Alibaba will add the support in April 2022, so mention it again here.

What did you expect to happen?
All resources of the cluster should be put into the specified resource group (if any).

How to reproduce it (as minimally and precisely as possible)?
Always.

Anything else we need to know?

>FYI the cluster resources which were put into the default resource group (i.e. "rg-acfnw6kdej3hyai"):
$ aliyun resourcemanager ListResources --ResourceGroupId rg-acfnw6kdej3hyai --Region cn-zhangjiakou --endpoint resourcemanager.cn-zhangjiakou.aliyuncs.com --PageSize 100 --output cols=CreateDate,ResourceType,Service,ResourceId rows=Resources.Resource[]
CreateDate                | ResourceType  | Service | ResourceId
----------                | ------------  | ------- | ----------
>2022-06-14T14:14:21+08:00 | eni           | ecs     | eni-8vbelaywtnd2qmba1ij6
2022-06-14T14:16:25+08:00 | eni           | ecs     | eni-8vbfrp1skeu21z4fxbpv
>2022-06-14T14:14:20+08:00 | securitygroup | ecs     | sg-8vbdu24ar7vg3n669bb6
2022-06-14T14:16:25+08:00 | securitygroup | ecs     | sg-8vbi1f8ddrryec1uwu1s
2022-06-14T14:17:34+08:00 | bucket        | oss     | weinliu4113-wklwj-bootstrap
>2022-06-14T14:34:00+08:00 | bucket        | oss     | jiwei-0614-02-pql8b-image-registry-cn-zhangjiakou-sbyafweixjvq
>2022-06-14T14:30:29+08:00 | loadbalancer  | slb     | lb-pn7xev3qlh4dyo77th0t5

$ 
$ aliyun vpc DescribeNatGateways --RegionId cn-zhangjiakou --endpoint vpc.cn-zhangjiakou.aliyuncs.com --VpcId vpc-8vb3j3qa39gh21co3ckv4 --output cols=CreationTime,Name,NatGatewayId,NatGatewayPrivateInfo.EniInstanceId rows=NatGateways.NatGateway[]
CreationTime | Name| NatGatewayId| NatGatewayPrivateInfo.EniInstanceId
------------ | ----| ------------| -----------------------------------
2022-06-14T06:14:18Z | jiwei-0614-02-pql8b-ngw | ngw-8vbiq3jcrk3is571u2dwk | eni-8vbelaywtnd2qmba1ij6

$ 
$ aliyun ecs DescribeSecurityGroups --RegionId cn-zhangjiakou --endpoint ecs.cn-zhangjiakou.aliyuncs.com --VpcId vpc-8vb3j3qa39gh21co3ckv4 --output cols=CreationTime,SecurityGroupId,SecurityGroupName,ResourceGroupId rows=SecurityGroups.SecurityGroup[]
CreationTime | SecurityGroupId | SecurityGroupName| ResourceGroupId
------------ | --------------- | -----------------| ---------------
2022-06-14T06:14:20Z | sg-8vbdu24ar7vg3n669bb6 | ngw-8vbiq3jcrk3is571u2dwk_security_group | 
2022-06-14T06:14:05Z | sg-8vba3jdemw3vhj5w6n15 | jiwei-0614-02-pql8b-sg-master| rg-aek2wky7lxk4f5y
2022-06-14T06:14:04Z | sg-8vbb1lwld0k1w42qm7d5 | jiwei-0614-02-pql8b-sg-worker| rg-aek2wky7lxk4f5y

$ 
$ ossutil bucket-tagging --method get oss://jiwei-0614-02-pql8b-image-registry-cn-zhangjiakou-sbyafweixjvq --endpoint oss-cn-zhangjiakou.aliyuncs.com
index tag key tag value
---------------------------------------------------
0 "GISV""ocp"
1 "Name""jiwei-0614-02-pql8b-image-registry"
2 "kubernetes.io/cluster/jiwei-0614-02-pql8b" "owned"
3 "sigs.k8s.io/cloud-provider-alibaba/origin" "ocp"



0.200506(s) elapsed
$ 
$ aliyun slb DescribeLoadBalancers --RegionId cn-zhangjiakou --endpoint slb.cn-zhangjiakou.aliyuncs.com --Tags "[{'TagKey': 'ack.aliyun.com', 'Tagvalue': 'jiwei-0614-02-pql8b'}]" --output cols=CreateTime,LoadBalancerId,LoadBalancerName,Address,AddressType,ResourceGroupId rows=LoadBalancers.LoadBalancer[]
CreateTime| LoadBalancerId | LoadBalancerName | Address | AddressType | ResourceGroupId
----------| -------------- | ---------------- | ------- | ----------- | ---------------
2022-06-14T14:30Z | lb-pn7xev3qlh4dyo77th0t5 | a48907b2146534d1ab47644ee3f6cc0d | 39.103.202.37 | internet| rg-acfnw6kdej3hyai

$ aliyun slb DescribeLoadBalancers --RegionId cn-zhangjiakou --endpoint slb.cn-zhangjiakou.aliyuncs.com --Tags "[{'TagKey': 'kubernetes.io/cluster/jiwei-0614-02-pql8b', 'Tagvalue': 'owned'}]" --output cols=CreateTime,LoadBalancerId,LoadBalancerName,Address,AddressType,ResourceGroupId rows=LoadBalancers.LoadBalancer[]
CreateTime| LoadBalancerId | LoadBalancerName | Address | AddressType | ResourceGroupId
----------| -------------- | ---------------- | ------- | ----------- | ---------------
2022-06-14T14:14Z | lb-pn7kc9xenftrk5prr7zn2 | jiwei-0614-02-pql8b-slb-internal | 10.0.64.146 | intranet| rg-aek2wky7lxk4f5y
2022-06-14T14:14Z | lb-pn73stphc1u8j53lbb6zu | jiwei-0614-02-pql8b-slb-external | 39.99.253.248 | internet| rg-aek2wky7lxk4f5y

$ 

FYI the flexy-install job: https://mastern-jenkins-csb-openshift-qe.apps.ocp-c1.prod.psi.redhat.com/job/ocp-common/job/Flexy-install/111704/

Comment 1 Jianli Wei 2022-06-16 02:22:46 UTC
Set to High severity as CEE has customer concerned with it.

Comment 2 Jie Wu 2022-06-16 11:15:57 UTC
As per Alibabacloud engineer answer, the ingress router resource can be exposed in the specific resource group,
it needs to add the following annotations into the service.

metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-resource-group-id: "rg-xxxx"

Reference Link:
https://help.aliyun.com/document_detail/86531.html?spm=5176.10695662.1996646101.searchclickresult.87d74fdf8ZwPdN

Example:
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-resource-group-id: "rg-xxxx"
  name: nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Comment 3 Jianli Wei 2022-06-17 09:14:59 UTC
FYI the work-around, where "rg-acfnw6kdej3hyai" is the default resource group, and "rg-aek2c4huej7f3ni" is the cluster's resource group:

$ aliyun resourcemanager ListResources --ResourceGroupId rg-acfnw6kdej3hyai --Region ap-northeast-1 --endpoint resourcemanager.ap-northeast-1.aliyuncs.com --PageSize 100 --output cols=CreateDate,ResourceType,Service,ResourceId rows=Resources.Resource[]
CreateDate                | ResourceType  | Service | ResourceId
----------                | ------------  | ------- | ----------
2022-06-17T15:18:06+08:00 | eni           | ecs     | eni-6we4hzheokv39s2601e4
2022-06-17T15:18:05+08:00 | securitygroup | ecs     | sg-6we4hzheokv39s26e2zd
2022-06-17T15:42:54+08:00 | bucket        | oss     | jiwei-ali-0617-v77pg-image-registry-ap-northeast-1-itlmiinylbo
2022-06-17T15:30:36+08:00 | loadbalancer  | slb     | lb-0iw5uki0agqe128mz9gyg

$

Login alibabacloud web console, in the Resource Group page, edit the cluster's resource group, then "Transfer In" the above resources from the default resource group, all of them could be moved to the cluster's resource group successfully.

$ aliyun resourcemanager ListResources --ResourceGroupId rg-aek2c4huej7f3ni --Region ap-northeast-1
 --endpoint resourcemanager.ap-northeast-1.aliyuncs.com --PageSize 100 --output cols=CreateDate,ResourceType,Service,ResourceId rows=Res
ources.Resource[]
CreateDate                | ResourceType  | Service | ResourceId
----------                | ------------  | ------- | ----------
2022-06-17T15:18:05+08:00 | disk          | ecs     | d-6we4hzheokv39s25wvdo
2022-06-17T15:18:05+08:00 | disk          | ecs     | d-6web7p31yzd854iold26
2022-06-17T15:18:05+08:00 | disk          | ecs     | d-6web7p31yzd854iold27
2022-06-17T15:31:06+08:00 | disk          | ecs     | d-6we791q0d3dxoanaqyuw
2022-06-17T15:33:59+08:00 | disk          | ecs     | d-6weiltmay2tlvyvpnzhx
2022-06-17T15:18:05+08:00 | eni           | ecs     | eni-6weiltmay2tlvqzkifuv
2022-06-17T15:18:05+08:00 | eni           | ecs     | eni-6we3nql0i2eqffphr077
2022-06-17T15:18:05+08:00 | eni           | ecs     | eni-6we010bkz1u6xcfh3nsh
2022-06-17T15:18:06+08:00 | eni           | ecs     | eni-6we4hzheokv39s2601e4
2022-06-17T15:31:06+08:00 | eni           | ecs     | eni-6we010bkz1u6xickftqc
2022-06-17T15:33:59+08:00 | eni           | ecs     | eni-6web7p31yzd85cesvjk4
2022-06-17T15:18:05+08:00 | instance      | ecs     | i-6weiltmay2tlvqzf7wnp
2022-06-17T15:18:05+08:00 | instance      | ecs     | i-6we9b32dx7lf5jox070v
2022-06-17T15:18:05+08:00 | instance      | ecs     | i-6we791q0d3dxo4q26kz8
2022-06-17T15:31:06+08:00 | instance      | ecs     | i-6we1cz4b7ehcrzh7dhmm
2022-06-17T15:33:59+08:00 | instance      | ecs     | i-6we3nql0i2eqfnlryo6f
2022-06-17T15:17:51+08:00 | securitygroup | ecs     | sg-6web7p31yzd854itajo0
2022-06-17T15:17:51+08:00 | securitygroup | ecs     | sg-6weiltmay2tlvqzofxpb
2022-06-17T15:18:05+08:00 | securitygroup | ecs     | sg-6we4hzheokv39s26e2zd
2022-06-17T15:17:48+08:00 | eip           | eip     | eip-6wets2s0vjo6rjkb815fl
2022-06-17T15:42:54+08:00 | bucket        | oss     | jiwei-ali-0617-v77pg-image-registry-ap-northeast-1-itlmiinylbo
2022-06-17T15:17:51+08:00 | loadbalancer  | slb     | lb-0iwhvc1ea6wlr275ut9ua
2022-06-17T15:18:06+08:00 | loadbalancer  | slb     | lb-0iwmnqp6iw4yxf96iznkr
2022-06-17T15:30:36+08:00 | loadbalancer  | slb     | lb-0iw5uki0agqe128mz9gyg
2022-06-17T15:18:03+08:00 | natgateway    | vpc     | ngw-6wew5594w05advw5rvqne
2022-06-17T15:17:46+08:00 | vpc           | vpc     | vpc-6we0phc99lmmmwq217j5u

$

Comment 5 Bin Hu 2022-09-12 04:06:01 UTC
Any progress, kindly expedite, thanks

Comment 6 Shiftzilla 2023-03-09 01:21:28 UTC
OpenShift has moved to Jira for its defect tracking! This bug can now be found in the OCPBUGS project in Jira.

https://issues.redhat.com/browse/OCPBUGS-9318


Note You need to log in before you can comment on or make changes to this bug.