Version: ./openshift-install 4.11.0-0.nightly-2022-06-11-054027 built from commit e7724f20e2384e50e73bf8d1dce2a5eb52c1fdab release image registry.ci.openshift.org/ocp/release@sha256:158f720226a184361c116b7bfa6c14637c283098459db3f131634ce05d7195e7 release architecture amd64 Platform: alibabacloud Please specify: IPI What happened? With "resourceGroupID" specified (see below example snippet) in install-config.yaml, some resources, including 1 eni and 1 security group (for a nat gateway), 1 oss bucket and 1 slb, are put into the default resource group, rather than the specified one. platform: alibabacloud: region: cn-zhangjiakou resourceGroupID: rg-aek2wky7lxk4f5y Note that the OSS bucket issue has a bug while its https://bugzilla.redhat.com/show_bug.cgi?id=2039304#c3 says Alibaba will add the support in April 2022, so mention it again here. What did you expect to happen? All resources of the cluster should be put into the specified resource group (if any). How to reproduce it (as minimally and precisely as possible)? Always. Anything else we need to know? >FYI the cluster resources which were put into the default resource group (i.e. "rg-acfnw6kdej3hyai"): $ aliyun resourcemanager ListResources --ResourceGroupId rg-acfnw6kdej3hyai --Region cn-zhangjiakou --endpoint resourcemanager.cn-zhangjiakou.aliyuncs.com --PageSize 100 --output cols=CreateDate,ResourceType,Service,ResourceId rows=Resources.Resource[] CreateDate | ResourceType | Service | ResourceId ---------- | ------------ | ------- | ---------- >2022-06-14T14:14:21+08:00 | eni | ecs | eni-8vbelaywtnd2qmba1ij6 2022-06-14T14:16:25+08:00 | eni | ecs | eni-8vbfrp1skeu21z4fxbpv >2022-06-14T14:14:20+08:00 | securitygroup | ecs | sg-8vbdu24ar7vg3n669bb6 2022-06-14T14:16:25+08:00 | securitygroup | ecs | sg-8vbi1f8ddrryec1uwu1s 2022-06-14T14:17:34+08:00 | bucket | oss | weinliu4113-wklwj-bootstrap >2022-06-14T14:34:00+08:00 | bucket | oss | jiwei-0614-02-pql8b-image-registry-cn-zhangjiakou-sbyafweixjvq >2022-06-14T14:30:29+08:00 | loadbalancer | slb | lb-pn7xev3qlh4dyo77th0t5 $ $ aliyun vpc DescribeNatGateways --RegionId cn-zhangjiakou --endpoint vpc.cn-zhangjiakou.aliyuncs.com --VpcId vpc-8vb3j3qa39gh21co3ckv4 --output cols=CreationTime,Name,NatGatewayId,NatGatewayPrivateInfo.EniInstanceId rows=NatGateways.NatGateway[] CreationTime | Name| NatGatewayId| NatGatewayPrivateInfo.EniInstanceId ------------ | ----| ------------| ----------------------------------- 2022-06-14T06:14:18Z | jiwei-0614-02-pql8b-ngw | ngw-8vbiq3jcrk3is571u2dwk | eni-8vbelaywtnd2qmba1ij6 $ $ aliyun ecs DescribeSecurityGroups --RegionId cn-zhangjiakou --endpoint ecs.cn-zhangjiakou.aliyuncs.com --VpcId vpc-8vb3j3qa39gh21co3ckv4 --output cols=CreationTime,SecurityGroupId,SecurityGroupName,ResourceGroupId rows=SecurityGroups.SecurityGroup[] CreationTime | SecurityGroupId | SecurityGroupName| ResourceGroupId ------------ | --------------- | -----------------| --------------- 2022-06-14T06:14:20Z | sg-8vbdu24ar7vg3n669bb6 | ngw-8vbiq3jcrk3is571u2dwk_security_group | 2022-06-14T06:14:05Z | sg-8vba3jdemw3vhj5w6n15 | jiwei-0614-02-pql8b-sg-master| rg-aek2wky7lxk4f5y 2022-06-14T06:14:04Z | sg-8vbb1lwld0k1w42qm7d5 | jiwei-0614-02-pql8b-sg-worker| rg-aek2wky7lxk4f5y $ $ ossutil bucket-tagging --method get oss://jiwei-0614-02-pql8b-image-registry-cn-zhangjiakou-sbyafweixjvq --endpoint oss-cn-zhangjiakou.aliyuncs.com index tag key tag value --------------------------------------------------- 0 "GISV""ocp" 1 "Name""jiwei-0614-02-pql8b-image-registry" 2 "kubernetes.io/cluster/jiwei-0614-02-pql8b" "owned" 3 "sigs.k8s.io/cloud-provider-alibaba/origin" "ocp" 0.200506(s) elapsed $ $ aliyun slb DescribeLoadBalancers --RegionId cn-zhangjiakou --endpoint slb.cn-zhangjiakou.aliyuncs.com --Tags "[{'TagKey': 'ack.aliyun.com', 'Tagvalue': 'jiwei-0614-02-pql8b'}]" --output cols=CreateTime,LoadBalancerId,LoadBalancerName,Address,AddressType,ResourceGroupId rows=LoadBalancers.LoadBalancer[] CreateTime| LoadBalancerId | LoadBalancerName | Address | AddressType | ResourceGroupId ----------| -------------- | ---------------- | ------- | ----------- | --------------- 2022-06-14T14:30Z | lb-pn7xev3qlh4dyo77th0t5 | a48907b2146534d1ab47644ee3f6cc0d | 39.103.202.37 | internet| rg-acfnw6kdej3hyai $ aliyun slb DescribeLoadBalancers --RegionId cn-zhangjiakou --endpoint slb.cn-zhangjiakou.aliyuncs.com --Tags "[{'TagKey': 'kubernetes.io/cluster/jiwei-0614-02-pql8b', 'Tagvalue': 'owned'}]" --output cols=CreateTime,LoadBalancerId,LoadBalancerName,Address,AddressType,ResourceGroupId rows=LoadBalancers.LoadBalancer[] CreateTime| LoadBalancerId | LoadBalancerName | Address | AddressType | ResourceGroupId ----------| -------------- | ---------------- | ------- | ----------- | --------------- 2022-06-14T14:14Z | lb-pn7kc9xenftrk5prr7zn2 | jiwei-0614-02-pql8b-slb-internal | 10.0.64.146 | intranet| rg-aek2wky7lxk4f5y 2022-06-14T14:14Z | lb-pn73stphc1u8j53lbb6zu | jiwei-0614-02-pql8b-slb-external | 39.99.253.248 | internet| rg-aek2wky7lxk4f5y $ FYI the flexy-install job: https://mastern-jenkins-csb-openshift-qe.apps.ocp-c1.prod.psi.redhat.com/job/ocp-common/job/Flexy-install/111704/
Set to High severity as CEE has customer concerned with it.
As per Alibabacloud engineer answer, the ingress router resource can be exposed in the specific resource group, it needs to add the following annotations into the service. metadata: annotations: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-resource-group-id: "rg-xxxx" Reference Link: https://help.aliyun.com/document_detail/86531.html?spm=5176.10695662.1996646101.searchclickresult.87d74fdf8ZwPdN Example: apiVersion: v1 kind: Service metadata: annotations: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-resource-group-id: "rg-xxxx" name: nginx spec: externalTrafficPolicy: Local ports: - port: 80 protocol: TCP targetPort: 80 selector: app: nginx type: LoadBalancer
FYI the work-around, where "rg-acfnw6kdej3hyai" is the default resource group, and "rg-aek2c4huej7f3ni" is the cluster's resource group: $ aliyun resourcemanager ListResources --ResourceGroupId rg-acfnw6kdej3hyai --Region ap-northeast-1 --endpoint resourcemanager.ap-northeast-1.aliyuncs.com --PageSize 100 --output cols=CreateDate,ResourceType,Service,ResourceId rows=Resources.Resource[] CreateDate | ResourceType | Service | ResourceId ---------- | ------------ | ------- | ---------- 2022-06-17T15:18:06+08:00 | eni | ecs | eni-6we4hzheokv39s2601e4 2022-06-17T15:18:05+08:00 | securitygroup | ecs | sg-6we4hzheokv39s26e2zd 2022-06-17T15:42:54+08:00 | bucket | oss | jiwei-ali-0617-v77pg-image-registry-ap-northeast-1-itlmiinylbo 2022-06-17T15:30:36+08:00 | loadbalancer | slb | lb-0iw5uki0agqe128mz9gyg $ Login alibabacloud web console, in the Resource Group page, edit the cluster's resource group, then "Transfer In" the above resources from the default resource group, all of them could be moved to the cluster's resource group successfully. $ aliyun resourcemanager ListResources --ResourceGroupId rg-aek2c4huej7f3ni --Region ap-northeast-1 --endpoint resourcemanager.ap-northeast-1.aliyuncs.com --PageSize 100 --output cols=CreateDate,ResourceType,Service,ResourceId rows=Res ources.Resource[] CreateDate | ResourceType | Service | ResourceId ---------- | ------------ | ------- | ---------- 2022-06-17T15:18:05+08:00 | disk | ecs | d-6we4hzheokv39s25wvdo 2022-06-17T15:18:05+08:00 | disk | ecs | d-6web7p31yzd854iold26 2022-06-17T15:18:05+08:00 | disk | ecs | d-6web7p31yzd854iold27 2022-06-17T15:31:06+08:00 | disk | ecs | d-6we791q0d3dxoanaqyuw 2022-06-17T15:33:59+08:00 | disk | ecs | d-6weiltmay2tlvyvpnzhx 2022-06-17T15:18:05+08:00 | eni | ecs | eni-6weiltmay2tlvqzkifuv 2022-06-17T15:18:05+08:00 | eni | ecs | eni-6we3nql0i2eqffphr077 2022-06-17T15:18:05+08:00 | eni | ecs | eni-6we010bkz1u6xcfh3nsh 2022-06-17T15:18:06+08:00 | eni | ecs | eni-6we4hzheokv39s2601e4 2022-06-17T15:31:06+08:00 | eni | ecs | eni-6we010bkz1u6xickftqc 2022-06-17T15:33:59+08:00 | eni | ecs | eni-6web7p31yzd85cesvjk4 2022-06-17T15:18:05+08:00 | instance | ecs | i-6weiltmay2tlvqzf7wnp 2022-06-17T15:18:05+08:00 | instance | ecs | i-6we9b32dx7lf5jox070v 2022-06-17T15:18:05+08:00 | instance | ecs | i-6we791q0d3dxo4q26kz8 2022-06-17T15:31:06+08:00 | instance | ecs | i-6we1cz4b7ehcrzh7dhmm 2022-06-17T15:33:59+08:00 | instance | ecs | i-6we3nql0i2eqfnlryo6f 2022-06-17T15:17:51+08:00 | securitygroup | ecs | sg-6web7p31yzd854itajo0 2022-06-17T15:17:51+08:00 | securitygroup | ecs | sg-6weiltmay2tlvqzofxpb 2022-06-17T15:18:05+08:00 | securitygroup | ecs | sg-6we4hzheokv39s26e2zd 2022-06-17T15:17:48+08:00 | eip | eip | eip-6wets2s0vjo6rjkb815fl 2022-06-17T15:42:54+08:00 | bucket | oss | jiwei-ali-0617-v77pg-image-registry-ap-northeast-1-itlmiinylbo 2022-06-17T15:17:51+08:00 | loadbalancer | slb | lb-0iwhvc1ea6wlr275ut9ua 2022-06-17T15:18:06+08:00 | loadbalancer | slb | lb-0iwmnqp6iw4yxf96iznkr 2022-06-17T15:30:36+08:00 | loadbalancer | slb | lb-0iw5uki0agqe128mz9gyg 2022-06-17T15:18:03+08:00 | natgateway | vpc | ngw-6wew5594w05advw5rvqne 2022-06-17T15:17:46+08:00 | vpc | vpc | vpc-6we0phc99lmmmwq217j5u $
Any progress, kindly expedite, thanks
OpenShift has moved to Jira for its defect tracking! This bug can now be found in the OCPBUGS project in Jira. https://issues.redhat.com/browse/OCPBUGS-9318