Bug 2039689

Summary: [IPI on Alibabacloud] Pay-by-specification NAT is no longer supported
Product: OpenShift Container Platform Reporter: husun <husun>
Component: InstallerAssignee: aos-install
Installer sub component: openshift-installer QA Contact: Jianli Wei <jiwei>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: unspecified CC: padillon
Version: 4.10   
Target Milestone: ---   
Target Release: 4.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-03-10 16:38:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description husun 2022-01-12 09:05:12 UTC 
Thanks for opening a bug report!
Before hitting the button, please fill in as much of the template below as you can.
If you leave out information, it's harder to help you.
Be ready for follow-up questions, and please respond in a timely manner.
If we can't reproduce a bug we might close your issue.
If we're wrong, PLEASE feel free to reopen it and explain why.

Version:

./bin/openshift-install unreleased-master-5473-g28cfc831cee01eb503a2340b4d5365fd281bf867
built from commit 28cfc831cee01eb503a2340b4d5365fd281bf867
release image registry.ci.openshift.org/origin/release:4.10
release architecture amd64

Platform: Alibabacloud

#Please specify the platform type: aws, libvirt, openstack or baremetal etc.

Please specify: IPI
* IPI (automated install with `openshift-install`. If you don't know, then it's IPI)
* UPI (semi-manual installation on customized infrastructure)

What happened?

Pay-by-specification NAT is no longer supported. Newly purchased pay-as-you-go NAT gateways only support the pay-by-CU metering method


#Enter text here.

#See the troubleshooting documentation (https://github.com/openshift/installer/blob/master/docs/user/troubleshooting.md) for ideas about what information to collect.

#For example, 

# If the installer fails to create resources (https://github.com/openshift/installer/blob/master/docs/user/troubleshooting.md#installer-fails-to-create-resources), attach the relevant portions of your `.openshift_install.log.`
# If the installer fails to bootstrap the cluster (https://github.com/openshift/installer/blob/master/docs/user/troubleshootingbootstrap.md), attach the bootstrap log bundle.
# If the installer fails to complete installation after bootstrapping completes (https://github.com/openshift/installer/blob/master/docs/user/troubleshooting.md#installer-fails-to-initialize-the-cluster), attach the must-gather log bundle using `oc adm must-gather`

# Always at least include the `.openshift_install.log`

What did you expect to happen?

Create a cluster normally

#Enter text here.

How to reproduce it (as minimally and precisely as possible)?
Always

$ your-commands-here

Anything else we need to know?

./bin/openshift-install create cluster

ERROR                                              
ERROR Error: [ERROR] terraform-provider-alicloud/alicloud/resource_alicloud_nat_gateway.go:241: Resource alicloud_nat_gateway CreateNatGateway Failed!!! [SDK alibaba-cloud-sdk-go ERROR]: 
ERROR SDKError:                                    
ERROR    Code: Forbidden.NatPayBySpec              
ERROR    Message: code: 400, Pay-by-specification NAT is no longer supported. Newly purchased pay-as-you-go NAT gateways only support the pay-by-CU metering method. request id: FEEFA54A-08E7-57C9-8048-CB1269C80875 
ERROR    Data: {"Code":"Forbidden.NatPayBySpec","HostId":"vpc.aliyuncs.com","Message":"Pay-by-specification NAT is no longer supported. Newly purchased pay-as-you-go NAT gateways only support the pay-by-CU metering method.","Recommend":"https://error-center.aliyun.com/status/search?Keyword=Forbidden.NatPayBySpec\u0026source=PopGw","RequestId":"FEEFA54A-08E7-57C9-8048-CB1269C80875"} 
ERROR                                              
ERROR                                              
ERROR   on ../../../../../../../private/var/folders/y3/60m_c1jx4wvg8p031kvt6vvc0000gp/T/openshift-install-cluster-1825023758/vpc/nat_gateway.tf line 2, in resource "alicloud_nat_gateway" "nat_gateway": 
ERROR    2: resource "alicloud_nat_gateway" "nat_gateway" { 
ERROR                                              
ERROR                                              
FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change 

#Enter text here.
Comment 3 Jianli Wei 2022-01-14 03:55:49 UTC 
Tested IPI installation and Passed, in 4.10.0-0.nightly-2022-01-14-015144, but with CCO in manual mode and along using the work-around of bug https://bugzilla.redhat.com/show_bug.cgi?id=2035757. Mark as verified, thanks! 

$ openshift-install version
openshift-install 4.10.0-0.nightly-2022-01-14-015144
built from commit 8fca1ade5b096d9b2cd312c4599881d099439288
release image registry.ci.openshift.org/ocp/release@sha256:0aa02c29abf6555abdbc7d987e1a643a0bb25cc2e3597482dcb36575565d02bb
release architecture amd64
$ 
$ openshift-install create install-config --dir work
? SSH Public Key /home/fedora/.ssh/ali.pub
? Platform alibabacloud
? Region us-east-1
? Base Domain alicloud-qe.devcluster.openshift.com
? Cluster Name jiwei-502
? Pull Secret [? for help] ******
>$ echo 'credentialsMode: Manual' >> work/install-config.yaml
$ openshift-install create manifests --dir work
INFO Consuming Install Config from target directory
INFO Manifests created in: work/manifests and work/openshift
$ 
$ export CCO_IMAGE=$(oc adm -a pull_secret.json release info --image-for='cloud-credential-operator' registry.c
i.openshift.org/ocp/release:4.10.0-0.nightly-2022-01-14-015144)
$ oc image extract ${CCO_IMAGE} --file="/usr/bin/ccoctl" -a pull_secret.json
$ chmod u+x ccoctl
$ oc adm -a pull_secret.json release extract --credentials-requests --cloud=alibabacloud --to="cco-credrequests
" registry.ci.openshift.org/ocp/release:4.10.0-0.nightly-2022-01-14-015144
$ ls cco-credrequests/ -l
total 16
-rw-rw-r--. 1 fedora fedora  767 Jan 14 02:50 0000_30_machine-api-operator_00_credentials-request.yaml
-rw-rw-r--. 1 fedora fedora 2113 Jan 14 02:50 0000_50_cluster-image-registry-operator_01-registry-credentials-request-alibaba.yaml
-rw-rw-r--. 1 fedora fedora  763 Jan 14 02:50 0000_50_cluster-ingress-operator_00-ingress-credentials-request.yaml
-rw-rw-r--. 1 fedora fedora 1517 Jan 14 02:50 0000_50_cluster-storage-operator_03_credentials_request_alibaba.yaml
$ ./ccoctl alibabacloud create-ram-users --region us-east-1 --name $(awk '/infrastructureName:/{print $2}' work
/manifests/cluster-infrastructure-02-config.yml) --credentials-requests-dir cco-credrequests --output-dir cco-manifests
2022/01/14 02:55:18 Created RAM User: jiwei-502-bwvwt-openshift-machine-api-alibabacloud-credentials
2022/01/14 02:55:19 Ready for creating new ram policy jiwei-502-bwvwt-openshift-machine-api-alibabacloud-credentials-policy-policy
2022/01/14 02:55:19 RAM policy jiwei-502-bwvwt-openshift-machine-api-alibabacloud-credentials-policy-policy has created
2022/01/14 02:55:19 Policy jiwei-502-bwvwt-openshift-machine-api-alibabacloud-credentials-policy-policy has attached on user jiwei-502-b
wvwt-openshift-machine-api-alibabacloud-credentials
2022/01/14 02:55:20 Created access keys for RAM User: jiwei-502-bwvwt-openshift-machine-api-alibabacloud-credentials
2022/01/14 02:55:20 Saved credentials configuration to: cco-manifests/manifests/openshift-machine-api-alibabacloud-credentials-credentia
ls.yaml
2022/01/14 02:55:20 Created RAM User: jiwei-502-bwvwt-openshift-image-registry-installer-cloud-credent
2022/01/14 02:55:21 Ready for creating new ram policy jiwei-502-bwvwt-openshift-image-registry-installer-cloud-credentials-policy-policy
2022/01/14 02:55:21 RAM policy jiwei-502-bwvwt-openshift-image-registry-installer-cloud-credentials-policy-policy has created
2022/01/14 02:55:21 Policy jiwei-502-bwvwt-openshift-image-registry-installer-cloud-credentials-policy-policy has attached on user jiwei
-502-bwvwt-openshift-image-registry-installer-cloud-credent
2022/01/14 02:55:22 Created access keys for RAM User: jiwei-502-bwvwt-openshift-image-registry-installer-cloud-credent
2022/01/14 02:55:22 Saved credentials configuration to: cco-manifests/manifests/openshift-image-registry-installer-cloud-credentials-credentials.yaml
2022/01/14 02:55:22 Created RAM User: jiwei-502-bwvwt-openshift-ingress-operator-cloud-credentials
2022/01/14 02:55:23 Ready for creating new ram policy jiwei-502-bwvwt-openshift-ingress-operator-cloud-credentials-policy-policy
2022/01/14 02:55:23 RAM policy jiwei-502-bwvwt-openshift-ingress-operator-cloud-credentials-policy-policy has created
2022/01/14 02:55:24 Policy jiwei-502-bwvwt-openshift-ingress-operator-cloud-credentials-policy-policy has attached on user jiwei-502-bwvwt-openshift-ingress-operator-cloud-credentials
2022/01/14 02:55:24 Created access keys for RAM User: jiwei-502-bwvwt-openshift-ingress-operator-cloud-credentials
2022/01/14 02:55:24 Saved credentials configuration to: cco-manifests/manifests/openshift-ingress-operator-cloud-credentials-credentials.yaml
2022/01/14 02:55:24 Created RAM User: jiwei-502-bwvwt-openshift-cluster-csi-drivers-alibaba-disk-crede
2022/01/14 02:55:25 Ready for creating new ram policy jiwei-502-bwvwt-openshift-cluster-csi-drivers-alibaba-disk-credentials-policy-policy
2022/01/14 02:55:25 RAM policy jiwei-502-bwvwt-openshift-cluster-csi-drivers-alibaba-disk-credentials-policy-policy has created
2022/01/14 02:55:26 Policy jiwei-502-bwvwt-openshift-cluster-csi-drivers-alibaba-disk-credentials-policy-policy has attached on user jiwei-502-bwvwt-openshift-cluster-csi-drivers-alibaba-disk-crede
2022/01/14 02:55:26 Created access keys for RAM User: jiwei-502-bwvwt-openshift-cluster-csi-drivers-alibaba-disk-crede
2022/01/14 02:55:26 Saved credentials configuration to: cco-manifests/manifests/openshift-cluster-csi-drivers-alibaba-disk-credentials-credentials.yaml
$ tree cco-manifests/
cco-manifests/
└── manifests
    ├── openshift-cluster-csi-drivers-alibaba-disk-credentials-credentials.yaml
    ├── openshift-image-registry-installer-cloud-credentials-credentials.yaml
    ├── openshift-ingress-operator-cloud-credentials-credentials.yaml
    └── openshift-machine-api-alibabacloud-credentials-credentials.yaml

1 directory, 4 files
$ cp cco-manifests/manifests/*.yaml work/manifests/
$ 
$ openshift-install create cluster --dir work --log-level info
INFO Consuming OpenShift Install (Manifests) from target directory
INFO Consuming Openshift Manifests from target directory
INFO Consuming Worker Machines from target directory
INFO Consuming Master Machines from target directory
INFO Consuming Common Manifests from target directory
INFO Creating infrastructure resources...
INFO Waiting up to 20m0s (until 3:20AM) for the Kubernetes API at https://api.jiwei-502.alicloud-qe.devcluster.openshift.com:6443...
INFO API v1.23.0+dba670a up
INFO Waiting up to 30m0s (until 3:32AM) for bootstrapping to complete...
INFO Destroying the bootstrap resources...
INFO Waiting up to 40m0s (until 3:55AM) for the cluster at https://api.jiwei-502.alicloud-qe.devcluster.openshift.com:6443 to initialize...
E0114 03:23:57.704252  400511 reflector.go:138] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *v1.ClusterVersion: failed to list *v1.ClusterVersion: Get "https://api.jiwei-502.alicloud-qe.devcluster.openshift.com:6443/apis/config.openshift.io/v1/clusterversions?fieldSelector=metadata.name%3Dversion&resourceVersion=21130": dial tcp 47.253.194.71:6443: connect: connection refused
E0114 03:24:03.459209  400511 reflector.go:138] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *v1.ClusterVersion: failed to list *v1.ClusterVersion: Get "https://api.jiwei-502.alicloud-qe.devcluster.openshift.com:6443/apis/config.openshift.io/v1/clusterversions?fieldSelector=metadata.name%3Dversion&resourceVersion=21130": dial tcp 47.253.194.71:6443: connect: connection refused
E0114 03:24:14.360142  400511 reflector.go:138] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *v1.ClusterVersion: failed to list *v1.ClusterVersion: Get "https://api.jiwei-502.alicloud-qe.devcluster.openshift.com:6443/apis/config.openshift.io/v1/clusterversions?fieldSelector=metadata.name%3Dversion&resourceVersion=21130": dial tcp 47.253.194.71:6443: connect: connection refused
E0114 03:24:35.036814  400511 reflector.go:138] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *v1.ClusterVersion: failed to list *v1.ClusterVersion: Get "https://api.jiwei-502.alicloud-qe.devcluster.openshift.com:6443/apis/config.openshift.io/v1/clusterversions?fieldSelector=metadata.name%3Dversion&resourceVersion=21130": dial tcp 47.253.194.71:6443: connect: connection refused
INFO Waiting up to 10m0s (until 3:45AM) for the openshift-console route to be created...
INFO Install complete!
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/home/fedora/work/auth/kubeconfig'
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.jiwei-502.alicloud-qe.devcluster.openshift.com
INFO Login to the console with user: "kubeadmin", and password: "WYs23-DA7Gy-JyXAS-uTEQw"
INFO Time elapsed: 38m58s
$ 
$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.0-0.nightly-2022-01-14-015144   True        False         13m     Cluster version is 4.10.0-0.nightly-2022-01-14-015144
$ oc get nodes
NAME                                      STATUS   ROLES    AGE   VERSION
jiwei-502-bwvwt-master-0                  Ready    master   24m   v1.23.0+dba670a
jiwei-502-bwvwt-master-1                  Ready    master   43m   v1.23.0+dba670a
jiwei-502-bwvwt-master-2                  Ready    master   44m   v1.23.0+dba670a
jiwei-502-bwvwt-worker-us-east-1a-cdfmq   Ready    worker   28m   v1.23.0+dba670a
jiwei-502-bwvwt-worker-us-east-1b-hmcf7   Ready    worker   28m   v1.23.0+dba670a
jiwei-502-bwvwt-worker-us-east-1b-tdfkh   Ready    worker   28m   v1.23.0+dba670a
$ 
$ oc get co
NAME                                       VERSION                              AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.10.0-0.nightly-2022-01-14-015144   True        False         False      16m
baremetal                                  4.10.0-0.nightly-2022-01-14-015144   True        False         False      41m
cloud-controller-manager                   4.10.0-0.nightly-2022-01-14-015144   True        False         False      44m
cloud-credential                           4.10.0-0.nightly-2022-01-14-015144   True        False         False      40m     
cluster-autoscaler                         4.10.0-0.nightly-2022-01-14-015144   True        False         False      41m     
config-operator                            4.10.0-0.nightly-2022-01-14-015144   True        False         False      42m     
console                                    4.10.0-0.nightly-2022-01-14-015144   True        False         False      14m     
csi-snapshot-controller                    4.10.0-0.nightly-2022-01-14-015144   True        False         False      42m     
dns                                        4.10.0-0.nightly-2022-01-14-015144   True        False         False      41m     
etcd                                       4.10.0-0.nightly-2022-01-14-015144   True        False         False      40m     
image-registry                             4.10.0-0.nightly-2022-01-14-015144   True        False         False      28m     
ingress                                    4.10.0-0.nightly-2022-01-14-015144   True        False         False      22m     
insights                                   4.10.0-0.nightly-2022-01-14-015144   True        False         False      20m     
kube-apiserver                             4.10.0-0.nightly-2022-01-14-015144   True        False         False      19m     
kube-controller-manager                    4.10.0-0.nightly-2022-01-14-015144   True        False         False      39m     
kube-scheduler                             4.10.0-0.nightly-2022-01-14-015144   True        False         False      37m     
kube-storage-version-migrator              4.10.0-0.nightly-2022-01-14-015144   True        False         False      42m     
machine-api                                4.10.0-0.nightly-2022-01-14-015144   True        False         False      37m     
machine-approver                           4.10.0-0.nightly-2022-01-14-015144   True        False         False      41m     
machine-config                             4.10.0-0.nightly-2022-01-14-015144   True        False         False      41m     
marketplace                                4.10.0-0.nightly-2022-01-14-015144   True        False         False      41m     
monitoring                                 4.10.0-0.nightly-2022-01-14-015144   True        False         False      16m     
network                                    4.10.0-0.nightly-2022-01-14-015144   True        False         False      42m     
node-tuning                                4.10.0-0.nightly-2022-01-14-015144   True        False         False      41m     
openshift-apiserver                        4.10.0-0.nightly-2022-01-14-015144   True        False         False      16m     
openshift-controller-manager               4.10.0-0.nightly-2022-01-14-015144   True        False         False      28m     
openshift-samples                          4.10.0-0.nightly-2022-01-14-015144   True        False         False      16m     
operator-lifecycle-manager                 4.10.0-0.nightly-2022-01-14-015144   True        False         False      41m
operator-lifecycle-manager-catalog         4.10.0-0.nightly-2022-01-14-015144   True        False         False      41m
operator-lifecycle-manager-packageserver   4.10.0-0.nightly-2022-01-14-015144   True        False         False      31m
service-ca                                 4.10.0-0.nightly-2022-01-14-015144   True        False         False      42m
storage                                    4.10.0-0.nightly-2022-01-14-015144   True        False         False      41m
$
Comment 6 errata-xmlrpc 2022-03-10 16:38:43 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056