Thanks for opening a bug report! Before hitting the button, please fill in as much of the template below as you can. If you leave out information, it's harder to help you. Be ready for follow-up questions, and please respond in a timely manner. If we can't reproduce a bug we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. Version: ./bin/openshift-install unreleased-master-5473-g28cfc831cee01eb503a2340b4d5365fd281bf867 built from commit 28cfc831cee01eb503a2340b4d5365fd281bf867 release image registry.ci.openshift.org/origin/release:4.10 release architecture amd64 Platform: Alibabacloud #Please specify the platform type: aws, libvirt, openstack or baremetal etc. Please specify: IPI * IPI (automated install with `openshift-install`. If you don't know, then it's IPI) * UPI (semi-manual installation on customized infrastructure) What happened? Pay-by-specification NAT is no longer supported. Newly purchased pay-as-you-go NAT gateways only support the pay-by-CU metering method #Enter text here. #See the troubleshooting documentation (https://github.com/openshift/installer/blob/master/docs/user/troubleshooting.md) for ideas about what information to collect. #For example, # If the installer fails to create resources (https://github.com/openshift/installer/blob/master/docs/user/troubleshooting.md#installer-fails-to-create-resources), attach the relevant portions of your `.openshift_install.log.` # If the installer fails to bootstrap the cluster (https://github.com/openshift/installer/blob/master/docs/user/troubleshootingbootstrap.md), attach the bootstrap log bundle. # If the installer fails to complete installation after bootstrapping completes (https://github.com/openshift/installer/blob/master/docs/user/troubleshooting.md#installer-fails-to-initialize-the-cluster), attach the must-gather log bundle using `oc adm must-gather` # Always at least include the `.openshift_install.log` What did you expect to happen? Create a cluster normally #Enter text here. How to reproduce it (as minimally and precisely as possible)? Always $ your-commands-here Anything else we need to know? ./bin/openshift-install create cluster ERROR ERROR Error: [ERROR] terraform-provider-alicloud/alicloud/resource_alicloud_nat_gateway.go:241: Resource alicloud_nat_gateway CreateNatGateway Failed!!! [SDK alibaba-cloud-sdk-go ERROR]: ERROR SDKError: ERROR Code: Forbidden.NatPayBySpec ERROR Message: code: 400, Pay-by-specification NAT is no longer supported. Newly purchased pay-as-you-go NAT gateways only support the pay-by-CU metering method. request id: FEEFA54A-08E7-57C9-8048-CB1269C80875 ERROR Data: {"Code":"Forbidden.NatPayBySpec","HostId":"vpc.aliyuncs.com","Message":"Pay-by-specification NAT is no longer supported. Newly purchased pay-as-you-go NAT gateways only support the pay-by-CU metering method.","Recommend":"https://error-center.aliyun.com/status/search?Keyword=Forbidden.NatPayBySpec\u0026source=PopGw","RequestId":"FEEFA54A-08E7-57C9-8048-CB1269C80875"} ERROR ERROR ERROR on ../../../../../../../private/var/folders/y3/60m_c1jx4wvg8p031kvt6vvc0000gp/T/openshift-install-cluster-1825023758/vpc/nat_gateway.tf line 2, in resource "alicloud_nat_gateway" "nat_gateway": ERROR 2: resource "alicloud_nat_gateway" "nat_gateway" { ERROR ERROR FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change #Enter text here.
Tested IPI installation and Passed, in 4.10.0-0.nightly-2022-01-14-015144, but with CCO in manual mode and along using the work-around of bug https://bugzilla.redhat.com/show_bug.cgi?id=2035757. Mark as verified, thanks! $ openshift-install version openshift-install 4.10.0-0.nightly-2022-01-14-015144 built from commit 8fca1ade5b096d9b2cd312c4599881d099439288 release image registry.ci.openshift.org/ocp/release@sha256:0aa02c29abf6555abdbc7d987e1a643a0bb25cc2e3597482dcb36575565d02bb release architecture amd64 $ $ openshift-install create install-config --dir work ? SSH Public Key /home/fedora/.ssh/ali.pub ? Platform alibabacloud ? Region us-east-1 ? Base Domain alicloud-qe.devcluster.openshift.com ? Cluster Name jiwei-502 ? Pull Secret [? for help] ****** >$ echo 'credentialsMode: Manual' >> work/install-config.yaml $ openshift-install create manifests --dir work INFO Consuming Install Config from target directory INFO Manifests created in: work/manifests and work/openshift $ $ export CCO_IMAGE=$(oc adm -a pull_secret.json release info --image-for='cloud-credential-operator' registry.c i.openshift.org/ocp/release:4.10.0-0.nightly-2022-01-14-015144) $ oc image extract ${CCO_IMAGE} --file="/usr/bin/ccoctl" -a pull_secret.json $ chmod u+x ccoctl $ oc adm -a pull_secret.json release extract --credentials-requests --cloud=alibabacloud --to="cco-credrequests " registry.ci.openshift.org/ocp/release:4.10.0-0.nightly-2022-01-14-015144 $ ls cco-credrequests/ -l total 16 -rw-rw-r--. 1 fedora fedora 767 Jan 14 02:50 0000_30_machine-api-operator_00_credentials-request.yaml -rw-rw-r--. 1 fedora fedora 2113 Jan 14 02:50 0000_50_cluster-image-registry-operator_01-registry-credentials-request-alibaba.yaml -rw-rw-r--. 1 fedora fedora 763 Jan 14 02:50 0000_50_cluster-ingress-operator_00-ingress-credentials-request.yaml -rw-rw-r--. 1 fedora fedora 1517 Jan 14 02:50 0000_50_cluster-storage-operator_03_credentials_request_alibaba.yaml $ ./ccoctl alibabacloud create-ram-users --region us-east-1 --name $(awk '/infrastructureName:/{print $2}' work /manifests/cluster-infrastructure-02-config.yml) --credentials-requests-dir cco-credrequests --output-dir cco-manifests 2022/01/14 02:55:18 Created RAM User: jiwei-502-bwvwt-openshift-machine-api-alibabacloud-credentials 2022/01/14 02:55:19 Ready for creating new ram policy jiwei-502-bwvwt-openshift-machine-api-alibabacloud-credentials-policy-policy 2022/01/14 02:55:19 RAM policy jiwei-502-bwvwt-openshift-machine-api-alibabacloud-credentials-policy-policy has created 2022/01/14 02:55:19 Policy jiwei-502-bwvwt-openshift-machine-api-alibabacloud-credentials-policy-policy has attached on user jiwei-502-b wvwt-openshift-machine-api-alibabacloud-credentials 2022/01/14 02:55:20 Created access keys for RAM User: jiwei-502-bwvwt-openshift-machine-api-alibabacloud-credentials 2022/01/14 02:55:20 Saved credentials configuration to: cco-manifests/manifests/openshift-machine-api-alibabacloud-credentials-credentia ls.yaml 2022/01/14 02:55:20 Created RAM User: jiwei-502-bwvwt-openshift-image-registry-installer-cloud-credent 2022/01/14 02:55:21 Ready for creating new ram policy jiwei-502-bwvwt-openshift-image-registry-installer-cloud-credentials-policy-policy 2022/01/14 02:55:21 RAM policy jiwei-502-bwvwt-openshift-image-registry-installer-cloud-credentials-policy-policy has created 2022/01/14 02:55:21 Policy jiwei-502-bwvwt-openshift-image-registry-installer-cloud-credentials-policy-policy has attached on user jiwei -502-bwvwt-openshift-image-registry-installer-cloud-credent 2022/01/14 02:55:22 Created access keys for RAM User: jiwei-502-bwvwt-openshift-image-registry-installer-cloud-credent 2022/01/14 02:55:22 Saved credentials configuration to: cco-manifests/manifests/openshift-image-registry-installer-cloud-credentials-credentials.yaml 2022/01/14 02:55:22 Created RAM User: jiwei-502-bwvwt-openshift-ingress-operator-cloud-credentials 2022/01/14 02:55:23 Ready for creating new ram policy jiwei-502-bwvwt-openshift-ingress-operator-cloud-credentials-policy-policy 2022/01/14 02:55:23 RAM policy jiwei-502-bwvwt-openshift-ingress-operator-cloud-credentials-policy-policy has created 2022/01/14 02:55:24 Policy jiwei-502-bwvwt-openshift-ingress-operator-cloud-credentials-policy-policy has attached on user jiwei-502-bwvwt-openshift-ingress-operator-cloud-credentials 2022/01/14 02:55:24 Created access keys for RAM User: jiwei-502-bwvwt-openshift-ingress-operator-cloud-credentials 2022/01/14 02:55:24 Saved credentials configuration to: cco-manifests/manifests/openshift-ingress-operator-cloud-credentials-credentials.yaml 2022/01/14 02:55:24 Created RAM User: jiwei-502-bwvwt-openshift-cluster-csi-drivers-alibaba-disk-crede 2022/01/14 02:55:25 Ready for creating new ram policy jiwei-502-bwvwt-openshift-cluster-csi-drivers-alibaba-disk-credentials-policy-policy 2022/01/14 02:55:25 RAM policy jiwei-502-bwvwt-openshift-cluster-csi-drivers-alibaba-disk-credentials-policy-policy has created 2022/01/14 02:55:26 Policy jiwei-502-bwvwt-openshift-cluster-csi-drivers-alibaba-disk-credentials-policy-policy has attached on user jiwei-502-bwvwt-openshift-cluster-csi-drivers-alibaba-disk-crede 2022/01/14 02:55:26 Created access keys for RAM User: jiwei-502-bwvwt-openshift-cluster-csi-drivers-alibaba-disk-crede 2022/01/14 02:55:26 Saved credentials configuration to: cco-manifests/manifests/openshift-cluster-csi-drivers-alibaba-disk-credentials-credentials.yaml $ tree cco-manifests/ cco-manifests/ └── manifests ├── openshift-cluster-csi-drivers-alibaba-disk-credentials-credentials.yaml ├── openshift-image-registry-installer-cloud-credentials-credentials.yaml ├── openshift-ingress-operator-cloud-credentials-credentials.yaml └── openshift-machine-api-alibabacloud-credentials-credentials.yaml 1 directory, 4 files $ cp cco-manifests/manifests/*.yaml work/manifests/ $ $ openshift-install create cluster --dir work --log-level info INFO Consuming OpenShift Install (Manifests) from target directory INFO Consuming Openshift Manifests from target directory INFO Consuming Worker Machines from target directory INFO Consuming Master Machines from target directory INFO Consuming Common Manifests from target directory INFO Creating infrastructure resources... INFO Waiting up to 20m0s (until 3:20AM) for the Kubernetes API at https://api.jiwei-502.alicloud-qe.devcluster.openshift.com:6443... INFO API v1.23.0+dba670a up INFO Waiting up to 30m0s (until 3:32AM) for bootstrapping to complete... INFO Destroying the bootstrap resources... INFO Waiting up to 40m0s (until 3:55AM) for the cluster at https://api.jiwei-502.alicloud-qe.devcluster.openshift.com:6443 to initialize... E0114 03:23:57.704252 400511 reflector.go:138] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *v1.ClusterVersion: failed to list *v1.ClusterVersion: Get "https://api.jiwei-502.alicloud-qe.devcluster.openshift.com:6443/apis/config.openshift.io/v1/clusterversions?fieldSelector=metadata.name%3Dversion&resourceVersion=21130": dial tcp 47.253.194.71:6443: connect: connection refused E0114 03:24:03.459209 400511 reflector.go:138] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *v1.ClusterVersion: failed to list *v1.ClusterVersion: Get "https://api.jiwei-502.alicloud-qe.devcluster.openshift.com:6443/apis/config.openshift.io/v1/clusterversions?fieldSelector=metadata.name%3Dversion&resourceVersion=21130": dial tcp 47.253.194.71:6443: connect: connection refused E0114 03:24:14.360142 400511 reflector.go:138] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *v1.ClusterVersion: failed to list *v1.ClusterVersion: Get "https://api.jiwei-502.alicloud-qe.devcluster.openshift.com:6443/apis/config.openshift.io/v1/clusterversions?fieldSelector=metadata.name%3Dversion&resourceVersion=21130": dial tcp 47.253.194.71:6443: connect: connection refused E0114 03:24:35.036814 400511 reflector.go:138] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *v1.ClusterVersion: failed to list *v1.ClusterVersion: Get "https://api.jiwei-502.alicloud-qe.devcluster.openshift.com:6443/apis/config.openshift.io/v1/clusterversions?fieldSelector=metadata.name%3Dversion&resourceVersion=21130": dial tcp 47.253.194.71:6443: connect: connection refused INFO Waiting up to 10m0s (until 3:45AM) for the openshift-console route to be created... INFO Install complete! INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/home/fedora/work/auth/kubeconfig' INFO Access the OpenShift web-console here: https://console-openshift-console.apps.jiwei-502.alicloud-qe.devcluster.openshift.com INFO Login to the console with user: "kubeadmin", and password: "WYs23-DA7Gy-JyXAS-uTEQw" INFO Time elapsed: 38m58s $ $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.10.0-0.nightly-2022-01-14-015144 True False 13m Cluster version is 4.10.0-0.nightly-2022-01-14-015144 $ oc get nodes NAME STATUS ROLES AGE VERSION jiwei-502-bwvwt-master-0 Ready master 24m v1.23.0+dba670a jiwei-502-bwvwt-master-1 Ready master 43m v1.23.0+dba670a jiwei-502-bwvwt-master-2 Ready master 44m v1.23.0+dba670a jiwei-502-bwvwt-worker-us-east-1a-cdfmq Ready worker 28m v1.23.0+dba670a jiwei-502-bwvwt-worker-us-east-1b-hmcf7 Ready worker 28m v1.23.0+dba670a jiwei-502-bwvwt-worker-us-east-1b-tdfkh Ready worker 28m v1.23.0+dba670a $ $ oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.10.0-0.nightly-2022-01-14-015144 True False False 16m baremetal 4.10.0-0.nightly-2022-01-14-015144 True False False 41m cloud-controller-manager 4.10.0-0.nightly-2022-01-14-015144 True False False 44m cloud-credential 4.10.0-0.nightly-2022-01-14-015144 True False False 40m cluster-autoscaler 4.10.0-0.nightly-2022-01-14-015144 True False False 41m config-operator 4.10.0-0.nightly-2022-01-14-015144 True False False 42m console 4.10.0-0.nightly-2022-01-14-015144 True False False 14m csi-snapshot-controller 4.10.0-0.nightly-2022-01-14-015144 True False False 42m dns 4.10.0-0.nightly-2022-01-14-015144 True False False 41m etcd 4.10.0-0.nightly-2022-01-14-015144 True False False 40m image-registry 4.10.0-0.nightly-2022-01-14-015144 True False False 28m ingress 4.10.0-0.nightly-2022-01-14-015144 True False False 22m insights 4.10.0-0.nightly-2022-01-14-015144 True False False 20m kube-apiserver 4.10.0-0.nightly-2022-01-14-015144 True False False 19m kube-controller-manager 4.10.0-0.nightly-2022-01-14-015144 True False False 39m kube-scheduler 4.10.0-0.nightly-2022-01-14-015144 True False False 37m kube-storage-version-migrator 4.10.0-0.nightly-2022-01-14-015144 True False False 42m machine-api 4.10.0-0.nightly-2022-01-14-015144 True False False 37m machine-approver 4.10.0-0.nightly-2022-01-14-015144 True False False 41m machine-config 4.10.0-0.nightly-2022-01-14-015144 True False False 41m marketplace 4.10.0-0.nightly-2022-01-14-015144 True False False 41m monitoring 4.10.0-0.nightly-2022-01-14-015144 True False False 16m network 4.10.0-0.nightly-2022-01-14-015144 True False False 42m node-tuning 4.10.0-0.nightly-2022-01-14-015144 True False False 41m openshift-apiserver 4.10.0-0.nightly-2022-01-14-015144 True False False 16m openshift-controller-manager 4.10.0-0.nightly-2022-01-14-015144 True False False 28m openshift-samples 4.10.0-0.nightly-2022-01-14-015144 True False False 16m operator-lifecycle-manager 4.10.0-0.nightly-2022-01-14-015144 True False False 41m operator-lifecycle-manager-catalog 4.10.0-0.nightly-2022-01-14-015144 True False False 41m operator-lifecycle-manager-packageserver 4.10.0-0.nightly-2022-01-14-015144 True False False 31m service-ca 4.10.0-0.nightly-2022-01-14-015144 True False False 42m storage 4.10.0-0.nightly-2022-01-14-015144 True False False 41m $
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056