Bug 2041750 - [IPI on Alibabacloud] trying "create install-config" with region "cn-wulanchabu (China (Ulanqab))" (or "ap-southeast-6 (Philippines (Manila))", "cn-guangzhou (China (Guangzhou))") failed due to invalid endpoint
Summary: [IPI on Alibabacloud] trying "create install-config" with region "cn-wulancha...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.10
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.10.0
Assignee: aos-install
QA Contact: Jianli Wei
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-01-18 08:25 UTC by Jianli Wei
Modified: 2022-03-10 16:40 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-10 16:40:33 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 5571 0 None open Bug 2041750: [Alibaba] fix endpoint error in some regions 2022-01-24 14:56:50 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:40:44 UTC

Description Jianli Wei 2022-01-18 08:25:58 UTC
Version:
$ openshift-install version
openshift-install 4.10.0-0.ci.test-2022-01-18-015330-ci-ln-c2rvwfb-latest
built from commit c4bc155f6de2494b9baca767cd74dc665e2ec468
release image registry.build01.ci.openshift.org/ci-ln-c2rvwfb/release@sha256:105a191b4183a002f36cd4421a8db27ccb1e352d20a428e3899b0da491859451
release architecture amd64

Platform: alibabacloud

Please specify:
* IPI

What happened?
"create install-config" failed with region "cn-wulanchabu", with error "InvalidOperation.NotSupportedEndpoint".

What did you expect to happen?
The operation should succeed.

How to reproduce it (as minimally and precisely as possible)?
Always.

Anything else we need to know?
$ rm -fr work
$ openshift-install create install-config --dir work
? SSH Public Key /home/jiwei/.ssh/openshift-qe.pub
? Platform alibabacloud
? Region cn-wulanchabu
? Base Domain alicloud-qe.devcluster.openshift.com
? Cluster Name jiwei-206
? Pull Secret [? for help] *****
FATAL failed to fetch Install Config: failed to generate asset "Install Config": [controlPlane.platform.alibabacloud.instanceType: Internal error: SDK.ServerError
FATAL ErrorCode: InvalidOperation.NotSupportedEndpoint
FATAL Recommend: https://error-center.aliyun.com/status/search?Keyword=InvalidOperation.NotSupportedEndpoint&source=PopGw
FATAL RequestId: 9B314A9C-DD5D-5728-8D7B-027430483131
FATAL Message: The specified endpoint can't operate this region. Please use API DescribeRegions to get the appropriate endpoint, or upgrade your SDK to latest version., compute[0].platform.alibabacloud.instanceType: Internal error: SDK.ServerError
FATAL ErrorCode: InvalidOperation.NotSupportedEndpoint
FATAL Recommend: https://error-center.aliyun.com/status/search?Keyword=InvalidOperation.NotSupportedEndpoint&source=PopGw
FATAL RequestId: 80D57094-A156-53D6-88B0-C07C92960F2E
FATAL Message: The specified endpoint can't operate this region. Please use API DescribeRegions to get the appropriate endpoint, or upgrade your SDK to latest version.]
$ 

$ aliyun vpc DescribeRegions | jq -c ".Regions.Region[] | select(.RegionId | contains(\"cn-wulanchabu\"))" | jq -r .RegionEndpoint
vpc.cn-wulanchabu.aliyuncs.com
$ aliyun ecs DescribeRegions | jq -c ".Regions.Region[] | select(.RegionId | contains(\"cn-wulanchabu\"))" | jq -r .RegionEndpoint
ecs.cn-wulanchabu.aliyuncs.com
$ aliyun ecs DescribeAvailableResource --DestinationResource 'InstanceType' --RegionId cn-wulanchabu --IoOptimized 'optimized' --InstanceType ecs.g6.xlarge --endpoint ecs.cn-wulanchabu.aliyuncs.com | jq -r .AvailableZones.AvailableZone[].ZoneId
cn-wulanchabu-b
cn-wulanchabu-a
cn-wulanchabu-c
$ aliyun ecs DescribeAvailableResource --DestinationResource 'InstanceType' --RegionId cn-wulanchabu --IoOptimized 'optimized' --InstanceType ecs.g6.large --endpoint ecs.cn-wulanchabu.aliyuncs.com | jq -r .AvailableZones.AvailableZone[].ZoneId
cn-wulanchabu-b
cn-wulanchabu-a
cn-wulanchabu-c
$

Comment 1 Jianli Wei 2022-01-18 09:50:19 UTC
The region "ap-southeast-6 (Philippines (Manila))" has similar issue. 

$ openshift-install create install-config --dir work
? SSH Public Key /home/jiwei/.ssh/openshift-qe.pub
? Platform alibabacloud
? Region ap-southeast-6
? Base Domain alicloud-qe.devcluster.openshift.com
? Cluster Name jiwei-manila
? Pull Secret [? for help] ************
FATAL failed to fetch Install Config: failed to generate asset "Install Config": [controlPlane.platform.alibabacloud.instanceType: Internal error: Post "http://ecs-cn-hangzhou.aliyuncs.com/?AccessKeyId=LTAI5tMF5BSS3jz9UQ1WvFPU&Action=DescribeAvailableResource&DestinationResource=InstanceType&Format=JSON&InstanceType=ecs.g6.xlarge&RegionId=ap-southeast-6&Signature=Wg06yf6M7Y%2FIrMZ5zp4ZyJ751fk%3D&SignatureMethod=HMAC-SHA1&SignatureNonce=971a0e2b59222879b8f2b7beb61d744e&SignatureType=&SignatureVersion=1.0&Timestamp=2022-01-18T09%3A16%3A43Z&Version=2014-05-26": dial tcp: i/o timeout, compute[0].platform.alibabacloud.instanceType: Internal error: SDK.ServerError
FATAL ErrorCode: InvalidOperation.NotSupportedEndpoint
FATAL Recommend: https://error-center.aliyun.com/status/search?Keyword=InvalidOperation.NotSupportedEndpoint&source=PopGw
FATAL RequestId: C2D013CA-599F-54C3-877A-367AE5E21B16
FATAL Message: The specified endpoint can't operate this region. Please use API DescribeRegions to get the appropriate endpoint, or upgrade your SDK to latest version.]
$

$ aliyun vpc DescribeRegions | jq -c ".Regions.Region[] | select(.RegionId | contains(\"ap-southeast-6\"))" | jq -r .RegionEndpoint
vpc.ap-southeast-6.aliyuncs.com
$ aliyun ecs DescribeRegions | jq -c ".Regions.Region[] | select(.RegionId | contains(\"ap-southeast-6\"))" | jq -r .RegionEndpoint
ecs.ap-southeast-6.aliyuncs.com
$ aliyun ecs DescribeAvailableResource --DestinationResource 'InstanceType' --RegionId ap-southeast-6 --IoOptimized 'optimized' --InstanceType ecs.g6.xlarge --endpoint ecs.ap-southeast-6.aliyuncs.com | jq -r .AvailableZones.AvailableZone[].ZoneId
ap-southeast-6a
$ aliyun ecs DescribeAvailableResource --DestinationResource 'InstanceType' --RegionId ap-southeast-6 --IoOptimized 'optimized' --InstanceType ecs.g6.large --endpoint ecs.ap-southeast-6.aliyuncs.com | jq -r .AvailableZones.AvailableZone[].ZoneId
ap-southeast-6a
$

Comment 2 Jianli Wei 2022-01-18 09:57:16 UTC
The region "cn-guangzhou (China (Guangzhou))" has similar issue.

$ openshift-install create install-config --dir work                                                               
? SSH Public Key /home/jiwei/.ssh/openshift-qe.pub
? Platform alibabacloud
? Region cn-guangzhou
? Base Domain alicloud-qe.devcluster.openshift.com
? Cluster Name jiwei-guangzhou
? Pull Secret [? for help] *******
FATAL failed to fetch Install Config: failed to generate asset "Install Config": [controlPlane.platform.alibabacloud.instanceType: Internal error: SDK.ServerError 
FATAL ErrorCode: InvalidOperation.NotSupportedEndpoint 
FATAL Recommend: https://error-center.aliyun.com/status/search?Keyword=InvalidOperation.NotSupportedEndpoint&source=PopGw 
FATAL RequestId: 27B16911-8B1D-5EAA-A415-0B198C0C5599 
FATAL Message: The specified endpoint can't operate this region. Please use API DescribeRegions to get the appropriate endpoint, or upgrade your SDK to latest version., compute[0].platform.alibabacloud.instanceType: Internal error: SDK.ServerError 
FATAL ErrorCode: InvalidOperation.NotSupportedEndpoint 
FATAL Recommend: https://error-center.aliyun.com/status/search?Keyword=InvalidOperation.NotSupportedEndpoint&source=PopGw 
FATAL RequestId: EA326A86-7E6E-5DBD-8F2C-796734DA9EE1 
FATAL Message: The specified endpoint can't operate this region. Please use API DescribeRegions to get the appropriate endpoint, or upgrade your SDK to latest version.] 
$ aliyun vpc DescribeRegions | jq -c ".Regions.Region[] | select(.RegionId | contains(\"cn-guangzhou\"))" | jq -r .RegionEndpoint
vpc.cn-guangzhou.aliyuncs.com
$ aliyun ecs DescribeRegions | jq -c ".Regions.Region[] | select(.RegionId | contains(\"cn-guangzhou\"))" | jq -r .RegionEndpoint
ecs.cn-guangzhou.aliyuncs.com
$

Comment 3 Matthew Staebler 2022-01-19 10:46:47 UTC
Setting this as a non-blocker for now since the installation fails early in regions that exhibit this issue. However, before we release we should either (1) fix the endpoint issue or (2) exclude the problematic regions from the selection list.

Comment 4 Brian Lu 2022-01-21 01:09:30 UTC
root cause has been found, sunhui is working on it, PR will be submitted soon.

Comment 5 husun 2022-01-24 14:54:33 UTC
Have fixed it on the PR https://github.com/openshift/installer/pull/5571

Comment 9 Jianli Wei 2022-01-29 08:46:12 UTC
Tested and now the install-config.yaml can be created successfully for the 3 regions, mark as verified. 

$ openshift-install version
openshift-install 4.10.0-0.nightly-2022-01-29-015515
built from commit 4fc9fa88c22221b6cede2456b1c33847943b75c9
release image registry.ci.openshift.org/ocp/release@sha256:b6bded497818f2e07401988576f15c62cd6fe45c385d177b50a43d6dabaf4524
release architecture amd64
$ openshift-install create install-config --dir test1
? SSH Public Key /home/fedora/.ssh/openshift-qe.pub
? Platform alibabacloud
? Region cn-wulanchabu
? Base Domain alicloud-cn.devcluster.openshift.com
? Cluster Name jiwei-620
? Pull Secret [? for help] *******
INFO Install-Config created in: test1
$ 
$ openshift-install create install-config --dir test2
? SSH Public Key /home/fedora/.ssh/openshift-qe.pub
? Platform alibabacloud
? Region ap-southeast-6
? Base Domain alicloud-cn.devcluster.openshift.com
? Cluster Name jiwei-621
? Pull Secret [? for help] *******
INFO Install-Config created in: test2
$ 
$ openshift-install create install-config --dir test3
? SSH Public Key /home/fedora/.ssh/openshift-qe.pub
? Platform alibabacloud
? Region cn-guangzhou
? Base Domain alicloud-cn.devcluster.openshift.com
? Cluster Name jiwei-622
? Pull Secret [? for help] *******
INFO Install-Config created in: test3
$

Comment 12 errata-xmlrpc 2022-03-10 16:40:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.