OCP 4.6 supports custom endpoints, cluster could be installed on a public regions without native support for RHCOS https://github.com/openshift/enhancements/blob/master/enhancements/installer/aws-custom-region-and-endpoints.md Try to install a cluster into af-south-1, get following error: time="2020-07-30T08:57:06Z" level=debug msg="resolved AWS service route53 (af-south-1) to \"https://route53.amazonaws.com\"" time="2020-07-30T08:57:06Z" level=fatal msg="failed to fetch Common Manifests: failed to fetch dependency of \"Common Manifests\": failed to generate asset \"DNS Config\": getting public zone for \"qe.devcluster.openshift.com\": listing hosted zones: SignatureDoesNotMatch: Credential should be scoped to a valid region, not 'af-south-1'. \n\tstatus code: 403, request id: 352dc7d4-a5ae-4eb7-aa41-373c3c8f170c" install config: --- apiVersion: v1 controlPlane: architecture: amd64 hyperthreading: Enabled name: master platform: {} replicas: 3 compute: - architecture: amd64 hyperthreading: Enabled name: worker platform: {} replicas: 3 metadata: name: yunjiang-af2 platform: aws: region: af-south-1 serviceEndpoints: - name: ec2 url: https://ec2.af-south-1.amazonaws.com - name: elasticloadbalancing url: https://elasticloadbalancing.af-south-1.amazonaws.com - name: s3 url: https://s3.af-south-1.amazonaws.com - name: iam url: https://iam.af-south-1.amazonaws.com - name: tagging url: https://tagging.af-south-1.amazonaws.com - name: route53 url: https://route53.amazonaws.com pullSecret: HIDDEN networking: clusterNetwork: - cidr: 10.128.0.0/14 hostPrefix: 23 serviceNetwork: - 172.30.0.0/16 machineNetwork: - cidr: 10.0.0.0/16 networkType: OpenShiftSDN publish: External baseDomain: qe.devcluster.openshift.com sshKey: HIDDEN Version-Release number of the following components: Cluster version is 4.6.0-0.nightly-2020-07-25-091217 How reproducible: 100% Steps to Reproduce: 1. Create install config 2. Set up endpoints: aws: region: af-south-1 serviceEndpoints: - name: ec2 url: https://ec2.af-south-1.amazonaws.com - name: elasticloadbalancing url: https://elasticloadbalancing.af-south-1.amazonaws.com - name: s3 url: https://s3.af-south-1.amazonaws.com - name: iam url: https://iam.af-south-1.amazonaws.com - name: tagging url: https://tagging.af-south-1.amazonaws.com - name: route53 url: https://route53.amazonaws.com 3. create cluster Actual results: SignatureDoesNotMatch: Credential should be scoped to a valid region, not 'af-south-1'. Expected results: No errors, create cluster successfully Additional info: 4.6.0-0.nightly-2020-07-25-091217 could be installed successfully in us-east-2
verified on 4.6.0-0.nightly-2020-08-02-091622 - FAILED the original route53 problem was resolved, but got IAM problem (install log is attached), error log: time="2020-08-03T15:00:49Z" level=debug msg="module.dns.aws_route53_record.api_internal: Creation complete after 57s [id=Z00683592YKJ1PSSGQYWP_api-int.yunjiang-af03bz.qe.devcluster.openshift.com_A]" time="2020-08-03T15:00:49Z" level=error time="2020-08-03T15:00:49Z" level=error msg="Error: Error creating IAM Role yunjiang-af03bz-fr9mc-bootstrap-role: SignatureDoesNotMatch: Credential should be scoped to a valid region, not 'af-south-1'. " time="2020-08-03T15:00:49Z" level=error msg="\tstatus code: 403, request id: 4213a2c0-6b69-4731-8483-b683c7cfd00c" time="2020-08-03T15:00:49Z" level=error time="2020-08-03T15:00:49Z" level=error msg=" on ../../../../../tmp/openshift-install-787325111/bootstrap/main.tf line 51, in resource \"aws_iam_role\" \"bootstrap\":" time="2020-08-03T15:00:49Z" level=error msg=" 51: resource \"aws_iam_role\" \"bootstrap\" {" time="2020-08-03T15:00:49Z" level=error time="2020-08-03T15:00:49Z" level=error time="2020-08-03T15:00:49Z" level=error time="2020-08-03T15:00:49Z" level=error msg="Error: Error creating IAM Role yunjiang-af03bz-fr9mc-worker-role: SignatureDoesNotMatch: Credential should be scoped to a valid region, not 'af-south-1'. " time="2020-08-03T15:00:49Z" level=error msg="\tstatus code: 403, request id: b9fbb43c-cbfa-45c1-80a4-ea2a09029197" time="2020-08-03T15:00:49Z" level=error time="2020-08-03T15:00:49Z" level=error msg=" on ../../../../../tmp/openshift-install-787325111/iam/main.tf line 13, in resource \"aws_iam_role\" \"worker_role\":" time="2020-08-03T15:00:49Z" level=error msg=" 13: resource \"aws_iam_role\" \"worker_role\" {" time="2020-08-03T15:00:49Z" level=error time="2020-08-03T15:00:49Z" level=error time="2020-08-03T15:00:49Z" level=error time="2020-08-03T15:00:49Z" level=error msg="Error: Error creating IAM Role yunjiang-af03bz-fr9mc-master-role: SignatureDoesNotMatch: Credential should be scoped to a valid region, not 'af-south-1'. " time="2020-08-03T15:00:49Z" level=error msg="\tstatus code: 403, request id: 345e615d-9b91-4154-865f-3718a567588c" time="2020-08-03T15:00:49Z" level=error time="2020-08-03T15:00:49Z" level=error msg=" on ../../../../../tmp/openshift-install-787325111/master/main.tf line 17, in resource \"aws_iam_role\" \"master_role\":" time="2020-08-03T15:00:49Z" level=error msg=" 17: resource \"aws_iam_role\" \"master_role\" {" time="2020-08-03T15:00:49Z" level=error time="2020-08-03T15:00:49Z" level=error time="2020-08-03T15:00:49Z" level=error time="2020-08-03T15:00:49Z" level=error msg="Error: Error creating EIP: AddressLimitExceeded: The maximum number of addresses has been reached." time="2020-08-03T15:00:49Z" level=error msg="\tstatus code: 400, request id: f2f41aa5-d18a-44c9-8a82-b1339d578f08" time="2020-08-03T15:00:49Z" level=error time="2020-08-03T15:00:49Z" level=error msg=" on ../../../../../tmp/openshift-install-787325111/vpc/vpc-public.tf line 68, in resource \"aws_eip\" \"nat_eip\":" time="2020-08-03T15:00:49Z" level=error msg=" 68: resource \"aws_eip\" \"nat_eip\" {" time="2020-08-03T15:00:49Z" level=error time="2020-08-03T15:00:49Z" level=error time="2020-08-03T15:00:49Z" level=fatal msg="failed to fetch Cluster: failed to generate asset \"Cluster\": failed to create cluster: failed to apply Terraform: failed to complete the change" config: ... platform: aws: region: af-south-1 serviceEndpoints: - name: ec2 url: https://ec2.af-south-1.amazonaws.com - name: elasticloadbalancing url: https://elasticloadbalancing.af-south-1.amazonaws.com - name: s3 url: https://s3.af-south-1.amazonaws.com - name: iam url: https://iam.amazonaws.com - name: tagging url: https://tagging.af-south-1.amazonaws.com - name: route53 url: https://route53.amazonaws.com
Created attachment 1703722 [details] install log (IAM)
Can you try without the IAM and route53 endpoints because for public regions these are already known and probably do not need the override. ``` platform: aws: region: af-south-1 serviceEndpoints: - name: ec2 url: https://ec2.af-south-1.amazonaws.com - name: elasticloadbalancing url: https://elasticloadbalancing.af-south-1.amazonaws.com - name: s3 url: https://s3.af-south-1.amazonaws.com - name: tagging url: https://tagging.af-south-1.amazonaws.com ```
it works after removing IAM and route53 endpoints the bootstrap process completed: time="2020-08-05T08:11:43Z" level=info msg="API v4.6.0-202008031851.p0-dirty up" time="2020-08-05T08:11:43Z" level=info msg="Waiting up to 30m0s for bootstrapping to complete..." time="2020-08-05T08:27:25Z" level=debug msg="Bootstrap status: complete" time="2020-08-05T08:27:25Z" level=info msg="Destroying the bootstrap resources..." but the install process failed due to some operators error: time="2020-08-05T08:59:25Z" level=fatal msg="failed to initialize the cluster: Cluster operator console is reporting a failure: RouteHealthDegraded: failed to GET route (https://console-openshift-console.apps.yunjiang-05bug.qe.devcluster.openshift.com/health): Get \"https://console-openshift-console.apps.yunjiang-05bug.qe.devcluster.openshift.com/health\": dial tcp: lookup console-openshift-console.apps.yunjiang-05bug.qe.devcluster.openshift.com on 172.30.0.10:53: no such host" probably it is another issue, install log is attached.
Created attachment 1710489 [details] install log without IAM and route53
seems the ingress operator is affected by the service endpoints, please refer to https://bugzilla.redhat.com/show_bug.cgi?id=1866299
Please see https://bugzilla.redhat.com/show_bug.cgi?id=1866299#c2
see https://bugzilla.redhat.com/show_bug.cgi?id=1866299#c4 It works well after updating the tagging endpoint as below: - name: tagging url: https://tagging.us-east-1.amazonaws.com
*** This bug has been marked as a duplicate of bug 1866299 ***
the issue as described is still there: 1. according to the document [1], ec2/elb/s3/iam/tagging/route53 could be provided by user, if user provide above endpoints correctly (even it overrides the default), the cluster should be installed successfully. 2. according to comments [2][3], following config could work <--snip--> serviceEndpoints: - name: ec2 url: https://ec2.af-south-1.amazonaws.com - name: elasticloadbalancing url: https://elasticloadbalancing.af-south-1.amazonaws.com - name: s3 url: https://s3.af-south-1.amazonaws.com - name: tagging url: https://tagging.us-east-1.amazonaws.com <--snip--> 3. tried to install using following config (note for tagging endpoint, it's `us-east-1` as comment [2] mentioned) <--snip--> platform: aws: region: af-south-1 serviceEndpoints: - name: ec2 url: https://ec2.af-south-1.amazonaws.com - name: elasticloadbalancing url: https://elasticloadbalancing.af-south-1.amazonaws.com - name: s3 url: https://s3.af-south-1.amazonaws.com - name: iam url: https://iam.amazonaws.com - name: tagging url: https://tagging.us-east-1.amazonaws.com - name: route53 url: https://route53.amazonaws.com <--snip--> got following error: level=error msg="Error: Error creating IAM Role yunjiang-18af4-6p56w-bootstrap-role: SignatureDoesNotMatch: Credential should be scoped to a valid region, not 'af-south-1'. " Is this the same issue as bug 1866299 described? Will the 'SignatureDoesNotMatch' issue be fixed? Need Daneyon and Abhinav to confirm, thanks. [1] https://github.com/openshift/enhancements/blob/master/enhancements/installer/aws-custom-region-and-endpoints.md [2] https://bugzilla.redhat.com/show_bug.cgi?id=1866299#c2 [3] https://bugzilla.redhat.com/show_bug.cgi?id=1866299#c4
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days