Bug 1862065 - [aws-custom-region] error "listing hosted zones: SignatureDoesNotMatch" occurred when creating cluster in af-south-1 region
Summary: [aws-custom-region] error "listing hosted zones: SignatureDoesNotMatch" occur...
Keywords:
Status: CLOSED DUPLICATE of bug 1866299
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.6
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.6.0
Assignee: Abhinav Dahiya
QA Contact: Yunfei Jiang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-07-30 10:40 UTC by Yunfei Jiang
Modified: 2023-09-14 06:04 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-08-12 21:15:37 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
install log (IAM) (155.76 KB, text/plain)
2020-08-03 15:57 UTC, Yunfei Jiang
no flags Details
install log without IAM and route53 (190.80 KB, text/plain)
2020-08-05 09:24 UTC, Yunfei Jiang
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 3990 0 None closed Bug 1862065: aws: use default resolver for signing_region of global APIs 2021-01-15 09:53:41 UTC

Description Yunfei Jiang 2020-07-30 10:40:46 UTC
OCP 4.6 supports custom endpoints, cluster could be installed on a public regions without native support for RHCOS
https://github.com/openshift/enhancements/blob/master/enhancements/installer/aws-custom-region-and-endpoints.md


Try to install a cluster into af-south-1, get following error:

time="2020-07-30T08:57:06Z" level=debug msg="resolved AWS service route53 (af-south-1) to \"https://route53.amazonaws.com\""
time="2020-07-30T08:57:06Z" level=fatal msg="failed to fetch Common Manifests: failed to fetch dependency of \"Common Manifests\": failed to generate asset \"DNS Config\": getting public zone for \"qe.devcluster.openshift.com\": listing hosted zones: SignatureDoesNotMatch: Credential should be scoped to a valid region, not 'af-south-1'. \n\tstatus code: 403, request id: 352dc7d4-a5ae-4eb7-aa41-373c3c8f170c"

install config:
---
apiVersion: v1
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  platform: {}
  replicas: 3
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  platform: {}
  replicas: 3
metadata:
  name: yunjiang-af2
platform:
  aws:
    region: af-south-1
    serviceEndpoints:
    - name: ec2
      url: https://ec2.af-south-1.amazonaws.com
    - name: elasticloadbalancing
      url: https://elasticloadbalancing.af-south-1.amazonaws.com
    - name: s3
      url: https://s3.af-south-1.amazonaws.com
    - name: iam
      url: https://iam.af-south-1.amazonaws.com
    - name: tagging
      url: https://tagging.af-south-1.amazonaws.com
    - name: route53
      url: https://route53.amazonaws.com
pullSecret: HIDDEN
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  serviceNetwork:
  - 172.30.0.0/16
  machineNetwork:
  - cidr: 10.0.0.0/16
  networkType: OpenShiftSDN
publish: External
baseDomain: qe.devcluster.openshift.com
sshKey: HIDDEN


Version-Release number of the following components:
Cluster version is 4.6.0-0.nightly-2020-07-25-091217

How reproducible:
100%

Steps to Reproduce:
1. Create install config
2. Set up endpoints:
  aws:
    region: af-south-1
    serviceEndpoints:
    - name: ec2
      url: https://ec2.af-south-1.amazonaws.com
    - name: elasticloadbalancing
      url: https://elasticloadbalancing.af-south-1.amazonaws.com
    - name: s3
      url: https://s3.af-south-1.amazonaws.com
    - name: iam
      url: https://iam.af-south-1.amazonaws.com
    - name: tagging
      url: https://tagging.af-south-1.amazonaws.com
    - name: route53
      url: https://route53.amazonaws.com
3. create cluster

Actual results:
SignatureDoesNotMatch: Credential should be scoped to a valid region, not 'af-south-1'. 

Expected results:
No errors, create cluster successfully

Additional info:
4.6.0-0.nightly-2020-07-25-091217 could be installed successfully in us-east-2

Comment 3 Yunfei Jiang 2020-08-03 15:56:24 UTC
verified on 4.6.0-0.nightly-2020-08-02-091622 - FAILED

the original route53 problem was resolved, but got IAM problem (install log is attached), error log:

time="2020-08-03T15:00:49Z" level=debug msg="module.dns.aws_route53_record.api_internal: Creation complete after 57s [id=Z00683592YKJ1PSSGQYWP_api-int.yunjiang-af03bz.qe.devcluster.openshift.com_A]"
time="2020-08-03T15:00:49Z" level=error
time="2020-08-03T15:00:49Z" level=error msg="Error: Error creating IAM Role yunjiang-af03bz-fr9mc-bootstrap-role: SignatureDoesNotMatch: Credential should be scoped to a valid region, not 'af-south-1'. "
time="2020-08-03T15:00:49Z" level=error msg="\tstatus code: 403, request id: 4213a2c0-6b69-4731-8483-b683c7cfd00c"
time="2020-08-03T15:00:49Z" level=error
time="2020-08-03T15:00:49Z" level=error msg="  on ../../../../../tmp/openshift-install-787325111/bootstrap/main.tf line 51, in resource \"aws_iam_role\" \"bootstrap\":"
time="2020-08-03T15:00:49Z" level=error msg="  51: resource \"aws_iam_role\" \"bootstrap\" {"
time="2020-08-03T15:00:49Z" level=error
time="2020-08-03T15:00:49Z" level=error
time="2020-08-03T15:00:49Z" level=error
time="2020-08-03T15:00:49Z" level=error msg="Error: Error creating IAM Role yunjiang-af03bz-fr9mc-worker-role: SignatureDoesNotMatch: Credential should be scoped to a valid region, not 'af-south-1'. "
time="2020-08-03T15:00:49Z" level=error msg="\tstatus code: 403, request id: b9fbb43c-cbfa-45c1-80a4-ea2a09029197"
time="2020-08-03T15:00:49Z" level=error
time="2020-08-03T15:00:49Z" level=error msg="  on ../../../../../tmp/openshift-install-787325111/iam/main.tf line 13, in resource \"aws_iam_role\" \"worker_role\":"
time="2020-08-03T15:00:49Z" level=error msg="  13: resource \"aws_iam_role\" \"worker_role\" {"
time="2020-08-03T15:00:49Z" level=error
time="2020-08-03T15:00:49Z" level=error
time="2020-08-03T15:00:49Z" level=error
time="2020-08-03T15:00:49Z" level=error msg="Error: Error creating IAM Role yunjiang-af03bz-fr9mc-master-role: SignatureDoesNotMatch: Credential should be scoped to a valid region, not 'af-south-1'. "
time="2020-08-03T15:00:49Z" level=error msg="\tstatus code: 403, request id: 345e615d-9b91-4154-865f-3718a567588c"
time="2020-08-03T15:00:49Z" level=error
time="2020-08-03T15:00:49Z" level=error msg="  on ../../../../../tmp/openshift-install-787325111/master/main.tf line 17, in resource \"aws_iam_role\" \"master_role\":"
time="2020-08-03T15:00:49Z" level=error msg="  17: resource \"aws_iam_role\" \"master_role\" {"
time="2020-08-03T15:00:49Z" level=error
time="2020-08-03T15:00:49Z" level=error
time="2020-08-03T15:00:49Z" level=error
time="2020-08-03T15:00:49Z" level=error msg="Error: Error creating EIP: AddressLimitExceeded: The maximum number of addresses has been reached."
time="2020-08-03T15:00:49Z" level=error msg="\tstatus code: 400, request id: f2f41aa5-d18a-44c9-8a82-b1339d578f08"
time="2020-08-03T15:00:49Z" level=error
time="2020-08-03T15:00:49Z" level=error msg="  on ../../../../../tmp/openshift-install-787325111/vpc/vpc-public.tf line 68, in resource \"aws_eip\" \"nat_eip\":"
time="2020-08-03T15:00:49Z" level=error msg="  68: resource \"aws_eip\" \"nat_eip\" {"
time="2020-08-03T15:00:49Z" level=error
time="2020-08-03T15:00:49Z" level=error
time="2020-08-03T15:00:49Z" level=fatal msg="failed to fetch Cluster: failed to generate asset \"Cluster\": failed to create cluster: failed to apply Terraform: failed to complete the change"

config:
...
platform:
  aws:
    region: af-south-1
    serviceEndpoints:
    - name: ec2
      url: https://ec2.af-south-1.amazonaws.com
    - name: elasticloadbalancing
      url: https://elasticloadbalancing.af-south-1.amazonaws.com
    - name: s3
      url: https://s3.af-south-1.amazonaws.com
    - name: iam
      url: https://iam.amazonaws.com
    - name: tagging
      url: https://tagging.af-south-1.amazonaws.com
    - name: route53
      url: https://route53.amazonaws.com

Comment 4 Yunfei Jiang 2020-08-03 15:57:16 UTC
Created attachment 1703722 [details]
install log (IAM)

Comment 5 Abhinav Dahiya 2020-08-03 18:16:54 UTC
Can you try without the IAM and route53 endpoints because for public regions these are already known and probably do not need the override.

```
platform:
  aws:
    region: af-south-1
    serviceEndpoints:
    - name: ec2
      url: https://ec2.af-south-1.amazonaws.com
    - name: elasticloadbalancing
      url: https://elasticloadbalancing.af-south-1.amazonaws.com
    - name: s3
      url: https://s3.af-south-1.amazonaws.com
    - name: tagging
      url: https://tagging.af-south-1.amazonaws.com
```

Comment 6 Yunfei Jiang 2020-08-05 09:23:40 UTC
it works after removing IAM and route53 endpoints

the bootstrap process completed:

time="2020-08-05T08:11:43Z" level=info msg="API v4.6.0-202008031851.p0-dirty up"
time="2020-08-05T08:11:43Z" level=info msg="Waiting up to 30m0s for bootstrapping to complete..."
time="2020-08-05T08:27:25Z" level=debug msg="Bootstrap status: complete"
time="2020-08-05T08:27:25Z" level=info msg="Destroying the bootstrap resources..."

but the install process failed due to some operators error:

time="2020-08-05T08:59:25Z" level=fatal msg="failed to initialize the cluster: Cluster operator console is reporting a failure: RouteHealthDegraded: failed to GET route (https://console-openshift-console.apps.yunjiang-05bug.qe.devcluster.openshift.com/health): Get \"https://console-openshift-console.apps.yunjiang-05bug.qe.devcluster.openshift.com/health\": dial tcp: lookup console-openshift-console.apps.yunjiang-05bug.qe.devcluster.openshift.com on 172.30.0.10:53: no such host"

probably it is another issue, install log is attached.

Comment 7 Yunfei Jiang 2020-08-05 09:24:20 UTC
Created attachment 1710489 [details]
install log without IAM and route53

Comment 8 Yunfei Jiang 2020-08-05 10:46:54 UTC
seems the ingress operator is affected by the service endpoints, please refer to https://bugzilla.redhat.com/show_bug.cgi?id=1866299

Comment 9 Daneyon Hansen 2020-08-06 17:35:28 UTC
Please see https://bugzilla.redhat.com/show_bug.cgi?id=1866299#c2

Comment 10 Hongan Li 2020-08-11 02:49:27 UTC
see https://bugzilla.redhat.com/show_bug.cgi?id=1866299#c4
It works well after updating the tagging endpoint as below:
      - name: tagging
        url: https://tagging.us-east-1.amazonaws.com

Comment 11 Daneyon Hansen 2020-08-12 21:15:37 UTC

*** This bug has been marked as a duplicate of bug 1866299 ***

Comment 12 Yunfei Jiang 2020-08-18 09:25:43 UTC
the issue as described is still there:

1. according to the document [1], ec2/elb/s3/iam/tagging/route53 could be provided by user, if user provide above endpoints correctly (even it overrides the default), the cluster should be installed successfully. 
2. according to comments [2][3], following config could work
<--snip-->
      serviceEndpoints:
      - name: ec2
        url: https://ec2.af-south-1.amazonaws.com
      - name: elasticloadbalancing
        url: https://elasticloadbalancing.af-south-1.amazonaws.com
      - name: s3
        url: https://s3.af-south-1.amazonaws.com
      - name: tagging
        url: https://tagging.us-east-1.amazonaws.com
<--snip-->

3. tried to install using following config (note for tagging endpoint, it's `us-east-1` as comment [2] mentioned)
<--snip-->
platform:
  aws:
    region: af-south-1
    serviceEndpoints:
    - name: ec2
      url: https://ec2.af-south-1.amazonaws.com
    - name: elasticloadbalancing
      url: https://elasticloadbalancing.af-south-1.amazonaws.com
    - name: s3
      url: https://s3.af-south-1.amazonaws.com
    - name: iam
      url: https://iam.amazonaws.com
    - name: tagging
      url: https://tagging.us-east-1.amazonaws.com
    - name: route53
      url: https://route53.amazonaws.com
<--snip-->

got following error:

level=error msg="Error: Error creating IAM Role yunjiang-18af4-6p56w-bootstrap-role: SignatureDoesNotMatch: Credential should be scoped to a valid region, not 'af-south-1'. " 

Is this the same issue as bug 1866299 described? Will the 'SignatureDoesNotMatch' issue be fixed?

Need Daneyon and Abhinav to confirm, thanks.


[1] https://github.com/openshift/enhancements/blob/master/enhancements/installer/aws-custom-region-and-endpoints.md
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1866299#c2
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1866299#c4

Comment 13 Red Hat Bugzilla 2023-09-14 06:04:33 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days


Note You need to log in before you can comment on or make changes to this bug.