Bug 1867969

Summary:

[aws-custom-region] Private Cluster can not be created in us-gov region due to "NoSuchHostedZone: The specified hosted zone does not exist."

Product:

OpenShift Container Platform

Reporter:

Yunfei Jiang <yunjiang>

Component:

Installer

Assignee:

aos-install

Installer sub component:

openshift-installer

QA Contact:

Yunfei Jiang <yunjiang>

Status:

CLOSED DUPLICATE

Docs Contact:

Severity:

urgent

Priority:

urgent

CC:

adahiya, jhocutt, wzheng

Version:

4.6

Keywords:

TestBlocker

Target Milestone:

---

Target Release:

---

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2020-08-11 17:05:15 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
install log	none

Description Yunfei Jiang 2020-08-11 09:50:03 UTC

Created attachment 1711054 [details]
install log

Description of problem: 

Since public domain is not supported in US-Gov region, so we are tring to install a private cluster in us-gov-west-1 region, but got following error:

ERROR
ERROR Error: [ERR]: Error building changeset: NoSuchHostedZone: The specified hosted zone does not exist.
ERROR     status code: 404, request id: d3f68fd6-8d28-4e1d-b164-3702a611291d
ERROR
ERROR   on ../../../tmp/openshift-install-937580347/route53/base.tf line 47, in resource "aws_route53_record" "api_internal":
ERROR   47: resource "aws_route53_record" "api_internal" {
ERROR
ERROR
ERROR
ERROR Error: [ERR]: Error building changeset: NoSuchHostedZone: The specified hosted zone does not exist.
ERROR     status code: 404, request id: c25a90c8-0908-44d2-b8cc-1ee0b46bf9a4
ERROR
ERROR   on ../../../tmp/openshift-install-937580347/route53/base.tf line 59, in resource "aws_route53_record" "api_external_internal_zone":
ERROR   59: resource "aws_route53_record" "api_external_internal_zone" {
ERROR
ERROR
FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change

After the above failure, even if manully add api/api-int DNS A record in cluster private route53 zone on aws webconosle, set alais to api elb DNS, hostzone ID is automatically accociated, when clicking create button, failed with the same error, saying:
The specified hosted zone does not exist.

Version-Release number of the following components: 
4.6.0-0.nightly-2020-08-09-151434
 
How reproducible: 
Always 
 
Steps to Reproduce: 
1. Create private cluster

install config
---
apiVersion: v1
baseDomain: yunjiang-gov-test.com
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  platform:
    aws:
      amiID: ami-0dc4fa6c
  replicas: 3
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  platform:
    aws:
      amiID: ami-0dc4fa6c
  replicas: 3
metadata:
  creationTimestamp: null
  name: yunjiang-4ep
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineNetwork:
  - cidr: 10.0.0.0/16
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16
platform:
  aws:
    region: us-gov-west-1
    subnets:
    - subnet-0668e9b924e0350ff
    - subnet-0e8f8622f5fd1e606
    - subnet-0d9d36c7bd56d7b03
    serviceEndpoints:
    - name: ec2
      url: https://ec2.us-gov-west-1.amazonaws.com
    - name: elasticloadbalancing
      url: https://elasticloadbalancing.us-gov-west-1.amazonaws.com
    - name: s3
      url: https://s3.us-gov-west-1.amazonaws.com
    - name: tagging
      url: https://tagging.us-gov-west-1.amazonaws.com
publish: Internal
...
 
Actual results: 
ERROR Error: [ERR]: Error building changeset: NoSuchHostedZone: The specified hosted zone does not exist.
ERROR     status code: 404, request id: d3f68fd6-8d28-4e1d-b164-3702a611291d
...
FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change

Expected results: 
Create cluster successfully
 
Additional info:

Comment 1 Yunfei Jiang 2020-08-11 10:27:07 UTC

All tests against us-gov cloud is blocked, both public and private cluster can not be installed.
Even manually add DNS A record via web console, still get errors.

Comment 2 Yunfei Jiang 2020-08-11 10:27:33 UTC

All tests against us-gov cloud is blocked, both public and private cluster can not be installed.
Even manually add DNS A record via web console, still get errors.

Comment 3 Jared Hocutt 2020-08-11 13:11:16 UTC

I saw this BZ pop up and wanted to see if I could reproduce since I have previously had this working in 4.5 (even though it's obviously not GA). I was thinking there might be a mistake with defining the serviceEndpoints since I didn't specifically see the Route53 endpoint listed.

I was able to reproduce and it does seem to be a bug.

I trimmed my install-config.yaml down to let the serviceEndpoints use the default values since Terraform is aware of GovCloud endpoints on it's own and still received the same error as Yunfei Jiang.

```
apiVersion: v1
baseDomain: govcloud.rdht.io
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  replicas: 3
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  replicas: 3
metadata:
  creationTimestamp: null
  name: test2
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineNetwork:
  - cidr: 10.0.0.0/16
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16
platform:
  aws:
    region: us-gov-east-1
    amiID: ami-f7bd5486
publish: Internal
pullSecret: ''
sshKey: ''
```

I can also see in the debug output that Terraform reports that the private Route53 zone was created successfully before it attempted to create the Route53 entries.

DEBUG module.dns.aws_route53_zone.int: Still creating... [30s elapsed] 
DEBUG module.masters.aws_instance.master[1]: Creation complete after 17s [id=i-03c284c90232c124f] 
DEBUG module.masters.aws_instance.master[0]: Creation complete after 17s [id=i-03148575b5a0520c6] 
DEBUG module.vpc.aws_nat_gateway.nat_gw[0]: Still creating... [20s elapsed] 
DEBUG module.vpc.aws_nat_gateway.nat_gw[2]: Still creating... [20s elapsed] 
DEBUG module.vpc.aws_nat_gateway.nat_gw[1]: Still creating... [20s elapsed] 
DEBUG module.dns.aws_route53_zone.int: Creation complete after 35s [id=Z08675922OGBN6NBGKK23] 
DEBUG module.vpc.aws_lb.api_internal: Still creating... [20s elapsed] 
DEBUG module.masters.aws_instance.master[2]: Still creating... [20s elapsed] 
DEBUG module.masters.aws_instance.master[2]: Creation complete after 23s [id=i-0134be58567eb3376] 
DEBUG module.masters.aws_lb_target_group_attachment.master[5]: Creating... 
DEBUG module.masters.aws_lb_target_group_attachment.master[4]: Creating... 
DEBUG module.masters.aws_lb_target_group_attachment.master[0]: Creating... 
DEBUG module.masters.aws_lb_target_group_attachment.master[2]: Creating... 
DEBUG module.masters.aws_lb_target_group_attachment.master[3]: Creating... 
DEBUG module.masters.aws_lb_target_group_attachment.master[1]: Creating... 
DEBUG module.masters.aws_lb_target_group_attachment.master[2]: Creation complete after 0s [id=arn:aws-us-gov:elasticloadbalancing:us-gov-east-1:676111148133:targetgroup/test2-jw5gb-aint/e7068458f5faba54-20200811124646396500000007] 
DEBUG module.masters.aws_lb_target_group_attachment.master[3]: Creation complete after 0s [id=arn:aws-us-gov:elasticloadbalancing:us-gov-east-1:676111148133:targetgroup/test2-jw5gb-sint/d60fe080d98bef48-20200811124646402500000008] 
DEBUG module.masters.aws_lb_target_group_attachment.master[5]: Creation complete after 0s [id=arn:aws-us-gov:elasticloadbalancing:us-gov-east-1:676111148133:targetgroup/test2-jw5gb-sint/d60fe080d98bef48-20200811124646415000000009] 
DEBUG module.masters.aws_lb_target_group_attachment.master[4]: Creation complete after 0s [id=arn:aws-us-gov:elasticloadbalancing:us-gov-east-1:676111148133:targetgroup/test2-jw5gb-aint/e7068458f5faba54-2020081112464643400000000a] 
DEBUG module.masters.aws_lb_target_group_attachment.master[0]: Creation complete after 0s [id=arn:aws-us-gov:elasticloadbalancing:us-gov-east-1:676111148133:targetgroup/test2-jw5gb-aint/e7068458f5faba54-2020081112464646130000000b] 
DEBUG module.masters.aws_lb_target_group_attachment.master[1]: Creation complete after 0s [id=arn:aws-us-gov:elasticloadbalancing:us-gov-east-1:676111148133:targetgroup/test2-jw5gb-sint/d60fe080d98bef48-2020081112464648000000000c] 
DEBUG module.vpc.aws_nat_gateway.nat_gw[0]: Still creating... [30s elapsed] 
DEBUG module.vpc.aws_nat_gateway.nat_gw[2]: Still creating... [30s elapsed] 
DEBUG module.vpc.aws_nat_gateway.nat_gw[1]: Still creating... [30s elapsed] 
DEBUG module.vpc.aws_lb.api_internal: Still creating... [30s elapsed] 
DEBUG module.vpc.aws_nat_gateway.nat_gw[0]: Still creating... [40s elapsed] 
DEBUG module.vpc.aws_nat_gateway.nat_gw[1]: Still creating... [40s elapsed] 
DEBUG module.vpc.aws_nat_gateway.nat_gw[2]: Still creating... [40s elapsed] 
DEBUG module.vpc.aws_lb.api_internal: Still creating... [40s elapsed] 
DEBUG module.vpc.aws_nat_gateway.nat_gw[0]: Still creating... [50s elapsed] 
DEBUG module.vpc.aws_nat_gateway.nat_gw[1]: Still creating... [50s elapsed] 
DEBUG module.vpc.aws_nat_gateway.nat_gw[2]: Still creating... [50s elapsed] 
DEBUG module.vpc.aws_lb.api_internal: Still creating... [50s elapsed] 
DEBUG module.vpc.aws_nat_gateway.nat_gw[0]: Still creating... [1m0s elapsed] 
DEBUG module.vpc.aws_nat_gateway.nat_gw[2]: Still creating... [1m0s elapsed] 
DEBUG module.vpc.aws_nat_gateway.nat_gw[1]: Still creating... [1m0s elapsed] 
DEBUG module.vpc.aws_lb.api_internal: Still creating... [1m0s elapsed] 
DEBUG module.vpc.aws_nat_gateway.nat_gw[0]: Still creating... [1m10s elapsed] 
DEBUG module.vpc.aws_nat_gateway.nat_gw[1]: Still creating... [1m10s elapsed] 
DEBUG module.vpc.aws_nat_gateway.nat_gw[2]: Still creating... [1m10s elapsed] 
DEBUG module.vpc.aws_lb.api_internal: Still creating... [1m10s elapsed] 
DEBUG module.vpc.aws_nat_gateway.nat_gw[0]: Still creating... [1m20s elapsed] 
DEBUG module.vpc.aws_nat_gateway.nat_gw[2]: Still creating... [1m20s elapsed] 
DEBUG module.vpc.aws_nat_gateway.nat_gw[1]: Still creating... [1m20s elapsed] 
DEBUG module.vpc.aws_lb.api_internal: Still creating... [1m20s elapsed] 
DEBUG module.vpc.aws_nat_gateway.nat_gw[1]: Creation complete after 1m25s [id=nat-03dd0d4c7cebd5420] 
DEBUG module.vpc.aws_nat_gateway.nat_gw[0]: Still creating... [1m30s elapsed] 
DEBUG module.vpc.aws_nat_gateway.nat_gw[2]: Still creating... [1m30s elapsed] 
DEBUG module.vpc.aws_lb.api_internal: Still creating... [1m30s elapsed] 
DEBUG module.vpc.aws_nat_gateway.nat_gw[0]: Still creating... [1m40s elapsed] 
DEBUG module.vpc.aws_nat_gateway.nat_gw[2]: Still creating... [1m40s elapsed] 
DEBUG module.vpc.aws_lb.api_internal: Still creating... [1m40s elapsed] 
DEBUG module.vpc.aws_nat_gateway.nat_gw[0]: Creation complete after 1m47s [id=nat-03dde738be1758b74] 
DEBUG module.vpc.aws_nat_gateway.nat_gw[2]: Creation complete after 1m46s [id=nat-00d5567dc1de1b9de] 
DEBUG module.vpc.aws_route.to_nat_gw[1]: Creating... 
DEBUG module.vpc.aws_route.to_nat_gw[2]: Creating... 
DEBUG module.vpc.aws_route.to_nat_gw[0]: Creating... 
DEBUG module.vpc.aws_route.to_nat_gw[1]: Creation complete after 0s [id=r-rtb-070732f8ab84001bd1080289494] 
DEBUG module.vpc.aws_route.to_nat_gw[0]: Creation complete after 0s [id=r-rtb-07bdedc890709a3b31080289494] 
DEBUG module.vpc.aws_route.to_nat_gw[2]: Creation complete after 0s [id=r-rtb-0e43b4c528b4675f81080289494] 
DEBUG module.vpc.aws_lb.api_internal: Still creating... [1m50s elapsed] 
DEBUG module.vpc.aws_lb.api_internal: Still creating... [2m0s elapsed] 
DEBUG module.vpc.aws_lb.api_internal: Still creating... [2m10s elapsed] 
DEBUG module.vpc.aws_lb.api_internal: Still creating... [2m20s elapsed] 
DEBUG module.vpc.aws_lb.api_internal: Still creating... [2m30s elapsed] 
DEBUG module.vpc.aws_lb.api_internal: Still creating... [2m40s elapsed] 
DEBUG module.vpc.aws_lb.api_internal: Still creating... [2m50s elapsed] 
DEBUG module.vpc.aws_lb.api_internal: Still creating... [3m0s elapsed] 
DEBUG module.vpc.aws_lb.api_internal: Creation complete after 3m3s [id=arn:aws-us-gov:elasticloadbalancing:us-gov-east-1:676111148133:loadbalancer/net/test2-jw5gb-int/eceb7c4d08960127] 
DEBUG module.dns.aws_route53_record.api_external_internal_zone: Creating... 
DEBUG module.dns.aws_route53_record.api_internal: Creating... 
DEBUG module.vpc.aws_lb_listener.api_internal_api: Creating... 
DEBUG module.vpc.aws_lb_listener.api_internal_services: Creating... 
DEBUG module.vpc.aws_lb_listener.api_internal_api: Creation complete after 1s [id=arn:aws-us-gov:elasticloadbalancing:us-gov-east-1:676111148133:listener/net/test2-jw5gb-int/eceb7c4d08960127/43214fef7c43568c] 
DEBUG module.vpc.aws_lb_listener.api_internal_services: Creation complete after 1s [id=arn:aws-us-gov:elasticloadbalancing:us-gov-east-1:676111148133:listener/net/test2-jw5gb-int/eceb7c4d08960127/3131adf7286f17a9] 
DEBUG module.dns.aws_route53_record.api_internal: Still creating... [10s elapsed] 
DEBUG module.dns.aws_route53_record.api_external_internal_zone: Still creating... [10s elapsed] 
DEBUG module.dns.aws_route53_record.api_internal: Still creating... [20s elapsed] 
DEBUG module.dns.aws_route53_record.api_external_internal_zone: Still creating... [20s elapsed] 
DEBUG module.dns.aws_route53_record.api_internal: Still creating... [30s elapsed] 
DEBUG module.dns.aws_route53_record.api_external_internal_zone: Still creating... [30s elapsed] 
DEBUG module.dns.aws_route53_record.api_external_internal_zone: Still creating... [40s elapsed] 
DEBUG module.dns.aws_route53_record.api_internal: Still creating... [40s elapsed] 
DEBUG module.dns.aws_route53_record.api_internal: Still creating... [50s elapsed] 
DEBUG module.dns.aws_route53_record.api_external_internal_zone: Still creating... [50s elapsed] 
DEBUG module.dns.aws_route53_record.api_internal: Still creating... [1m0s elapsed] 
DEBUG module.dns.aws_route53_record.api_external_internal_zone: Still creating... [1m0s elapsed] 
ERROR                                              
ERROR Error: [ERR]: Error building changeset: NoSuchHostedZone: The specified hosted zone does not exist. 
ERROR 	status code: 404, request id: 0dd3f76e-fe03-42a3-ba70-d094689694ac 
ERROR                                              
ERROR   on ../../../../../tmp/openshift-install-706302076/route53/base.tf line 47, in resource "aws_route53_record" "api_internal": 
ERROR   47: resource "aws_route53_record" "api_internal" { 
ERROR                                              
ERROR                                              
ERROR                                              
ERROR Error: [ERR]: Error building changeset: NoSuchHostedZone: The specified hosted zone does not exist. 
ERROR 	status code: 404, request id: e9f98fc3-5493-479b-aae3-03717407bc82 
ERROR                                              
ERROR   on ../../../../../tmp/openshift-install-706302076/route53/base.tf line 59, in resource "aws_route53_record" "api_external_internal_zone": 
ERROR   59: resource "aws_route53_record" "api_external_internal_zone" { 
ERROR                                              
ERROR                                              
FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change

Comment 4 Abhinav Dahiya 2020-08-11 17:05:15 UTC


*** This bug has been marked as a duplicate of bug 1866949 ***