Created attachment 1711054 [details] install log Description of problem: Since public domain is not supported in US-Gov region, so we are tring to install a private cluster in us-gov-west-1 region, but got following error: ERROR ERROR Error: [ERR]: Error building changeset: NoSuchHostedZone: The specified hosted zone does not exist. ERROR status code: 404, request id: d3f68fd6-8d28-4e1d-b164-3702a611291d ERROR ERROR on ../../../tmp/openshift-install-937580347/route53/base.tf line 47, in resource "aws_route53_record" "api_internal": ERROR 47: resource "aws_route53_record" "api_internal" { ERROR ERROR ERROR ERROR Error: [ERR]: Error building changeset: NoSuchHostedZone: The specified hosted zone does not exist. ERROR status code: 404, request id: c25a90c8-0908-44d2-b8cc-1ee0b46bf9a4 ERROR ERROR on ../../../tmp/openshift-install-937580347/route53/base.tf line 59, in resource "aws_route53_record" "api_external_internal_zone": ERROR 59: resource "aws_route53_record" "api_external_internal_zone" { ERROR ERROR FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change After the above failure, even if manully add api/api-int DNS A record in cluster private route53 zone on aws webconosle, set alais to api elb DNS, hostzone ID is automatically accociated, when clicking create button, failed with the same error, saying: The specified hosted zone does not exist. Version-Release number of the following components: 4.6.0-0.nightly-2020-08-09-151434 How reproducible: Always Steps to Reproduce: 1. Create private cluster install config --- apiVersion: v1 baseDomain: yunjiang-gov-test.com compute: - architecture: amd64 hyperthreading: Enabled name: worker platform: aws: amiID: ami-0dc4fa6c replicas: 3 controlPlane: architecture: amd64 hyperthreading: Enabled name: master platform: aws: amiID: ami-0dc4fa6c replicas: 3 metadata: creationTimestamp: null name: yunjiang-4ep networking: clusterNetwork: - cidr: 10.128.0.0/14 hostPrefix: 23 machineNetwork: - cidr: 10.0.0.0/16 networkType: OpenShiftSDN serviceNetwork: - 172.30.0.0/16 platform: aws: region: us-gov-west-1 subnets: - subnet-0668e9b924e0350ff - subnet-0e8f8622f5fd1e606 - subnet-0d9d36c7bd56d7b03 serviceEndpoints: - name: ec2 url: https://ec2.us-gov-west-1.amazonaws.com - name: elasticloadbalancing url: https://elasticloadbalancing.us-gov-west-1.amazonaws.com - name: s3 url: https://s3.us-gov-west-1.amazonaws.com - name: tagging url: https://tagging.us-gov-west-1.amazonaws.com publish: Internal ... Actual results: ERROR Error: [ERR]: Error building changeset: NoSuchHostedZone: The specified hosted zone does not exist. ERROR status code: 404, request id: d3f68fd6-8d28-4e1d-b164-3702a611291d ... FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change Expected results: Create cluster successfully Additional info:
All tests against us-gov cloud is blocked, both public and private cluster can not be installed. Even manually add DNS A record via web console, still get errors.
I saw this BZ pop up and wanted to see if I could reproduce since I have previously had this working in 4.5 (even though it's obviously not GA). I was thinking there might be a mistake with defining the serviceEndpoints since I didn't specifically see the Route53 endpoint listed. I was able to reproduce and it does seem to be a bug. I trimmed my install-config.yaml down to let the serviceEndpoints use the default values since Terraform is aware of GovCloud endpoints on it's own and still received the same error as Yunfei Jiang. ``` apiVersion: v1 baseDomain: govcloud.rdht.io compute: - architecture: amd64 hyperthreading: Enabled name: worker replicas: 3 controlPlane: architecture: amd64 hyperthreading: Enabled name: master replicas: 3 metadata: creationTimestamp: null name: test2 networking: clusterNetwork: - cidr: 10.128.0.0/14 hostPrefix: 23 machineNetwork: - cidr: 10.0.0.0/16 networkType: OpenShiftSDN serviceNetwork: - 172.30.0.0/16 platform: aws: region: us-gov-east-1 amiID: ami-f7bd5486 publish: Internal pullSecret: '' sshKey: '' ``` I can also see in the debug output that Terraform reports that the private Route53 zone was created successfully before it attempted to create the Route53 entries. DEBUG module.dns.aws_route53_zone.int: Still creating... [30s elapsed] DEBUG module.masters.aws_instance.master[1]: Creation complete after 17s [id=i-03c284c90232c124f] DEBUG module.masters.aws_instance.master[0]: Creation complete after 17s [id=i-03148575b5a0520c6] DEBUG module.vpc.aws_nat_gateway.nat_gw[0]: Still creating... [20s elapsed] DEBUG module.vpc.aws_nat_gateway.nat_gw[2]: Still creating... [20s elapsed] DEBUG module.vpc.aws_nat_gateway.nat_gw[1]: Still creating... [20s elapsed] DEBUG module.dns.aws_route53_zone.int: Creation complete after 35s [id=Z08675922OGBN6NBGKK23] DEBUG module.vpc.aws_lb.api_internal: Still creating... [20s elapsed] DEBUG module.masters.aws_instance.master[2]: Still creating... [20s elapsed] DEBUG module.masters.aws_instance.master[2]: Creation complete after 23s [id=i-0134be58567eb3376] DEBUG module.masters.aws_lb_target_group_attachment.master[5]: Creating... DEBUG module.masters.aws_lb_target_group_attachment.master[4]: Creating... DEBUG module.masters.aws_lb_target_group_attachment.master[0]: Creating... DEBUG module.masters.aws_lb_target_group_attachment.master[2]: Creating... DEBUG module.masters.aws_lb_target_group_attachment.master[3]: Creating... DEBUG module.masters.aws_lb_target_group_attachment.master[1]: Creating... DEBUG module.masters.aws_lb_target_group_attachment.master[2]: Creation complete after 0s [id=arn:aws-us-gov:elasticloadbalancing:us-gov-east-1:676111148133:targetgroup/test2-jw5gb-aint/e7068458f5faba54-20200811124646396500000007] DEBUG module.masters.aws_lb_target_group_attachment.master[3]: Creation complete after 0s [id=arn:aws-us-gov:elasticloadbalancing:us-gov-east-1:676111148133:targetgroup/test2-jw5gb-sint/d60fe080d98bef48-20200811124646402500000008] DEBUG module.masters.aws_lb_target_group_attachment.master[5]: Creation complete after 0s [id=arn:aws-us-gov:elasticloadbalancing:us-gov-east-1:676111148133:targetgroup/test2-jw5gb-sint/d60fe080d98bef48-20200811124646415000000009] DEBUG module.masters.aws_lb_target_group_attachment.master[4]: Creation complete after 0s [id=arn:aws-us-gov:elasticloadbalancing:us-gov-east-1:676111148133:targetgroup/test2-jw5gb-aint/e7068458f5faba54-2020081112464643400000000a] DEBUG module.masters.aws_lb_target_group_attachment.master[0]: Creation complete after 0s [id=arn:aws-us-gov:elasticloadbalancing:us-gov-east-1:676111148133:targetgroup/test2-jw5gb-aint/e7068458f5faba54-2020081112464646130000000b] DEBUG module.masters.aws_lb_target_group_attachment.master[1]: Creation complete after 0s [id=arn:aws-us-gov:elasticloadbalancing:us-gov-east-1:676111148133:targetgroup/test2-jw5gb-sint/d60fe080d98bef48-2020081112464648000000000c] DEBUG module.vpc.aws_nat_gateway.nat_gw[0]: Still creating... [30s elapsed] DEBUG module.vpc.aws_nat_gateway.nat_gw[2]: Still creating... [30s elapsed] DEBUG module.vpc.aws_nat_gateway.nat_gw[1]: Still creating... [30s elapsed] DEBUG module.vpc.aws_lb.api_internal: Still creating... [30s elapsed] DEBUG module.vpc.aws_nat_gateway.nat_gw[0]: Still creating... [40s elapsed] DEBUG module.vpc.aws_nat_gateway.nat_gw[1]: Still creating... [40s elapsed] DEBUG module.vpc.aws_nat_gateway.nat_gw[2]: Still creating... [40s elapsed] DEBUG module.vpc.aws_lb.api_internal: Still creating... [40s elapsed] DEBUG module.vpc.aws_nat_gateway.nat_gw[0]: Still creating... [50s elapsed] DEBUG module.vpc.aws_nat_gateway.nat_gw[1]: Still creating... [50s elapsed] DEBUG module.vpc.aws_nat_gateway.nat_gw[2]: Still creating... [50s elapsed] DEBUG module.vpc.aws_lb.api_internal: Still creating... [50s elapsed] DEBUG module.vpc.aws_nat_gateway.nat_gw[0]: Still creating... [1m0s elapsed] DEBUG module.vpc.aws_nat_gateway.nat_gw[2]: Still creating... [1m0s elapsed] DEBUG module.vpc.aws_nat_gateway.nat_gw[1]: Still creating... [1m0s elapsed] DEBUG module.vpc.aws_lb.api_internal: Still creating... [1m0s elapsed] DEBUG module.vpc.aws_nat_gateway.nat_gw[0]: Still creating... [1m10s elapsed] DEBUG module.vpc.aws_nat_gateway.nat_gw[1]: Still creating... [1m10s elapsed] DEBUG module.vpc.aws_nat_gateway.nat_gw[2]: Still creating... [1m10s elapsed] DEBUG module.vpc.aws_lb.api_internal: Still creating... [1m10s elapsed] DEBUG module.vpc.aws_nat_gateway.nat_gw[0]: Still creating... [1m20s elapsed] DEBUG module.vpc.aws_nat_gateway.nat_gw[2]: Still creating... [1m20s elapsed] DEBUG module.vpc.aws_nat_gateway.nat_gw[1]: Still creating... [1m20s elapsed] DEBUG module.vpc.aws_lb.api_internal: Still creating... [1m20s elapsed] DEBUG module.vpc.aws_nat_gateway.nat_gw[1]: Creation complete after 1m25s [id=nat-03dd0d4c7cebd5420] DEBUG module.vpc.aws_nat_gateway.nat_gw[0]: Still creating... [1m30s elapsed] DEBUG module.vpc.aws_nat_gateway.nat_gw[2]: Still creating... [1m30s elapsed] DEBUG module.vpc.aws_lb.api_internal: Still creating... [1m30s elapsed] DEBUG module.vpc.aws_nat_gateway.nat_gw[0]: Still creating... [1m40s elapsed] DEBUG module.vpc.aws_nat_gateway.nat_gw[2]: Still creating... [1m40s elapsed] DEBUG module.vpc.aws_lb.api_internal: Still creating... [1m40s elapsed] DEBUG module.vpc.aws_nat_gateway.nat_gw[0]: Creation complete after 1m47s [id=nat-03dde738be1758b74] DEBUG module.vpc.aws_nat_gateway.nat_gw[2]: Creation complete after 1m46s [id=nat-00d5567dc1de1b9de] DEBUG module.vpc.aws_route.to_nat_gw[1]: Creating... DEBUG module.vpc.aws_route.to_nat_gw[2]: Creating... DEBUG module.vpc.aws_route.to_nat_gw[0]: Creating... DEBUG module.vpc.aws_route.to_nat_gw[1]: Creation complete after 0s [id=r-rtb-070732f8ab84001bd1080289494] DEBUG module.vpc.aws_route.to_nat_gw[0]: Creation complete after 0s [id=r-rtb-07bdedc890709a3b31080289494] DEBUG module.vpc.aws_route.to_nat_gw[2]: Creation complete after 0s [id=r-rtb-0e43b4c528b4675f81080289494] DEBUG module.vpc.aws_lb.api_internal: Still creating... [1m50s elapsed] DEBUG module.vpc.aws_lb.api_internal: Still creating... [2m0s elapsed] DEBUG module.vpc.aws_lb.api_internal: Still creating... [2m10s elapsed] DEBUG module.vpc.aws_lb.api_internal: Still creating... [2m20s elapsed] DEBUG module.vpc.aws_lb.api_internal: Still creating... [2m30s elapsed] DEBUG module.vpc.aws_lb.api_internal: Still creating... [2m40s elapsed] DEBUG module.vpc.aws_lb.api_internal: Still creating... [2m50s elapsed] DEBUG module.vpc.aws_lb.api_internal: Still creating... [3m0s elapsed] DEBUG module.vpc.aws_lb.api_internal: Creation complete after 3m3s [id=arn:aws-us-gov:elasticloadbalancing:us-gov-east-1:676111148133:loadbalancer/net/test2-jw5gb-int/eceb7c4d08960127] DEBUG module.dns.aws_route53_record.api_external_internal_zone: Creating... DEBUG module.dns.aws_route53_record.api_internal: Creating... DEBUG module.vpc.aws_lb_listener.api_internal_api: Creating... DEBUG module.vpc.aws_lb_listener.api_internal_services: Creating... DEBUG module.vpc.aws_lb_listener.api_internal_api: Creation complete after 1s [id=arn:aws-us-gov:elasticloadbalancing:us-gov-east-1:676111148133:listener/net/test2-jw5gb-int/eceb7c4d08960127/43214fef7c43568c] DEBUG module.vpc.aws_lb_listener.api_internal_services: Creation complete after 1s [id=arn:aws-us-gov:elasticloadbalancing:us-gov-east-1:676111148133:listener/net/test2-jw5gb-int/eceb7c4d08960127/3131adf7286f17a9] DEBUG module.dns.aws_route53_record.api_internal: Still creating... [10s elapsed] DEBUG module.dns.aws_route53_record.api_external_internal_zone: Still creating... [10s elapsed] DEBUG module.dns.aws_route53_record.api_internal: Still creating... [20s elapsed] DEBUG module.dns.aws_route53_record.api_external_internal_zone: Still creating... [20s elapsed] DEBUG module.dns.aws_route53_record.api_internal: Still creating... [30s elapsed] DEBUG module.dns.aws_route53_record.api_external_internal_zone: Still creating... [30s elapsed] DEBUG module.dns.aws_route53_record.api_external_internal_zone: Still creating... [40s elapsed] DEBUG module.dns.aws_route53_record.api_internal: Still creating... [40s elapsed] DEBUG module.dns.aws_route53_record.api_internal: Still creating... [50s elapsed] DEBUG module.dns.aws_route53_record.api_external_internal_zone: Still creating... [50s elapsed] DEBUG module.dns.aws_route53_record.api_internal: Still creating... [1m0s elapsed] DEBUG module.dns.aws_route53_record.api_external_internal_zone: Still creating... [1m0s elapsed] ERROR ERROR Error: [ERR]: Error building changeset: NoSuchHostedZone: The specified hosted zone does not exist. ERROR status code: 404, request id: 0dd3f76e-fe03-42a3-ba70-d094689694ac ERROR ERROR on ../../../../../tmp/openshift-install-706302076/route53/base.tf line 47, in resource "aws_route53_record" "api_internal": ERROR 47: resource "aws_route53_record" "api_internal" { ERROR ERROR ERROR ERROR Error: [ERR]: Error building changeset: NoSuchHostedZone: The specified hosted zone does not exist. ERROR status code: 404, request id: e9f98fc3-5493-479b-aae3-03717407bc82 ERROR ERROR on ../../../../../tmp/openshift-install-706302076/route53/base.tf line 59, in resource "aws_route53_record" "api_external_internal_zone": ERROR 59: resource "aws_route53_record" "api_external_internal_zone" { ERROR ERROR FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change
*** This bug has been marked as a duplicate of bug 1866949 ***