1957597 – Public DNS records were not deleted when destroying a cluster which is using byo private hosted zone

Bug 1957597 - Public DNS records were not deleted when destroying a cluster which is using byo private hosted zone

Summary: Public DNS records were not deleted when destroying a cluster which is using ...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Installer
Sub Component:
Version:	4.8
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	4.8.0
Assignee:	Matthew Staebler
QA Contact:	Yunfei Jiang
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1962435
TreeView+	depends on / blocked

Reported:	2021-05-06 06:26 UTC by Gaoyun Pei
Modified:	2021-07-27 23:07 UTC (History)
CC List:	4 users (show)
Fixed In Version:
Doc Type:	No Doc Update
Doc Text:	Bug in new feature
Clone Of:
Clones:	1962435 (view as bug list)
Environment:
Last Closed:	2021-07-27 23:06:39 UTC
Target Upstream Version:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	openshift installer pull 4933	0	None	open	Bug 1957597: aws: using dotted domain when looking for public hosted zone	2021-05-14 02:06:03 UTC
Red Hat Product Errata	RHSA-2021:2438	0	None	None	None	2021-07-27 23:07:12 UTC

Description Gaoyun Pei 2021-05-06 06:26:59 UTC

Version:
# ./openshift-install version
./openshift-install 4.7.0-0.nightly-2021-05-05-092347
built from commit 3d157f47000c2a9963527ad1dc8c69b77053a4a6
release image registry.ci.openshift.org/ocp/release@sha256:6372cb595b0bf1c7a7a8cbca6de741f2b896d76f536e556719d2a8261f41b4a6

Platform:
IPI-on-AWS

What happened?
When destroying an IPI-on-AWS cluster which is using byo private hosted zone, the A records in the public hosted zone were not deleted after the teardown finished. 
DNS records in the private hosted zone were deleted as expected.


What did you expect to happen?
The DNS records of the cluster in the public hosted zone should also be removed when destroying the cluster.


How to reproduce it (as minimally and precisely as possible)?

1. Provide the private hosted zone Z0197489DELWZQPZ0ZOL in the install-config.yaml, create the cluster
 
  [02:35:25] INFO> install-config.yaml:
  ---
  apiVersion: v1
  controlPlane:
    architecture: amd64
    hyperthreading: Enabled
    name: master
    platform: {}
    replicas: 3
  compute:
  - architecture: amd64
    hyperthreading: Enabled
    name: worker
    platform: {}
    replicas: 3
  metadata:
    name: gpei-bz1954803
  platform:
    aws:
      region: us-east-2
      subnets:
      - subnet-0706df2ecca1bdc62
      - subnet-06271fc2256e8bd90
      - subnet-03be7dfd33e6f5d24
      - subnet-0ea5e3f656ac32af9
      hostedZone: Z0197489DELWZQPZ0ZOL
  pullSecret: HIDDEN
  networking:
    clusterNetwork:
    - cidr: 10.128.0.0/14
      hostPrefix: 23
    serviceNetwork:
    - 172.30.0.0/16
    machineNetwork:
    - cidr: 10.0.0.0/16
    networkType: OpenShiftSDN
  publish: External
  baseDomain: qe.devcluster.openshift.com
  ...


2. Destroy the cluster and check the DNS records associated with the cluster

In the public hosted zone Z3B3KOVA3TRCWP(qe.devcluster.openshift.com.), the api and apps wildcard A records are still there.
# aws route53 list-resource-record-sets --hosted-zone-id Z3B3KOVA3TRCWP --query 'ResourceRecordSets[?contains(Name, `gpei-bz1954803`)]' | jq .[].Name
"api.gpei-bz1954803.qe.devcluster.openshift.com."
"\\052.apps.gpei-bz1954803.qe.devcluster.openshift.com."


The full destroy log:
# ./openshift-install destroy cluster --dir archive/workdir/install-dir/
INFO Credentials loaded from the "default" profile in file "/root/.aws/credentials" 
INFO Terminated                                    instance=i-001a9b5425b35390b
INFO Disassociated                                 id=i-07a6cc15e01bb2b48 name=gpei-bz1954803-h9klh-master-profile role=gpei-bz1954803-h9klh-master-role
INFO Deleted                                       InstanceProfileName=gpei-bz1954803-h9klh-master-profile arn=arn:aws:iam::301721915996:instance-profile/gpei-bz1954803-h9klh-master-profile id=i-07a6cc15e01bb2b48
INFO Disassociated                                 id=i-03ecdd8aea2a9e066 name=gpei-bz1954803-h9klh-worker-profile role=gpei-bz1954803-h9klh-worker-role
INFO Deleted                                       InstanceProfileName=gpei-bz1954803-h9klh-worker-profile arn=arn:aws:iam::301721915996:instance-profile/gpei-bz1954803-h9klh-worker-profile id=i-03ecdd8aea2a9e066
INFO Deleted                                       id=eni-0f0a11ff90405f673
INFO Deleted                                       id=gpei-bz1954803-h9klh-cloud-credential-operator-iam-ro-cbdmg policy=gpei-bz1954803-h9klh-cloud-credential-operator-iam-ro-cbdmg-policy
INFO Deleted                                       id=gpei-bz1954803-h9klh-cloud-credential-operator-iam-ro-cbdmg
INFO Deleted                                       id=gpei-bz1954803-h9klh-openshift-image-registry-dk59l policy=gpei-bz1954803-h9klh-openshift-image-registry-dk59l-policy
INFO Deleted                                       id=gpei-bz1954803-h9klh-openshift-image-registry-dk59l
INFO Deleted                                       id=gpei-bz1954803-h9klh-openshift-ingress-ndtft policy=gpei-bz1954803-h9klh-openshift-ingress-ndtft-policy
INFO Deleted                                       id=gpei-bz1954803-h9klh-openshift-ingress-ndtft
INFO Deleted                                       id=net/gpei-bz1954803-h9klh-int/cf57f66e4f0130b5
INFO Deleted                                       id=gpei-bz1954803-h9klh-master-role name=gpei-bz1954803-h9klh-master-role policy=gpei-bz1954803-h9klh-master-policy
INFO Deleted                                       id=gpei-bz1954803-h9klh-master-role name=gpei-bz1954803-h9klh-master-role
INFO Deleted                                       id=gpei-bz1954803-h9klh-worker-role name=gpei-bz1954803-h9klh-worker-role policy=gpei-bz1954803-h9klh-worker-policy
INFO Deleted                                       id=gpei-bz1954803-h9klh-worker-role name=gpei-bz1954803-h9klh-worker-role
INFO Deleted                                       id=sg-0bd55b74951742e3a
INFO Deleted                                       id=a29826abb078044158dab57c78d1ccba
INFO Deleted                                       id=gpei-bz1954803-h9klh-openshift-machine-api-aws-hzg5t policy=gpei-bz1954803-h9klh-openshift-machine-api-aws-hzg5t-policy
INFO Deleted                                       id=gpei-bz1954803-h9klh-openshift-machine-api-aws-hzg5t
INFO Deleted                                       id=net/gpei-bz1954803-h9klh-ext/6fa19b26d3c1b8e8
INFO Deleted                                       id=gpei-bz1954803-h9klh-aws-ebs-csi-driver-operator-5pvgx policy=gpei-bz1954803-h9klh-aws-ebs-csi-driver-operator-5pvgx-policy
INFO Deleted                                       id=gpei-bz1954803-h9klh-aws-ebs-csi-driver-operator-5pvgx
INFO Deleted                                       id=gpei-bz1954803-h9klh-sint/45b8e919cb05dee3
INFO Deleted                                       id=eni-07c73a52ad7e598d2
INFO Deleted                                      
INFO Deleted                                       id=eni-0796ee4c44dc44d2f
INFO Deleted                                       id=sg-0477a519356e201f7
INFO Deleted                                       id=gpei-bz1954803-h9klh-aext/556ddc8284c33964
INFO Deleted                                       id=gpei-bz1954803-h9klh-aint/a94d7dc50692148a
INFO Deleted                                       id=sg-0065f471b928101ab
INFO Removed tag kubernetes.io/cluster/gpei-bz1954803-h9klh: shared  arn=arn:aws:ec2:us-east-2:301721915996:subnet/subnet-0ea5e3f656ac32af9
INFO Removed tag kubernetes.io/cluster/gpei-bz1954803-h9klh: shared  arn=arn:aws:ec2:us-east-2:301721915996:subnet/subnet-06271fc2256e8bd90
INFO Removed tag kubernetes.io/cluster/gpei-bz1954803-h9klh: shared  arn=arn:aws:ec2:us-east-2:301721915996:subnet/subnet-03be7dfd33e6f5d24
INFO Removed tag kubernetes.io/cluster/gpei-bz1954803-h9klh: shared  arn=arn:aws:ec2:us-east-2:301721915996:subnet/subnet-0706df2ecca1bdc62
INFO Deleted                                       arn=arn:aws:route53:::hostedzone/Z0197489DELWZQPZ0ZOL id=Z0197489DELWZQPZ0ZOL record set=A api-int.gpei-bz1954803.qe.devcluster.openshift.com.
INFO Deleted                                       arn=arn:aws:route53:::hostedzone/Z0197489DELWZQPZ0ZOL id=Z0197489DELWZQPZ0ZOL record set=A api.gpei-bz1954803.qe.devcluster.openshift.com.
INFO Deleted                                       arn=arn:aws:route53:::hostedzone/Z0197489DELWZQPZ0ZOL id=Z0197489DELWZQPZ0ZOL record set=A \052.apps.gpei-bz1954803.qe.devcluster.openshift.com.
INFO Cleaned record sets from hosted zone          arn=arn:aws:route53:::hostedzone/Z0197489DELWZQPZ0ZOL id=Z0197489DELWZQPZ0ZOL
INFO Removed tag kubernetes.io/cluster/gpei-bz1954803-h9klh: shared  arn=arn:aws:route53:::hostedzone/Z0197489DELWZQPZ0ZOL
INFO Time elapsed: 6m23s

Comment 1 Gaoyun Pei 2021-05-06 08:44:09 UTC

Also reproducible in 4.8.0-0.nightly-2021-04-30-201824.

Comment 4 Yunfei Jiang 2021-05-20 03:59:52 UTC

verified. PASS.
OCP version: 4.8.0-0.nightly-2021-05-19-123944

Comment 5 Yunfei Jiang 2021-05-31 07:27:06 UTC

re-open this bug, since there is log message issue in the destroy log, please see https://bugzilla.redhat.com/show_bug.cgi?id=1962435#c3
> in the .openshift_install.log, the public zone id should be Z3B3KOVA3TRCWP (qe.devcluster.openshift.com), instead of the private zone id

Comment 6 Yunfei Jiang 2021-05-31 07:43:33 UTC

mark this bug as verified, will create a new bug to track the log issue.

Comment 9 errata-xmlrpc 2021-07-27 23:06:39 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438

Note You need to log in before you can comment on or make changes to this bug.