Bug 1965969 - [aws] the public hosted zone id is not correct in the destroy log, while destroying a cluster which is using BYO private hosted zone.
Summary: [aws] the public hosted zone id is not correct in the destroy log, while dest...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.8
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 4.11.0
Assignee: Nobody
QA Contact: Yunfei Jiang
URL:
Whiteboard:
Depends On:
Blocks: 2051333
TreeView+ depends on / blocked
 
Reported: 2021-05-31 09:27 UTC by Yunfei Jiang
Modified: 2022-08-10 10:36 UTC (History)
0 users

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The destroyer incorrectly reports the ID of the private route53 hosted zone for the cluster when deleting DNS records from the hosted zone of the base domain. Consequence: Wrong hosted zone ID reported in the log of the destroyer. Fix: Use the proper hosted zone ID in the log. Result: The log of the destroyer shows the correct hosted zone ID when destroying the DNS records in the base domain's hosted zone.
Clone Of:
: 2051333 (view as bug list)
Environment:
Last Closed: 2022-08-10 10:36:25 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 5343 0 None open Bug 1965969: aws: Output public zone id correctly when deleting hosted dns records 2021-10-28 21:48:53 UTC
Github openshift installer pull 5494 0 None open Bug 1965969: aws: Fix dns destroy logs 2021-12-16 19:53:17 UTC
Red Hat Product Errata RHSA-2022:5069 0 None None None 2022-08-10 10:36:35 UTC

Description Yunfei Jiang 2021-05-31 09:27:30 UTC 
What happened?
When destroying an IPI-on-AWS cluster which is using byo private hosted zone, the log message is not correct when deleting records in public hosted zone:

the public zone id should be the public hosted zone id (Z3B3KOVA3TRCWP, qe.devcluster.openshift.com), instead of the private zone id (Z06189806E885EX74815)

time="2021-05-31T02:22:42-04:00" level=debug msg="Deleted from public zone" arn="arn:aws:route53:::hostedzone/Z06189806E885EX74815" id=Z06189806E885EX74815 recordset="api-int.yunjiang-bz435a.qe.devcluster.openshift.com. (A)"
time="2021-05-31T02:22:43-04:00" level=debug msg="Deleted from public zone" arn="arn:aws:route53:::hostedzone/Z06189806E885EX74815" id=Z06189806E885EX74815 recordset="api.yunjiang-bz435a.qe.devcluster.openshift.com. (A)"
time="2021-05-31T02:22:43-04:00" level=debug msg="Deleted from public zone" arn="arn:aws:route53:::hostedzone/Z06189806E885EX74815" id=Z06189806E885EX74815 recordset="\\052.apps.yunjiang-bz435a.qe.devcluster.openshift.com. (A)"

the original bugs bug 1962435 (4.7) and bug 1957597 (4.8).

this issue affects both 4.7 and 4.8




OCP version:
4.7.0-0.nightly-2021-05-29-015423

What did you expect to happen?
the log message "Deleted from public zone" followed by public zone id.
Comment 1 Russell Teague 2021-07-12 18:15:38 UTC 
Will address this in a future sprint.
Comment 2 Russell Teague 2021-08-02 17:45:30 UTC 
Needs prioritized.
Comment 3 Russell Teague 2021-08-24 17:35:21 UTC 
Will review again for a future sprint.
Comment 6 Yunfei Jiang 2021-11-16 11:31:07 UTC 
verified. FAILED.
OCP version: 4.10.0-0.nightly-2021-11-15-034648

Got some confused info in log, e.g.
level=debug msg=Deleted arn=arn:aws:route53:::hostedzone/Z0839996PWILML3MAMMI id=/hostedzone/Z3B3KOVA3TRCWP recordset=api.yunjiang-969a.qe.devcluster.openshift.com. (A)

does api record come from public or private hosted zone?


config:
<--snip-->
baseDomain: qe.devcluster.openshift.com <-- zone id Z3B3KOVA3TRCWP
platform:
  aws:
    hostedZone: Z0839996PWILML3MAMMI
<--snip-->

destroy log:
level=debug msg=listing AWS hosted zones "yunjiang-969a.qe.devcluster.openshift.com." (page 0) arn=arn:aws:route53:::hostedzone/Z0839996PWILML3MAMMI
level=debug msg=listing AWS hosted zones "qe.devcluster.openshift.com." (page 0) arn=arn:aws:route53:::hostedzone/Z0839996PWILML3MAMMI
level=debug msg=Deleted from public zone arn=arn:aws:route53:::hostedzone/Z0839996PWILML3MAMMI id=/hostedzone/Z3B3KOVA3TRCWP recordset=api-int.yunjiang-969a.qe.devcluster.openshift.com. (A)
level=info msg=Deleted arn=arn:aws:route53:::hostedzone/Z0839996PWILML3MAMMI id=/hostedzone/Z3B3KOVA3TRCWP record set=A api-int.yunjiang-969a.qe.devcluster.openshift.com.
level=debug msg=Deleted arn=arn:aws:route53:::hostedzone/Z0839996PWILML3MAMMI id=/hostedzone/Z3B3KOVA3TRCWP recordset=api-int.yunjiang-969a.qe.devcluster.openshift.com. (A)
level=info msg=Deleted arn=arn:aws:route53:::hostedzone/Z0839996PWILML3MAMMI id=/hostedzone/Z3B3KOVA3TRCWP record set=A api.yunjiang-969a.qe.devcluster.openshift.com.
level=debug msg=Deleted from public zone arn=arn:aws:route53:::hostedzone/Z0839996PWILML3MAMMI id=/hostedzone/Z3B3KOVA3TRCWP recordset=api.yunjiang-969a.qe.devcluster.openshift.com. (A)
level=info msg=Deleted arn=arn:aws:route53:::hostedzone/Z0839996PWILML3MAMMI id=/hostedzone/Z3B3KOVA3TRCWP record set=A api.yunjiang-969a.qe.devcluster.openshift.com.
level=debug msg=Deleted arn=arn:aws:route53:::hostedzone/Z0839996PWILML3MAMMI id=/hostedzone/Z3B3KOVA3TRCWP recordset=api.yunjiang-969a.qe.devcluster.openshift.com. (A)
level=info msg=Deleted arn=arn:aws:route53:::hostedzone/Z0839996PWILML3MAMMI id=/hostedzone/Z3B3KOVA3TRCWP record set=A \052.apps.yunjiang-969a.qe.devcluster.openshift.com.
level=debug msg=Deleted from public zone arn=arn:aws:route53:::hostedzone/Z0839996PWILML3MAMMI id=/hostedzone/Z3B3KOVA3TRCWP recordset=\052.apps.yunjiang-969a.qe.devcluster.openshift.com. (A)
level=info msg=Deleted arn=arn:aws:route53:::hostedzone/Z0839996PWILML3MAMMI id=/hostedzone/Z3B3KOVA3TRCWP record set=A \052.apps.yunjiang-969a.qe.devcluster.openshift.com.
level=debug msg=Deleted arn=arn:aws:route53:::hostedzone/Z0839996PWILML3MAMMI id=/hostedzone/Z3B3KOVA3TRCWP recordset=\052.apps.yunjiang-969a.qe.devcluster.openshift.com. (A)
level=info msg=Cleaned record sets from hosted zone arn=arn:aws:route53:::hostedzone/Z0839996PWILML3MAMMI id=/hostedzone/Z3B3KOVA3TRCWP
level=info msg=Removed tag kubernetes.io/cluster/yunjiang-969a-lpj6h: shared arn=arn:aws:route53:::hostedzone/Z0839996PWILML3MAMMI
Comment 10 Yunfei Jiang 2022-01-10 11:34:05 UTC 
verified. FAILED.

OCP Version: 4.10.0-0.nightly-2022-01-10-014106

> grep hostedzone .openshift_install.log
time="2022-01-10T05:09:15-05:00" level=debug msg="No cluster domain specified in metadata; cannot clean the shared hosted zone" arn="arn:aws:route53:::hostedzone/Z017831332AP9QNK19P4I" id=Z017831332AP9QNK19P4I
time="2022-01-10T05:09:16-05:00" level=info msg="Removed tag kubernetes.io/cluster/yunjiang-bz969a-b24kb: shared" arn="arn:aws:route53:::hostedzone/Z017831332AP9QNK19P4I"

issues:
1. records in public zone were not deleted.
2. records in BYO private zone were not deleted.
Comment 12 Yunfei Jiang 2022-01-30 06:22:36 UTC 
Hello Staebler, I noticed the target release has been set to None now, but new issues (comment 10) have been introduced by https://github.com/openshift/installer/pull/5494, this needs to be resolved in 4.10.
Comment 13 Matthew Staebler 2022-01-31 21:04:10 UTC 
(In reply to Yunfei Jiang from comment #12)
> Hello Staebler, I noticed the target release has been set to None now, but
> new issues (comment 10) have been introduced by
> https://github.com/openshift/installer/pull/5494, this needs to be resolved
> in 4.10.

Please file a separate BZ for the new issue.
Comment 14 Matthew Staebler 2022-02-25 13:22:58 UTC 
Moving this back to ON_QA, but testing of this is blocked on https://bugzilla.redhat.com/show_bug.cgi?id=2051333.
Comment 15 Yunfei Jiang 2022-03-01 10:40:28 UTC 
verification failed.

OCP version: 4.11.0-0.nightly-2022-02-27-122819

> install-config:

platform:
  aws:
    region: us-east-2
    subnets:
    - subnet-0cce8bda6928d94e5
    - subnet-097e6bcbbe614f53b
    - subnet-02fe26fcb7a49f818
    - subnet-0c7522cad1fc1a938
    hostedZone: Z0273356ZY68YOEOT1OY
publish: External
baseDomain: qe.devcluster.openshift.com

> error messages in .openshift_install.log:


                                                private zone id                  public zone id
                                                        |                                |
                                                        V                                V
INFO Deleted arn=arn:aws:route53:::hostedzone/Z0273356ZY68YOEOT1OY id=/hostedzone/Z3B3KOVA3TRCWP record set=A api.yunjiang-bz969.qe.devcluster.openshift.com.
INFO Deleted arn=arn:aws:route53:::hostedzone/Z0273356ZY68YOEOT1OY id=/hostedzone/Z3B3KOVA3TRCWP record set=A \052.apps.yunjiang-bz969.qe.devcluster.openshift.com.


> .openshift_install.log:

<--SNIP-->
INFO Deleted                                       arn=arn:aws:route53:::hostedzone/Z0273356ZY68YOEOT1OY id=Z0273356ZY68YOEOT1OY record set=A api-int.yunjiang-bz969.qe.devcluster.openshift.com.
INFO Deleted                                       arn=arn:aws:route53:::hostedzone/Z0273356ZY68YOEOT1OY id=/hostedzone/Z3B3KOVA3TRCWP record set=A api.yunjiang-bz969.qe.devcluster.openshift.com.
INFO Deleted                                       arn=arn:aws:route53:::hostedzone/Z0273356ZY68YOEOT1OY id=Z0273356ZY68YOEOT1OY record set=A api.yunjiang-bz969.qe.devcluster.openshift.com.
INFO Deleted                                       arn=arn:aws:route53:::hostedzone/Z0273356ZY68YOEOT1OY id=/hostedzone/Z3B3KOVA3TRCWP record set=A \052.apps.yunjiang-bz969.qe.devcluster.openshift.com.
INFO Deleted                                       arn=arn:aws:route53:::hostedzone/Z0273356ZY68YOEOT1OY id=Z0273356ZY68YOEOT1OY record set=A \052.apps.yunjiang-bz969.qe.devcluster.openshift.com.
INFO Cleaned record sets from hosted zone          arn=arn:aws:route53:::hostedzone/Z0273356ZY68YOEOT1OY id=Z0273356ZY68YOEOT1OY
INFO Removed tag kubernetes.io/cluster/yunjiang-bz969-prnrb: shared  arn=arn:aws:route53:::hostedzone/Z0273356ZY68YOEOT1OY
INFO Time elapsed: 4m9s
Comment 16 Matthew Staebler 2022-03-01 16:25:17 UTC 
The arn field is always going to be for the private zone. The arn is the resource that the destroyer is responding to. The relevant part of the log message is the id field.
Comment 17 Yunfei Jiang 2022-03-02 03:25:52 UTC 
per comment 16, the result in comment 15 is as expected.
Comment 23 errata-xmlrpc 2022-08-10 10:36:25 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069
Note You need to log in before you can comment on or make changes to this bug.