Bug 1766141

Summary: [GCP] the DNS record for the wildcard apps is not updated after recreating the default ingresscontroller
Product: OpenShift Container Platform Reporter: Hongan Li <hongli>
Component: NetworkingAssignee: Daneyon Hansen <dhansen>
Networking sub component: router QA Contact: Hongan Li <hongli>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: aos-bugs
Version: 4.3.0   
Target Milestone: ---   
Target Release: 4.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1769882 (view as bug list) Environment:
Last Closed: 2020-01-23 11:09:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1769882    

Description Hongan Li 2019-10-28 12:04:53 UTC
Description of problem:
After recreating the default ingresscontroller, the LB svc gets a new external IP but DNS still has the old one.  

Version-Release number of selected component (if applicable):
4.3.0-0.nightly-2019-10-26-020736

How reproducible:
100%

Steps to Reproduce:
1. delete the default ingresscontroller
2. wait for new ingresscontroller and related resources are recreated
3.

Actual results:
the DNS record for the wildcard apps is not updated

$ oc get svc -n openshift-ingress
NAME                      TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                      AGE
router-default            LoadBalancer   172.30.5.186    34.66.229.177   80:31048/TCP,443:31638/TCP   8h

$ nslookup console-openshift-console.apps.hongli-g36.qe.gcp.devcluster.openshift.com
Server:		10.68.5.26
Address:	10.68.5.26#53

Non-authoritative answer:
Name:	console-openshift-console.apps.hongli-g36.qe.gcp.devcluster.openshift.com
Address: 35.226.137.24


$ nslookup api.hongli-g36.qe.gcp.devcluster.openshift.com
Server:		10.68.5.26
Address:	10.68.5.26#53

Non-authoritative answer:
Name:	api.hongli-g36.qe.gcp.devcluster.openshift.com
Address: 34.70.214.147


Expected results:
the DNS record for the wildcard apps should be updated with new external IP.

Additional info:

Comment 2 Dan Mace 2019-10-28 13:32:34 UTC
Thanks for the report!

Comment 3 Daneyon Hansen 2019-10-30 23:00:43 UTC
Can you provide the dnsrecord for a) before deleting the default ingresscontroller b) after the default ingresscontroller is deleted c) after the default ingresscontroller is recreated:

$ oc get dnsrecord/default-wildcard -n openshift-ingress-operator -o yaml

It would be helpful to attach the ingress-operator logs for the time period from when the default ingresscontroller gets deleted to when it gets recreated.

Comment 8 Hongan Li 2019-11-07 06:20:54 UTC
verified with 4.3.0-0.nightly-2019-11-06-184828 and issue has been fixed

Comment 10 errata-xmlrpc 2020-01-23 11:09:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0062

Comment 11 Daneyon Hansen 2020-01-27 21:07:50 UTC
To be clear, k8s v1.15 added support for garbage collecting services of type loadbalancer [1]. This feature was introduced in k8s v1.15 by enabling the ServiceLoadBalancerFinalizer feature gate [2]. The feature is enabled by default in subsequent k8s releases. [3][4] causes to ingress operator to remove the "ingress.openshift.io/operator" finalizer only if the associated lb service has been deleted. Prior to [1], a lb service will be deleted without the cloud infra being garbage collected.

[1] https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#garbage-collecting-load-balancers
[2] https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates
[3] https://github.com/openshift/cluster-ingress-operator/pull/320
[4] https://github.com/openshift/cluster-ingress-operator/pull/330