Bug 1944245

Summary: CoreDNS caches NXDOMAIN responses for up to 900 seconds
Product: OpenShift Container Platform Reporter: OpenShift BugZilla Robot <openshift-bugzilla-robot>
Component: NetworkingAssignee: Stephen Greene <sgreene>
Networking sub component: DNS QA Contact: Hongan Li <hongli>
Status: CLOSED ERRATA Docs Contact:
Severity: urgent    
Priority: urgent CC: aos-bugs, jdelft, jeder, mmasters, otuchfel, wking
Version: 4.6Keywords: ServiceDeliveryBlocker
Target Milestone: ---   
Target Release: 4.6.z   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Bug 1936587 set the global CoreDNS cache max TTL to 900 seconds. Consequence: NXDOMAIN records received from upstream resolvers are cached for 900 seconds. Fix: Explicitly cache negative DNS response records for maximum 30 seconds. Result: Resolving domains that are in the process of being published does not take at minimum 15 minutes.
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-04-20 19:27:50 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1943826    
Bug Blocks:    

Comment 1 Miciah Dashiel Butler Masters 2021-03-29 15:29:44 UTC
To fix bug 1936589, we configured cluster DNS to honor ttl values of up to 15 minutes from upstream resolvers and cap higher ttl values to 15 minutes.  Before that change, ttl values were capped to 30 seconds.  Capping ttl values for nxdomain responses to 15 minutes instead of 30 seconds causes long delays (15 minutes) when provisioning service load-balancers, including the default ingress load-balancer that is provisioned when a cluster is installed.  Bug 1936589 is verified but not shipped.  The justification for marking this new BZ as a blocker is that we want to fix the problem introduced by the fix for bug 1936589 before it ships.

Comment 3 Hongan Li 2021-04-02 02:44:10 UTC
Verified with the cluster launched by cluster-bot (launch openshift/cluster-dns-operator#256) and passed.

$ oc get clusterversion
NAME      VERSION                                           AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-0.ci.test-2021-04-02-005725-ci-ln-9c610wt   True        False         59m     Cluster version is 4.6.0-0.ci.test-2021-04-02-005725-ci-ln-9c610wt

$ oc -n openshift-dns get cm/dns-default -oyaml
apiVersion: v1
  Corefile: |
    .:5353 {
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
        prometheus :9153
        forward . /etc/resolv.conf {
            policy sequential
        cache 900 {
            denial 9984 30

### check the TTL of positive response
sh-4.4# dig stackoverflow.com

; <<>> DiG 9.11.13-RedHat-9.11.13-6.el8_2.1 <<>> stackoverflow.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10317
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

; EDNS: version: 0, flags:; udp: 4096
;stackoverflow.com.		IN	A

stackoverflow.com.	300	IN	A
stackoverflow.com.	300	IN	A

### check the TTL of negative response
sh-4.4# dig nxdomain.google.com

; <<>> DiG 9.11.13-RedHat-9.11.13-6.el8_2.1 <<>> nxdomain.google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 24399
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

; EDNS: version: 0, flags:; udp: 4096
;nxdomain.google.com.		IN	A

google.com.		27	IN	SOA	ns1.google.com. dns-admin.google.com. 366215971 900 900 1800 60

Comment 10 errata-xmlrpc 2021-04-20 19:27:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6.25 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.