Bug 1943578

Summary: CoreDNS caches NXDOMAIN responses for up to 900 seconds
Product: OpenShift Container Platform Reporter: Stephen Greene <sgreene>
Component: NetworkingAssignee: Stephen Greene <sgreene>
Networking sub component: DNS QA Contact: Arvind iyengar <aiyengar>
Status: CLOSED ERRATA Docs Contact:
Severity: urgent    
Priority: urgent CC: aiyengar, aos-bugs, hongli, mjoseph, oarribas, otuchfel
Version: 4.6   
Target Milestone: ---   
Target Release: 4.8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Bug 1933761 set the global CoreDNS cache max TTL to 900 seconds. Consequence: NXDOMAIN records received from upstream resolvers are cached for 900 seconds. Fix: Explicitly cache negative DNS response records for maximum 30 seconds. Result: Resolving domains that are in the process of being published does not take at minimum 15 minutes.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2021-07-27 22:56:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1943826    

Description Stephen Greene 2021-03-26 13:45:31 UTC 
Description of problem:

Bug 1933761 set that max TTL of DNS records in CoreDNS's cache to 900 seconds. The previous max TTL before Bug 1933761 was 30 seconds.

With Bug 1933761 in place, CoreDNS now caches negative DNS responses from the upstream resolver (such as NXDOMAIN) for up to 900 seconds (depending on the TTL set by the an upstream resolver). This is undesired behavior since waiting 15 minutes to retry a query for a domain that may have recently been registered can increase cluster install time in some cases unnecessarily.


Version-Release number of selected component (if applicable):
4.6, 4.7, and 4.8

Note that this issue is more likely to cause problems in compact or single node clusters as the overall number of CoreDNS cache's decreases (the service proxy load-balances DNS queries, so the odds of hitting a CoreDNS instance with a cached NXDOMAIN response for a domain decreases as cluster size increases).
Comment 1 Stephen Greene 2021-03-26 19:59:35 UTC 
*** Bug 1939070 has been marked as a duplicate of this bug. ***
Comment 3 Arvind iyengar 2021-03-29 05:36:19 UTC 
Verified in "4.8.0-0.nightly-2021-03-29-000904" release version. With this payload it is observed that the additional configuration of 30 second TTL for negative records get set by default along with 900 seconds for positive record in cache plugin section:
-----
oc get clusterversion                           
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-03-29-000904   True        False         46m     Cluster version is 4.8.0-0.nightly-2021-03-29-000904

Use 'oc describe pod/dns-default-7xz8b -n openshift-dns' to see all of the containers in this pod.
.:5353 {
    errors
    health {
        lameduck 20s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        fallthrough in-addr.arpa ip6.arpa
    }
    prometheus 127.0.0.1:9153
    forward . /etc/resolv.conf {
        policy sequential
    }
    cache 900 {      <----
        denial 9984 30 <---
    }
    reload
}
-----
Comment 6 errata-xmlrpc 2021-07-27 22:56:00 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438