Bug 1933761 - Cluster DNS service caps TTLs too low and thus evicts from its cache too aggressively
Summary: Cluster DNS service caps TTLs too low and thus evicts from its cache too aggr...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.8
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.8.0
Assignee: Miciah Dashiel Butler Masters
QA Contact: jechen
URL:
Whiteboard:
Depends On:
Blocks: 1936587
TreeView+ depends on / blocked
 
Reported: 2021-03-01 16:27 UTC by Miciah Dashiel Butler Masters
Modified: 2022-08-04 22:39 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-07-27 22:48:44 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-dns-operator pull 240 0 None open Bug 1933761: Set CoreDNS's cache's maximum TTL to 900 seconds 2021-03-04 15:17:22 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 22:49:03 UTC

Internal Links: 1943578

Description Miciah Dashiel Butler Masters 2021-03-01 16:27:49 UTC
Description of problem:

The CoreDNS instances that provide cluster DNS service are configured with a cache timeout of 30 seconds.  This can cause DNS lookups to take an excessive amount of time if forwarded queries overload upstream resolvers.  


Version-Release number of selected component (if applicable):

All versions of OCP 4 so far.


Steps to Reproduce:

1. oc -n openshift-dns get configmaps/dns-default -o yaml


Actual results:

The maximum cache time is 30 seconds:

            cache 30


Expected results:

The maximum cache time should be higher (at least 900 seconds).


Additional info:

OpenShift 3 used dnsmasq (along with SkyDNS) for the cluster DNS service, and the dnsmasq configuration was largely unmanaged, so cluster administrators could configure customize its configuration with little or no restrictions.  OpenShift 4 replaces dnsmasq and SkyDNS with CoreDNS, and the DNS operator manages the CoreDNS configuration, which prevents administrators from making arbitrary modifications to it.  

The cache timeout effectively puts an upper limit on DNS records' TTLs.  When DNS records are properly configured, capping their TTLs provides no advantage but does have the disadvantage of causing more queries to be forwarded to the upstream resolver.  Therefore we should increase this upper limit to avoid performance problems.

Comment 1 Ben Bennett 2021-03-04 14:56:45 UTC
*** Bug 1921797 has been marked as a duplicate of this bug. ***

Comment 3 jechen 2021-03-08 15:26:57 UTC
Verified in 4.8.0-0.nightly-2021-03-08-092651

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-03-08-092651   True        False         68m     Cluster version is 4.8.0-0.nightly-2021-03-08-092651

$ oc -n openshift-dns get configmaps/dns-default -o yaml
apiVersion: v1
data:
  Corefile: |
    .:5353 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
        }
        prometheus 127.0.0.1:9153
        forward . /etc/resolv.conf {
            policy sequential
        }
        cache 900                <--- verified the fix with https://github.com/openshift/cluster-dns-operator/pull/240/
        reload
    }
<----snip----->

Comment 6 errata-xmlrpc 2021-07-27 22:48:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438


Note You need to log in before you can comment on or make changes to this bug.