Bug 1933761
| Summary: | Cluster DNS service caps TTLs too low and thus evicts from its cache too aggressively | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Miciah Dashiel Butler Masters <mmasters> |
| Component: | Networking | Assignee: | Miciah Dashiel Butler Masters <mmasters> |
| Networking sub component: | DNS | QA Contact: | jechen <jechen> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | high | ||
| Priority: | high | CC: | agabriel, aos-bugs, bjarolim, hongli, oarribas, openshift-bugs-escalate, sgreene |
| Version: | 4.8 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.8.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-07-27 22:48:44 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1936587 | ||
*** Bug 1921797 has been marked as a duplicate of this bug. *** Verified in 4.8.0-0.nightly-2021-03-08-092651
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.8.0-0.nightly-2021-03-08-092651 True False 68m Cluster version is 4.8.0-0.nightly-2021-03-08-092651
$ oc -n openshift-dns get configmaps/dns-default -o yaml
apiVersion: v1
data:
Corefile: |
.:5353 {
errors
health
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
}
prometheus 127.0.0.1:9153
forward . /etc/resolv.conf {
policy sequential
}
cache 900 <--- verified the fix with https://github.com/openshift/cluster-dns-operator/pull/240/
reload
}
<----snip----->
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438 |
Description of problem: The CoreDNS instances that provide cluster DNS service are configured with a cache timeout of 30 seconds. This can cause DNS lookups to take an excessive amount of time if forwarded queries overload upstream resolvers. Version-Release number of selected component (if applicable): All versions of OCP 4 so far. Steps to Reproduce: 1. oc -n openshift-dns get configmaps/dns-default -o yaml Actual results: The maximum cache time is 30 seconds: cache 30 Expected results: The maximum cache time should be higher (at least 900 seconds). Additional info: OpenShift 3 used dnsmasq (along with SkyDNS) for the cluster DNS service, and the dnsmasq configuration was largely unmanaged, so cluster administrators could configure customize its configuration with little or no restrictions. OpenShift 4 replaces dnsmasq and SkyDNS with CoreDNS, and the DNS operator manages the CoreDNS configuration, which prevents administrators from making arbitrary modifications to it. The cache timeout effectively puts an upper limit on DNS records' TTLs. When DNS records are properly configured, capping their TTLs provides no advantage but does have the disadvantage of causing more queries to be forwarded to the upstream resolver. Therefore we should increase this upper limit to avoid performance problems.