Bug 1953097

Summary: CoreDNS resolution failure for external hostnames with "A: dns: overflow unpacking uint16"
Product: OpenShift Container Platform Reporter: Stephen Greene <sgreene>
Component: NetworkingAssignee: Stephen Greene <sgreene>
Networking sub component: DNS QA Contact: jechen <jechen>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: amcdermo, aos-bugs, arghosh, dahernan, fgiloux, hongli, mapandey, mjoseph, palonsor, rcernin, rsales, sgreene
Version: 4.5   
Target Milestone: ---   
Target Release: 4.7.z   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Cluster upstream resolver returns DNS response that exceeds 512 bytes via UDP. Consequence: CoreDNS may return SERVFAIL and or log various error messages, sometimes forcing the client to retry over TCP. Fix: Enable the CoreDNS bufisze plugin with a UDP buffer size of 1232 bytes. Result: CoreDNS is less likely to return SERVFAIL or present any runtime errors when handling large DNS responses via UDP. Also, UDP packet fragmentation is less likely to occur.
 Story Points: ---
Clone Of: 1949361
: 1955632 (view as bug list) Environment:
Last Closed: 2021-05-19 15:16:26 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1949361    
Bug Blocks: 1955632    

Comment 4 jechen 2021-05-03 15:23:37 UTC 
Verified in 4.7.0-0.nightly-2021-05-01-081439

$ oc get clusterversions.config.openshift.io 
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.nightly-2021-05-01-081439   True        False         2m26s   Cluster version is 4.7.0-0.nightly-2021-05-01-081439


# verified bufsize in default DNS
[jechen@jechen ~]$ oc -n openshift-dns get cm/dns-default -oyaml
apiVersion: v1
data:
  Corefile: |
    .:5353 {
        bufsize 1232              <-- verified fix by https://github.com/openshift/cluster-dns-operator/pull/267
        errors
        health {
            lameduck 20s
        }
<--snip-->


# created a test pod as custom DNS server, added custom forwarding into coreDNS, verified bufsize in custom DNS
$ oc create -f https://raw.githubusercontent.com/lihongan/test-dnsmasq/master/test-dnsmasq.json 
pod/test-dnsmasq created

$ oc get pod -owide
NAME           READY   STATUS    RESTARTS   AGE   IP            NODE                                       NOMINATED NODE   READINESS GATES
test-dnsmasq   1/1     Running   0          58s   10.131.0.31   ci-ln-0bskzdt-f76d1-fptks-worker-b-rk66z   <none>           <none>

$ oc edit dns.operator/default 
<--snip-->
spec:
  servers:
  - forwardPlugin:
      upstreams:
      - 10.131.0.31
    name: test
    zones:
    - mytest.ocp
<--snip-->



$ oc get cm dns-default -n openshift-dns -o yaml
apiVersion: v1
data:
  Corefile: |
    # test
    mytest.ocp:5353 {
        forward . 10.131.0.31
        errors
        bufsize 1232                    <-- verified fix by https://github.com/openshift/cluster-dns-operator/pull/267
    }
    .:5353 {
        bufsize 1232                    <-- verified fix by https://github.com/openshift/cluster-dns-operator/pull/267
        errors
        health {
            lameduck 20s
        }
<--snip-->
Comment 5 Siddharth Sharma 2021-05-10 17:59:31 UTC 
This bug will be shipped as part of next z-stream release 4.7.11 on May 19th, as 4.7.10 was dropped due to a blocker https://bugzilla.redhat.com/show_bug.cgi?id=1958518.
Comment 9 errata-xmlrpc 2021-05-19 15:16:26 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.7.11 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1550