Bug 1614984

Summary: [3.10] Intermittent dnsmasq outages
Product: OpenShift Container Platform Reporter: Miciah Dashiel Butler Masters <mmasters>
Component: RoutingAssignee: Miciah Dashiel Butler Masters <mmasters>
Status: CLOSED ERRATA QA Contact: zhaozhanqi <zzhao>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 3.10.0CC: aos-bugs, bbennett, farandac, hongli, pdwyer, rbost, rfoyle, rhowe, weliang, zzhao
Target Milestone: ---   
Target Release: 3.10.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: By default, older versions of dnsmasq can use privileged, lower-numbered source ports for outbound DNS queries. Consequence: Outbound DNS queries may be dropped; for example, firewall rules may drop queries coming from reserved ports. Fix: We now configure dnsmasq using its min-port setting to set the minimum port number for outbound queries to 1024. Result: DNS queries should no longer be dropped. Additional information: dnsmasq 2.79 changes the default min-port setting to 1024.
 Story Points: ---
Clone Of: 1600551 Environment:
Last Closed: 2018-08-31 06:18:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1600551    
Bug Blocks:    

Description Miciah Dashiel Butler Masters 2018-08-10 23:06:44 UTC 
Cloned for 3.10.z backport.

+++ This bug was initially created as a clone of Bug #1600551 +++

Description of problem: Pods are experiencing intermittent DNS lookup failures when reaching out to dnsmasq. A similar upstream issue has been reported: https://github.com/kubernetes/kubernetes/issues/45976

[...]

--- Additional comment from Ryan Howe on 2018-08-09 10:28:47 EDT ---

Working another OpenShift dnsmasq issue we figured the issue to happen when dnsmasq uses a low port number. 

Setting min-port=1024 in dnsmasq worked around the issue. 

--min-port=<port>
              Do not use ports less than that given as source for outbound DNS queries. Dnsmasq picks random ports as source for outbound queries: when this option is given, the ports used will always to larger than that  specified. Useful for systems behind firewalls.


Dnsmasq bug was logged: 
  https://bugzilla.redhat.com/show_bug.cgi?id=1614331


I was not able to reproduce the issue again with this configuration in place.

--- Additional comment from Ryan Howe on 2018-08-09 11:11:26 EDT ---

Created PR to add this configuration via the ansible installer for OpenShift: 

https://github.com/openshift/openshift-ansible/pull/9505

[...]
Comment 1 Miciah Dashiel Butler Masters 2018-08-10 23:15:26 UTC 
OCP 3.10.z backport: https://github.com/openshift/openshift-ansible/pull/9542
Comment 3 Weibin Liang 2018-08-23 19:53:26 UTC 
Tested and verified in v3.10.35

[root@qe-weliang-3 ~]# oc version
oc v3.10.35
kubernetes v1.10.0+b81c8f8
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://qe-weliang-3.10master-etcd-nfs-1:8443
openshift v3.10.35
kubernetes v1.10.0+b81c8f8
[root@qe-weliang-3 ~]# cat /etc/dnsmasq.d/origin-dns.conf | grep min
min-port=1024
[root@qe-weliang-3 ~]# cat /etc/NetworkManager/dispatcher.d/99-origin-dns.sh | grep mi
  # couldn't find an existing method to determine if the interface owns the
min-port=1024
[root@qe-weliang-3 ~]#
Comment 5 errata-xmlrpc 2018-08-31 06:18:11 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2376