Bug 1614984 - [3.10] Intermittent dnsmasq outages
Summary: [3.10] Intermittent dnsmasq outages
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Routing
Version: 3.10.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.10.z
Assignee: Miciah Dashiel Butler Masters
QA Contact: zhaozhanqi
Depends On: 1600551
TreeView+ depends on / blocked
Reported: 2018-08-10 23:06 UTC by Miciah Dashiel Butler Masters
Modified: 2018-10-25 07:14 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: By default, older versions of dnsmasq can use privileged, lower-numbered source ports for outbound DNS queries. Consequence: Outbound DNS queries may be dropped; for example, firewall rules may drop queries coming from reserved ports. Fix: We now configure dnsmasq using its min-port setting to set the minimum port number for outbound queries to 1024. Result: DNS queries should no longer be dropped. Additional information: dnsmasq 2.79 changes the default min-port setting to 1024.
Clone Of: 1600551
Last Closed: 2018-08-31 06:18:11 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Priority Status Summary Last Updated
Github openshift openshift-ansible pull 9542 'None' 'closed' '[3.10] Adding min-port to dnsmasq configuration' 2019-11-19 04:20:16 UTC
Red Hat Product Errata RHBA-2018:2376 None None None 2018-08-31 06:18:22 UTC

Description Miciah Dashiel Butler Masters 2018-08-10 23:06:44 UTC
Cloned for 3.10.z backport.

+++ This bug was initially created as a clone of Bug #1600551 +++

Description of problem: Pods are experiencing intermittent DNS lookup failures when reaching out to dnsmasq. A similar upstream issue has been reported: https://github.com/kubernetes/kubernetes/issues/45976


--- Additional comment from Ryan Howe on 2018-08-09 10:28:47 EDT ---

Working another OpenShift dnsmasq issue we figured the issue to happen when dnsmasq uses a low port number. 

Setting min-port=1024 in dnsmasq worked around the issue. 

              Do not use ports less than that given as source for outbound DNS queries. Dnsmasq picks random ports as source for outbound queries: when this option is given, the ports used will always to larger than that  specified. Useful for systems behind firewalls.

Dnsmasq bug was logged: 

I was not able to reproduce the issue again with this configuration in place.

--- Additional comment from Ryan Howe on 2018-08-09 11:11:26 EDT ---

Created PR to add this configuration via the ansible installer for OpenShift: 



Comment 1 Miciah Dashiel Butler Masters 2018-08-10 23:15:26 UTC
OCP 3.10.z backport: https://github.com/openshift/openshift-ansible/pull/9542

Comment 3 Weibin Liang 2018-08-23 19:53:26 UTC
Tested and verified in v3.10.35

[root@qe-weliang-3 ~]# oc version
oc v3.10.35
kubernetes v1.10.0+b81c8f8
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://qe-weliang-3.10master-etcd-nfs-1:8443
openshift v3.10.35
kubernetes v1.10.0+b81c8f8
[root@qe-weliang-3 ~]# cat /etc/dnsmasq.d/origin-dns.conf | grep min
[root@qe-weliang-3 ~]# cat /etc/NetworkManager/dispatcher.d/99-origin-dns.sh | grep mi
  # couldn't find an existing method to determine if the interface owns the
[root@qe-weliang-3 ~]#

Comment 5 errata-xmlrpc 2018-08-31 06:18:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.