Bug 1614984 - [3.10] Intermittent dnsmasq outages
Summary: [3.10] Intermittent dnsmasq outages
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 3.10.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 3.10.z
Assignee: Miciah Dashiel Butler Masters
QA Contact: zhaozhanqi
URL:
Whiteboard:
Depends On: 1600551
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-08-10 23:06 UTC by Miciah Dashiel Butler Masters
Modified: 2022-08-04 22:20 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: By default, older versions of dnsmasq can use privileged, lower-numbered source ports for outbound DNS queries. Consequence: Outbound DNS queries may be dropped; for example, firewall rules may drop queries coming from reserved ports. Fix: We now configure dnsmasq using its min-port setting to set the minimum port number for outbound queries to 1024. Result: DNS queries should no longer be dropped. Additional information: dnsmasq 2.79 changes the default min-port setting to 1024.
Clone Of: 1600551
Environment:
Last Closed: 2018-08-31 06:18:11 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift openshift-ansible pull 9542 0 'None' closed [3.10] Adding min-port to dnsmasq configuration 2020-10-21 11:03:10 UTC
Red Hat Product Errata RHBA-2018:2376 0 None None None 2018-08-31 06:18:22 UTC

Description Miciah Dashiel Butler Masters 2018-08-10 23:06:44 UTC 
Cloned for 3.10.z backport.

+++ This bug was initially created as a clone of Bug #1600551 +++

Description of problem: Pods are experiencing intermittent DNS lookup failures when reaching out to dnsmasq. A similar upstream issue has been reported: https://github.com/kubernetes/kubernetes/issues/45976

[...]

--- Additional comment from Ryan Howe on 2018-08-09 10:28:47 EDT ---

Working another OpenShift dnsmasq issue we figured the issue to happen when dnsmasq uses a low port number. 

Setting min-port=1024 in dnsmasq worked around the issue. 

--min-port=<port>
              Do not use ports less than that given as source for outbound DNS queries. Dnsmasq picks random ports as source for outbound queries: when this option is given, the ports used will always to larger than that  specified. Useful for systems behind firewalls.


Dnsmasq bug was logged: 
  https://bugzilla.redhat.com/show_bug.cgi?id=1614331


I was not able to reproduce the issue again with this configuration in place.

--- Additional comment from Ryan Howe on 2018-08-09 11:11:26 EDT ---

Created PR to add this configuration via the ansible installer for OpenShift: 

https://github.com/openshift/openshift-ansible/pull/9505

[...]
Comment 1 Miciah Dashiel Butler Masters 2018-08-10 23:15:26 UTC 
OCP 3.10.z backport: https://github.com/openshift/openshift-ansible/pull/9542
Comment 3 Weibin Liang 2018-08-23 19:53:26 UTC 
Tested and verified in v3.10.35

[root@qe-weliang-3 ~]# oc version
oc v3.10.35
kubernetes v1.10.0+b81c8f8
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://qe-weliang-3.10master-etcd-nfs-1:8443
openshift v3.10.35
kubernetes v1.10.0+b81c8f8
[root@qe-weliang-3 ~]# cat /etc/dnsmasq.d/origin-dns.conf | grep min
min-port=1024
[root@qe-weliang-3 ~]# cat /etc/NetworkManager/dispatcher.d/99-origin-dns.sh | grep mi
  # couldn't find an existing method to determine if the interface owns the
min-port=1024
[root@qe-weliang-3 ~]#
Comment 5 errata-xmlrpc 2018-08-31 06:18:11 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2376
Note You need to log in before you can comment on or make changes to this bug.