Bug 1614983 - [3.9] Intermittent dnsmasq outages
Summary: [3.9] Intermittent dnsmasq outages
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 3.9.z
Assignee: Miciah Dashiel Butler Masters
QA Contact: zhaozhanqi
URL:
Whiteboard:
: 1626248 (view as bug list)
Depends On: 1600551
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-08-10 23:06 UTC by Miciah Dashiel Butler Masters
Modified: 2022-08-04 22:20 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: By default, older versions of dnsmasq can use privileged, lower-numbered source ports for outbound DNS queries. Consequence: Outbound DNS queries may be dropped; for example, firewall rules may drop queries coming from reserved ports. Fix: We now configure dnsmasq using its min-port setting to set the minimum port number for outbound queries to 1024. Result: DNS queries should no longer be dropped. Additional information: dnsmasq 2.79 changes the default min-port setting to 1024.
Clone Of: 1600551
Environment:
Last Closed: 2018-08-29 14:42:32 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift openshift-ansible pull 9541 0 None closed [3.9] Adding min-port to dnsmasq configuration 2020-12-24 08:20:28 UTC
Red Hat Product Errata RHBA-2018:2549 0 None None None 2018-08-29 14:43:24 UTC

Description Miciah Dashiel Butler Masters 2018-08-10 23:06:14 UTC 
Cloned for 3.9.z backport.

+++ This bug was initially created as a clone of Bug #1600551 +++

Description of problem: Pods are experiencing intermittent DNS lookup failures when reaching out to dnsmasq. A similar upstream issue has been reported: https://github.com/kubernetes/kubernetes/issues/45976

[...]

--- Additional comment from Ryan Howe on 2018-08-09 10:28:47 EDT ---

Working another OpenShift dnsmasq issue we figured the issue to happen when dnsmasq uses a low port number. 

Setting min-port=1024 in dnsmasq worked around the issue. 

--min-port=<port>
              Do not use ports less than that given as source for outbound DNS queries. Dnsmasq picks random ports as source for outbound queries: when this option is given, the ports used will always to larger than that  specified. Useful for systems behind firewalls.


Dnsmasq bug was logged: 
  https://bugzilla.redhat.com/show_bug.cgi?id=1614331


I was not able to reproduce the issue again with this configuration in place.

--- Additional comment from Ryan Howe on 2018-08-09 11:11:26 EDT ---

Created PR to add this configuration via the ansible installer for OpenShift: 

https://github.com/openshift/openshift-ansible/pull/9505

[...]
Comment 1 Miciah Dashiel Butler Masters 2018-08-10 23:14:11 UTC 
OCP 3.9.z backport: https://github.com/openshift/openshift-ansible/pull/9541
Comment 3 Weibin Liang 2018-08-23 20:00:32 UTC 
Tested and verified in v3.9.41

[root@qe-weliang-3 ~]# oc version
oc v3.9.41
kubernetes v1.9.1+a0ce1bc657
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://qe-weliang-3.9-2-master-etcd-nfs-1:8443
openshift v3.9.41
kubernetes v1.9.1+a0ce1bc657
[root@qe-weliang-3 ~]# cat /etc/dnsmasq.d/origin-dns.conf | grep min
min-port=1024
[root@qe-weliang-3 ~]# cat /etc/NetworkManager/dispatcher.d/99-origin-dns.sh | grep mi
  # couldn't find an existing method to determine if the interface owns the
min-port=1024
[root@qe-weliang-3 ~]#
Comment 5 errata-xmlrpc 2018-08-29 14:42:32 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2549
Comment 6 Stephen Cuppett 2018-10-05 17:27:26 UTC 
*** Bug 1626248 has been marked as a duplicate of this bug. ***
Note You need to log in before you can comment on or make changes to this bug.