Bug 1620230 - [3.7] Intermittent dnsmasq outages
Summary: [3.7] Intermittent dnsmasq outages
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 3.7.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.7.z
Assignee: Miciah Dashiel Butler Masters
QA Contact: zhaozhanqi
URL:
Whiteboard:
Depends On:
Blocks: 1600551
TreeView+ depends on / blocked
 
Reported: 2018-08-22 18:29 UTC by Miciah Dashiel Butler Masters
Modified: 2022-08-04 22:20 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: By default, older versions of dnsmasq can use privileged, lower-numbered source ports for outbound DNS queries. Consequence: Outbound DNS queries may be dropped; for example, firewall rules may drop queries coming from reserved ports. Fix: We now configure dnsmasq using its min-port setting to set the minimum port number for outbound queries to 1024. Result: DNS queries should no longer be dropped. Additional information: dnsmasq 2.79 changes the default min-port setting to 1024.
Clone Of: 1600551
Environment:
Last Closed: 2018-10-08 13:52:52 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift openshift-ansible pull 9539 0 None None None 2018-08-22 18:29:29 UTC

Description Miciah Dashiel Butler Masters 2018-08-22 18:29:29 UTC
Add the min-port=1024 setting to the dnsmasq configuration.

In the course of the ongoing effort to resolve bug 1600551, we fixed an issue related to the dnsmasq configuration.  Although bug 1600551 remains unresolved at this time and may have multiple underlying causes, we do want to verify and ship the change to the dnsmasq configuration.

+++ This bug was initially created as a clone of Bug #1600551 +++

Description of problem: Pods are experiencing intermittent DNS lookup failures when reaching out to dnsmasq. A similar upstream issue has been reported: https://github.com/kubernetes/kubernetes/issues/45976

[...]

--- Additional comment from Ryan Howe on 2018-08-09 10:28:47 EDT ---

Working another OpenShift dnsmasq issue we figured the issue to happen when dnsmasq uses a low port number. 

Setting min-port=1024 in dnsmasq worked around the issue. 

--min-port=<port>
              Do not use ports less than that given as source for outbound DNS queries. Dnsmasq picks random ports as source for outbound queries: when this option is given, the ports used will always to larger than that  specified. Useful for systems behind firewalls.


Dnsmasq bug was logged: 
  https://bugzilla.redhat.com/show_bug.cgi?id=1614331


I was not able to reproduce the issue again with this configuration in place.

--- Additional comment from Ryan Howe on 2018-08-09 11:11:26 EDT ---

Created PR to add this configuration via the ansible installer for OpenShift: 

https://github.com/openshift/openshift-ansible/pull/9505

[...]

Comment 1 Weibin Liang 2018-08-23 19:49:49 UTC
Tested and verified in v3.7.62

[root@ip-172-18-6-36 ~]# oc version
oc v3.7.62
kubernetes v1.7.6+a08f5eeb62
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://ip-172-18-6-36.ec2.internal:8443
openshift v3.7.62
kubernetes v1.7.6+a08f5eeb62
[root@ip-172-18-6-36 ~]# cat /etc/NetworkManager/dispatcher.d/99-origin-dns.sh | grep mi
  # couldn't find an existing method to determine if the interface owns the
min-port=1024
[root@ip-172-18-6-36 ~]# cat /etc/dnsmasq.d/origin-dns.conf | grep min
min-port=1024
[root@ip-172-18-6-36 ~]#


Note You need to log in before you can comment on or make changes to this bug.