Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1620230

Summary: [3.7] Intermittent dnsmasq outages
Product: OpenShift Container Platform Reporter: Miciah Dashiel Butler Masters <mmasters>
Component: NetworkingAssignee: Miciah Dashiel Butler Masters <mmasters>
Networking sub component: router QA Contact: zhaozhanqi <zzhao>
Status: CLOSED CURRENTRELEASE Docs Contact:
Severity: high    
Priority: high CC: aos-bugs, bbennett, dmace, farandac, hongli, pdwyer, rbost, rfoyle, rhowe, vlaad, weliang, zzhao
Version: 3.7.0   
Target Milestone: ---   
Target Release: 3.7.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: By default, older versions of dnsmasq can use privileged, lower-numbered source ports for outbound DNS queries. Consequence: Outbound DNS queries may be dropped; for example, firewall rules may drop queries coming from reserved ports. Fix: We now configure dnsmasq using its min-port setting to set the minimum port number for outbound queries to 1024. Result: DNS queries should no longer be dropped. Additional information: dnsmasq 2.79 changes the default min-port setting to 1024.
 Story Points: ---
Clone Of: 1600551 Environment:
Last Closed: 2018-10-08 13:52:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1600551    

Description Miciah Dashiel Butler Masters 2018-08-22 18:29:29 UTC 
Add the min-port=1024 setting to the dnsmasq configuration.

In the course of the ongoing effort to resolve bug 1600551, we fixed an issue related to the dnsmasq configuration.  Although bug 1600551 remains unresolved at this time and may have multiple underlying causes, we do want to verify and ship the change to the dnsmasq configuration.

+++ This bug was initially created as a clone of Bug #1600551 +++

Description of problem: Pods are experiencing intermittent DNS lookup failures when reaching out to dnsmasq. A similar upstream issue has been reported: https://github.com/kubernetes/kubernetes/issues/45976

[...]

--- Additional comment from Ryan Howe on 2018-08-09 10:28:47 EDT ---

Working another OpenShift dnsmasq issue we figured the issue to happen when dnsmasq uses a low port number. 

Setting min-port=1024 in dnsmasq worked around the issue. 

--min-port=<port>
              Do not use ports less than that given as source for outbound DNS queries. Dnsmasq picks random ports as source for outbound queries: when this option is given, the ports used will always to larger than that  specified. Useful for systems behind firewalls.


Dnsmasq bug was logged: 
  https://bugzilla.redhat.com/show_bug.cgi?id=1614331


I was not able to reproduce the issue again with this configuration in place.

--- Additional comment from Ryan Howe on 2018-08-09 11:11:26 EDT ---

Created PR to add this configuration via the ansible installer for OpenShift: 

https://github.com/openshift/openshift-ansible/pull/9505

[...]
Comment 1 Weibin Liang 2018-08-23 19:49:49 UTC 
Tested and verified in v3.7.62

[root@ip-172-18-6-36 ~]# oc version
oc v3.7.62
kubernetes v1.7.6+a08f5eeb62
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://ip-172-18-6-36.ec2.internal:8443
openshift v3.7.62
kubernetes v1.7.6+a08f5eeb62
[root@ip-172-18-6-36 ~]# cat /etc/NetworkManager/dispatcher.d/99-origin-dns.sh | grep mi
  # couldn't find an existing method to determine if the interface owns the
min-port=1024
[root@ip-172-18-6-36 ~]# cat /etc/dnsmasq.d/origin-dns.conf | grep min
min-port=1024
[root@ip-172-18-6-36 ~]#