Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1614983

Summary:	[3.9] Intermittent dnsmasq outages
Product:	OpenShift Container Platform	Reporter:	Miciah Dashiel Butler Masters <mmasters>
Component:	Networking	Assignee:	Miciah Dashiel Butler Masters <mmasters>
Networking sub component:	router	QA Contact:	zhaozhanqi <zzhao>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	unspecified
Priority:	unspecified	CC:	aos-bugs, bbennett, farandac, hongli, pdwyer, rbost, rfoyle, rhowe, stwalter, weliang, zzhao
Version:	3.9.0
Target Milestone:	---
Target Release:	3.9.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: By default, older versions of dnsmasq can use privileged, lower-numbered source ports for outbound DNS queries. Consequence: Outbound DNS queries may be dropped; for example, firewall rules may drop queries coming from reserved ports. Fix: We now configure dnsmasq using its min-port setting to set the minimum port number for outbound queries to 1024. Result: DNS queries should no longer be dropped. Additional information: dnsmasq 2.79 changes the default min-port setting to 1024.	Story Points:	---
Clone Of:	1600551	Environment:
Last Closed:	2018-08-29 14:42:32 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1600551
Bug Blocks:

Description Miciah Dashiel Butler Masters 2018-08-10 23:06:14 UTC

Cloned for 3.9.z backport.

+++ This bug was initially created as a clone of Bug #1600551 +++

Description of problem: Pods are experiencing intermittent DNS lookup failures when reaching out to dnsmasq. A similar upstream issue has been reported: https://github.com/kubernetes/kubernetes/issues/45976

[...]

--- Additional comment from Ryan Howe on 2018-08-09 10:28:47 EDT ---

Working another OpenShift dnsmasq issue we figured the issue to happen when dnsmasq uses a low port number. 

Setting min-port=1024 in dnsmasq worked around the issue. 

--min-port=<port>
              Do not use ports less than that given as source for outbound DNS queries. Dnsmasq picks random ports as source for outbound queries: when this option is given, the ports used will always to larger than that  specified. Useful for systems behind firewalls.


Dnsmasq bug was logged: 
  https://bugzilla.redhat.com/show_bug.cgi?id=1614331


I was not able to reproduce the issue again with this configuration in place.

--- Additional comment from Ryan Howe on 2018-08-09 11:11:26 EDT ---

Created PR to add this configuration via the ansible installer for OpenShift: 

https://github.com/openshift/openshift-ansible/pull/9505

[...]

Comment 1 Miciah Dashiel Butler Masters 2018-08-10 23:14:11 UTC

OCP 3.9.z backport: https://github.com/openshift/openshift-ansible/pull/9541

Comment 3 Weibin Liang 2018-08-23 20:00:32 UTC

Tested and verified in v3.9.41

[root@qe-weliang-3 ~]# oc version
oc v3.9.41
kubernetes v1.9.1+a0ce1bc657
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://qe-weliang-3.9-2-master-etcd-nfs-1:8443
openshift v3.9.41
kubernetes v1.9.1+a0ce1bc657
[root@qe-weliang-3 ~]# cat /etc/dnsmasq.d/origin-dns.conf | grep min
min-port=1024
[root@qe-weliang-3 ~]# cat /etc/NetworkManager/dispatcher.d/99-origin-dns.sh | grep mi
  # couldn't find an existing method to determine if the interface owns the
min-port=1024
[root@qe-weliang-3 ~]#

Comment 5 errata-xmlrpc 2018-08-29 14:42:32 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2549

Comment 6 Stephen Cuppett 2018-10-05 17:27:26 UTC

*** Bug 1626248 has been marked as a duplicate of this bug. ***