Bug 1705698

Summary:	OpenShift nodes are heavily querying DNS servers causing DNS lookup timeouts within pods
Product:	OpenShift Container Platform	Reporter:	Sam Yangsao <syangsao>
Component:	Networking	Assignee:	Dan Mace <dmace>
Networking sub component:	DNS	QA Contact:	Hongan Li <hongli>
Status:	CLOSED INSUFFICIENT_DATA	Docs Contact:
Severity:	high
Priority:	unspecified	CC:	aos-bugs, bbennett, dmace
Version:	3.11.0
Target Milestone:	---
Target Release:	3.11.z
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-08-09 12:57:17 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Sam Yangsao 2019-05-02 19:09:01 UTC

Description of problem:

OpenShift nodes are heavily querying DNS servers causing timeouts, are there ways to tune local dnsmasq to cache DNS entries?

Version-Release number of selected component (if applicable):

3.11

How reproducible:

Always

Steps to Reproduce:
1.
2.
3.

Actual results:

Customer has 1000s of OpenShift pods and 100s of nodes across multiple clusters.  For example, the master node would query their DCustomer has configured nscd and this has helped cache those DNS entries.  

NS server 12884 times in that 16 minutes.  This coupled with many other masters/nodes/pods, drags down the customers DNS servers and applications that lookup DNS with most of these lookups going to static addresses.

Expected results:

OpenShift should not continuously query DNS constantly.  There should be some alleviation of DNS queries by caching them locally to the OpenShift node either via dnsmasq or nscd.

Additional info:

Customer has configured nscd and this has helped cache those DNS lookups from hitting their internal DNS servers.  

1.  Do we support enabling nscd on OpenShift nodes?
2.  Is there a way to tune dnsmasq on OpenShift to accept the caching of DNS entries local to the node itself to alleviate hitting external DNS nameservers?

Comment 1 Sam Yangsao 2019-05-23 17:47:01 UTC

Hey Dan,

Customer was asking about this today in our call.  Do we have any other options with dnsmasq locally to cache the lookup entries versus relying on nscd being installed to handle this on the local node?

Thanks!

Comment 4 Sam Yangsao 2019-08-09 12:57:17 UTC

Closing this as we have not heard back from the customer