Bug 1489581

Scott, not sure if you meant to assign this to Pod component, but seems that this is either Networking or Installer related.  I would guess Installer (responsible for dnsmasq configuration) but can you move this to one of those two components?

Seth,

It'd be whichever component is responsible for running the dns service embedded in the node. Networking seems fine.

I need someone to help me figure out whether the DNS spec allows a forwarding resolver to fail a PTR request - if so we can fix this by carrying a patch on skydns that allows us to reject (SERVFAIL) the PTR.  However, may also need some help determining whether SERVFAIL is allowed for PTR responses.

Assuming the cluster is actually using private address ranges, then RFC 6761 "Special-Use Domain Names" lets us return any answer we want here:

   6.1.  Domain Name Reservation Considerations for Private Addresses

   The private-address [RFC1918] reverse-mapping domains listed below,
   and any names falling within those domains, are Special-Use Domain
   Names:

     10.in-addr.arpa.      21.172.in-addr.arpa.  26.172.in-addr.arpa.
     16.172.in-addr.arpa.  22.172.in-addr.arpa.  27.172.in-addr.arpa.
     17.172.in-addr.arpa.  30.172.in-addr.arpa.  28.172.in-addr.arpa.
     ...

   These domains, and any names falling within these domains, are
   special in the following ways:

   ....

   4.  Caching DNS servers SHOULD recognize these names as special and
       SHOULD NOT, by default, attempt to look up NS records for them,
       or otherwise query authoritative DNS servers in an attempt to
       resolve these names.  Instead, caching DNS servers SHOULD, by
       default, generate immediate (positive or negative) responses for
       all such queries.  This is to avoid unnecessary load on the root
       name servers and other name servers.  Caching DNS servers SHOULD
       offer a configuration option (disabled by default) to enable
       upstream resolution of such names, for use in private networks
       where private-address reverse-mapping names are known to be
       handled by an authoritative DNS server in said private network.



Even in the non-private-IP-range case, a non-authoritative resolver is still always allowed to return any error except NXDOMAIN (and all of the error codes apply to all query types). So yes, you could return SERVFAIL, although REFUSED would be better; SERVFAIL ("The name server was unable to process this query due to a problem with the name server") is sometimes treated as a temporary/retryable error (eg, getnameinfo() returns EAI_AGAIN and net.LookupAddr() returns an error with .IsTemporary == true). REFUSED ("The name server refuses to perform the specified operation for policy reasons") generally gets treated by clients the same as "not found".

Is there any trello card, git hub, or work on this that can be tracked?

There was discussion on this previously and then it was put on "UpcomingRelease" pointing to 3.7, then 3.8, and now 3.9.

With 3.9 node-dnsmasq.conf has:
--
 server=/in-addr.arpa/127.0.0.1
 server=/cluster.local/127.0.0.1
--

So it sends all reverse lookups to our local resolver.  Our resolver now returns useful answers for services:
--
 $ nslookup 172.30.0.1 172.30.0.1
 Server:		172.30.0.1
 Address:	172.30.0.1#53

 Non-authoritative answer:
 1.0.30.172.in-addr.arpa	name = kubernetes.default.svc.cluster.local.

 Authoritative answers can be found from:
--

But for a pod IP address it returns SERVFAIL which makes dnsmasq try the upstream servers.  However, that puts us at the mercy of the upstream resolver to be quick enough, and if we get a resonse, we will definitely get an incorrect name.

We could hack this by adding a line to the node-dnsmasq.conf in ansible that tells dnsmasq to handle the appropriate set of in-addr.arpa itself.  That would look something like:
  local=/130.10.in-addr.arpa/
(see http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html)

The issue there is that it only works when the SDN CIDR is 8-bit aligned.  So if you have a SDN CIDR that is /15 we will reply for more than we should :-(

@Eric: What problem is the customer still experiencing?  The service CIDRs are being resolved quickly.  Is the upstream resolver being slow when resolving pod IPs?

When I tested 1000 resolutions, it took ~6 seconds regardless of whether I was testing a pod or service address against our resolver directly, or whether I was using a good address against the upstream resolver.