Bug 1898101

Hi,

Let me add some detail here:

We have a 226 worker node environment in the scale lab. I'm seeing very high mDNS traffic there as Yuval mentioned. Running a packet capture on the baremetal interface on a master node in this environment for about 16 seconds, this is what I see.


Out of 1067004 packets captured, 922979 are mDNS packets(86.5%).

The average packet length is 204.35 bytes, so this is quite a bit of traffic on an idle cluster.

This testing was with OCP 4.6.4.

One possible workaround for this is to provide records for the masters and workers in external DNS and then disable mdns-publisher using the following machine config:

---
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: ${ROLE}
  name: 99-${ROLE}-disable-mdns
spec:
  config:
    ignition:
      config: {}
      security:
        tls: {}
      timeouts: {}
      version: 3.1.0
    networkd: {}
    passwd: {}
    storage:
      files:
        - contents:
            source: data:,
          overwrite: true
          mode: 420
          path: /etc/kubernetes/manifests/mdns-publisher.yaml

This MC will overwrite the mdns-publisher static pod with an empty file so it will not run. Note that you will see logs about the empty static pod definition, but this should not cause any harm.

The longer term solution to this will be to move away from mdns for generating these records, but that's a significant change that will likely take a while to implement.

Ben, Antoni,

I'm running a 250 node cluster internally and am seeing the same issue. Is there any downsides to disabling mdns-publisher using Ben's method? My baremetal interfaces get DNS from an external DNS servers and that's about it.

(In reply to Sai Sindhur Malleni from comment #14)
> Ben, Antoni,
> 
> I'm running a 250 node cluster internally and am seeing the same issue. Is
> there any downsides to disabling mdns-publisher using Ben's method? My
> baremetal interfaces get DNS from an external DNS servers and that's about
> it.

If you disable mdns in the cluster then you will need to provide external records for all of the masters and workers. If you can automate that using the DHCP assignments it may not be too bad, but if you have to create 250 records by hand then that's a big downside. :-)

*** This bug has been marked as a duplicate of bug 1898097 ***

Applying the machineconfig as mentioned in Ben as mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1898101#c4 still does not fix the issue. We see heavy amounts of mdns traffic.

[root@worker232-r640 core]# cat /etc/kubernetes/manifests/mdns-publisher.yaml
[root@worker232-r640 core]#

I noticed that two out of my 250+ worker nodes didn't have this config applied

[kni@e16-h18-b03-fc640 ~]$ oc get pods -o wide -A | grep mdns                                                                                                                                                                                
openshift-kni-infra                                mdns-publisher-storage001-r640                                    1/1     Running     0          15h     192.168.221.10    storage001-r640   <none>           <none>                      
openshift-kni-infra                                mdns-publisher-storage002-r640                                    1/1     Running     0          15h     192.168.221.11    storage002-r640   <none>           <none>       


I fixed them as well and still see high mdns traffic.