Description of problem:

node-exporter triggers alarm on NodeHighNumberConntrackEntriesUsed.

We think it may be related to https://github.com/torvalds/linux/commit/e15d4cdf27cb0c1e977270270b2cea12e0955edd, basically host-to-host communication over UDP port 4789 (VXLAN traffic) is getting dropped somewhere in the network.

UDP is quite unreliable, and this is no problem, as retries are handled by higher TCP layer. However because of the bug this is stale entry and never cleaned up.

From https://github.com/torvalds/linux/commit/e15d4cdf27cb0c1e977270270b2cea12e0955edd:
~~~
netfilter: conntrack: do not renew entry stuck in tcp SYN_SENT state
Consider:
  client -----> conntrack ---> Host

client sends a SYN, but $Host is unreachable/silent.
Client eventually gives up and the conntrack entry will time out.

However, if the client is restarted with same addr/port pair, it
may prevent the conntrack entry from timing out.

This is noticeable when the existing conntrack entry has no NAT
transformation or an outdated one and port reuse happens either
on client or due to a NAT middlebox.

This change prevents refresh of the timeout for SYN retransmits,
so entry is going away after nf_conntrack_tcp_timeout_syn_sent
seconds (default: 60).

Entry will be re-created on next connection attempt, but then
nat rules will be evaluated again.
~~~


Version-Release number of selected component (if applicable):

4.6