Bug 2005733

Summary: conntrack [UNREPLIED] state for UDP 4789
Product: OpenShift Container Platform Reporter: Robin Cernin <rcernin>
Component: NetworkingAssignee: Jacob Tanenbaum <jtanenba>
Networking sub component: openshift-sdn QA Contact: zhaozhanqi <zzhao>
Status: CLOSED DUPLICATE Docs Contact:
Severity: high    
Priority: unspecified CC: bbennett, vpickard
Version: 4.6.z   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-09-22 17:33:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Robin Cernin 2021-09-20 00:05:53 UTC 
Description of problem:

node-exporter triggers alarm on NodeHighNumberConntrackEntriesUsed.

We think it may be related to https://github.com/torvalds/linux/commit/e15d4cdf27cb0c1e977270270b2cea12e0955edd, basically host-to-host communication over UDP port 4789 (VXLAN traffic) is getting dropped somewhere in the network.

UDP is quite unreliable, and this is no problem, as retries are handled by higher TCP layer. However because of the bug this is stale entry and never cleaned up.

From https://github.com/torvalds/linux/commit/e15d4cdf27cb0c1e977270270b2cea12e0955edd:
~~~
netfilter: conntrack: do not renew entry stuck in tcp SYN_SENT state
Consider:
  client -----> conntrack ---> Host

client sends a SYN, but $Host is unreachable/silent.
Client eventually gives up and the conntrack entry will time out.

However, if the client is restarted with same addr/port pair, it
may prevent the conntrack entry from timing out.

This is noticeable when the existing conntrack entry has no NAT
transformation or an outdated one and port reuse happens either
on client or due to a NAT middlebox.

This change prevents refresh of the timeout for SYN retransmits,
so entry is going away after nf_conntrack_tcp_timeout_syn_sent
seconds (default: 60).

Entry will be re-created on next connection attempt, but then
nat rules will be evaluated again.
~~~


Version-Release number of selected component (if applicable):

4.6