Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1937949

Summary:	[4.4.29] Intermittent DNS resolution failures / hostname lookup delays from DNS resolvers
Product:	OpenShift Container Platform	Reporter:	Caden Marchese <cmarches>
Component:	Networking	Assignee:	Miciah Dashiel Butler Masters <mmasters>
Networking sub component:	DNS	QA Contact:	Hongan Li <hongli>
Status:	CLOSED DUPLICATE	Docs Contact:
Severity:	urgent
Priority:	urgent	CC:	aamarks, aconole, adeshpan, alchan, amcdermo, anbhat, aos-bugs, aos-network-edge-staff, bbennett, bjarolim, dporter, dsquirre, emahoney, fpan, jmaxwell, ktenzer, misalunk, mmasters, openshift-bugs-escalate, palonsor, psutter, rsandu, rupatel, scuppett, sgreene, skanakal, tmanor, vwalek
Version:	4.4	Flags:	scuppett: needinfo- scuppett: needinfo- scuppett: needinfo- scuppett: needinfo- skanakal: needinfo-
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-03-22 14:51:39 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Comment 90 Phil Sutter 2021-03-24 13:06:48 UTC

From what I remember from the last escalation back in September, the problem is
this:

* HAProxy does TCP-based health checks to a large number of backends ever 5s
* The iptables ruleset is huge and there's no shortcut for packets of a TCP
  handshake
* Excessive rule processing at health check time adds significant delay (maybe
  also buffer exhaustion/packet drops) to TCP handshake run-time
* Either due to timeout or packet drop, backends start retransmitting SYN-ACKs
  which makes things worse and netfilter packet processing is constantly at max
  CPU (high ksoftirq)
* The situation probably lasts long enough until HAProxy tests the unresponsive
  backends again, making it permanent.
* Obviously backends transition to offline state.

The obvious question is, why is netfilter throughput so bad to cause this and
there are a few aspects we have to consider:

* Customer might have changed the traffic/usage pattern, e.g. adding more nodes
  or whatever.
* OCP was updated from major 3 to 4 and therefore:
  * Base OS went from RHEL7 to RHEL8
  * iptables is actually iptables-nft now
  * OpenShift has changed (a black box from my perspective)

Customer claimed they actually reduced the load via the update. IIRC, they
claimed 27k routes were OK and now 11k cause trouble.

What we know is:

* Iptables-nft ruleset performance is worse than legacy iptables one and the
  ruleset OCP creates is a bad case (many rules with mostly address matches). I
  was able to improve address and other byte-boundary prefix matches (/8, /16,
  /24) by 100% in ideal cases. This is available in RHEL8.2.z[1]. The customer
  uses a hotfix version which improves address matches only (/32).

* A bug in openshift-sdn caused each packet to traverse the second-largest
  chain (KUBE-SERVICES in filter table) twice[2]. I don't know which OCP
  version is affected or not, neither do I know if the fix has been backported
  already.

* I pointed[3] at a minor reorg of nat table's KUBE-SERVICES chain which halves
  its size. This is WiP upstream[4] at an admittedly pretty slow pace.

Cheers, Phil

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1894620
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1879607#c8
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1879607#c6
[4] https://github.com/kubernetes/kubernetes/pull/96959

Comment 96 Red Hat Bugzilla 2023-09-15 01:33:14 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days