Bug 1801737

Summary:	[4.3] kube-proxy periodic iptables reloads are extremely disruptive in large clusters
Product:	OpenShift Container Platform	Reporter:	Dan Winship <danw>
Component:	Networking	Assignee:	Juan Luis de Sousa-Valadas <jdesousa>
Networking sub component:	openshift-sdn	QA Contact:	zhaozhanqi <zzhao>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	urgent
Priority:	unspecified	Keywords:	Reopened
Version:	3.6.0
Target Milestone:	---
Target Release:	4.3.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1727441	Environment:
Last Closed:	2020-03-24 14:33:37 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1803149
Bug Blocks:	1801742

Comment 1 Juan Luis de Sousa-Valadas 2020-02-14 10:59:20 UTC

Closed duplicate of: https://bugzilla.redhat.com/show_bug.cgi?id=1797042
As soon as this merges then I'll start working on the 3.11-4.2 backports.

*** This bug has been marked as a duplicate of bug 1797042 ***

Comment 4 zhaozhanqi 2020-03-13 03:54:01 UTC

@Dan BTW, do we need to large cluster to verified this kind of bug?

Comment 5 Dan Winship 2020-03-13 13:26:00 UTC

I don't know how to create a cluster that precisely reproduces the customer environments, but you ought to be able to fake it...

Bring up a minimal cluster, and define 5 services, each with one pod, so there's something for kube-proxy to do. Then, one one node, run

    for i in $(seq 1 30000); do iptables -w -A INPUT -s 10.1.$(($i / 256).$(($i % 256)) -j DROP; done

to clog up iptables with dummy rules to slow things down. Then run a script that runs "iptables -C OUTPUT -j KUBE-FIREWALL" once a second, for 10 minutes. Usually that command should output with status 0 and no stdout (meaning "yes, that rule exists"), but occasionally it will fail with exit code 4 and an error "blah blah could not get xtables lock". Also, that same "could not get xtables lock" error will occasionally show up in the sdn pod logs.

With old 4.3, you should get a noticeable number of errors. With old 4.3 and a large iptablesSyncPeriod value configured you should get not quite as many errors. With new 4.3 (and the default iptablesSyncPeriod) you should get very few errors in the script, and ideally you shouldn't see any in the sdn pod log. (But if you do see it in the pod log, it should be followed by a message about "Sync failed; retrying in 30s".)

Comment 8 zhaozhanqi 2020-03-19 12:15:34 UTC

Move this bug to verified for now since this issue cannot be reproduce according to comment 6.

Comment 10 errata-xmlrpc 2020-03-24 14:33:37 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0858