Bug 1801737 - [4.3] kube-proxy periodic iptables reloads are extremely disruptive in large clusters
Summary: [4.3] kube-proxy periodic iptables reloads are extremely disruptive in large ...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.3.z
Assignee: Juan Luis de Sousa-Valadas
QA Contact: zhaozhanqi
Depends On: 1803149
Blocks: 1801742
TreeView+ depends on / blocked
Reported: 2020-02-11 14:56 UTC by Dan Winship
Modified: 2020-04-03 12:53 UTC (History)
0 users

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1727441
Last Closed: 2020-03-24 14:33:37 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift sdn pull 39 0 None closed Bug 1801737: drop firewalld monitoring, add better iptables monitor 2021-01-26 05:04:23 UTC
Red Hat Product Errata RHBA-2020:0858 0 None None None 2020-03-24 14:34:03 UTC

Comment 1 Juan Luis de Sousa-Valadas 2020-02-14 10:59:20 UTC
Closed duplicate of: https://bugzilla.redhat.com/show_bug.cgi?id=1797042
As soon as this merges then I'll start working on the 3.11-4.2 backports.

*** This bug has been marked as a duplicate of bug 1797042 ***

Comment 4 zhaozhanqi 2020-03-13 03:54:01 UTC
@Dan BTW, do we need to large cluster to verified this kind of bug?

Comment 5 Dan Winship 2020-03-13 13:26:00 UTC
I don't know how to create a cluster that precisely reproduces the customer environments, but you ought to be able to fake it...

Bring up a minimal cluster, and define 5 services, each with one pod, so there's something for kube-proxy to do. Then, one one node, run

    for i in $(seq 1 30000); do iptables -w -A INPUT -s 10.1.$(($i / 256).$(($i % 256)) -j DROP; done

to clog up iptables with dummy rules to slow things down. Then run a script that runs "iptables -C OUTPUT -j KUBE-FIREWALL" once a second, for 10 minutes. Usually that command should output with status 0 and no stdout (meaning "yes, that rule exists"), but occasionally it will fail with exit code 4 and an error "blah blah could not get xtables lock". Also, that same "could not get xtables lock" error will occasionally show up in the sdn pod logs.

With old 4.3, you should get a noticeable number of errors. With old 4.3 and a large iptablesSyncPeriod value configured you should get not quite as many errors. With new 4.3 (and the default iptablesSyncPeriod) you should get very few errors in the script, and ideally you shouldn't see any in the sdn pod log. (But if you do see it in the pod log, it should be followed by a message about "Sync failed; retrying in 30s".)

Comment 8 zhaozhanqi 2020-03-19 12:15:34 UTC
Move this bug to verified for now since this issue cannot be reproduce according to comment 6.

Comment 10 errata-xmlrpc 2020-03-24 14:33:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.