Bug 2008247 - Delay in IPtables sync rules on the worker nodes
Summary: Delay in IPtables sync rules on the worker nodes
Keywords:
Status: CLOSED DUPLICATE of bug 2002291
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.7
Hardware: All
OS: Linux
unspecified
high
Target Milestone: ---
: ---
Assignee: Mohamed Mahmoud
QA Contact: zhaozhanqi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-09-27 17:09 UTC by Ashish Sharma
Modified: 2021-10-07 15:14 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-10-07 15:14:43 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Ashish Sharma 2021-09-27 17:09:18 UTC
Description of problem:
Delay in IPtables sync rules on the worker nodes 

Version-Release number of selected component (if applicable):
OCP 4.7.19


How reproducible:
I have tried to reproduce this issue in my lab environment but its not reproducible, In lab environment.


Steps to Reproduce:
1.
2.
3.

Actual results:
Customer is using OCP 4.7, whenever customer restart a pod, It take up to 10 mins to reach that endpoint using service IP, while checking the sdn pod logs, we found below messages.

2021-09-22T21:39:25.292283539Z I0922 21:39:25.292214    2766 proxier.go:871] Syncing iptables rules
2021-09-22T21:39:29.310260092Z I0922 21:39:29.310012    2766 trace.go:205] Trace[1178748399]: "iptables save" (22-Sep-2021 21:39:25.438) (total time: 3871ms):
2021-09-22T21:39:32.368260083Z I0922 21:39:32.368095    2766 trace.go:205] Trace[1113058674]: "iptables restore" (22-Sep-2021 21:39:29.355) (total time: 3012ms):
2021-09-22T21:39:32.368260083Z I0922 21:39:32.368174    2766 proxier.go:826] syncProxyRules took 7.07847288s
2021-09-22T21:47:36.286802199Z I0922 21:47:36.286755    2766 proxier.go:871] Syncing iptables rules
2021-09-22T21:47:40.280708718Z I0922 21:47:40.280623    2766 trace.go:205] Trace[1833141062]: "iptables save" (22-Sep-2021 21:47:36.444) (total time: 3836ms):
2021-09-22T21:47:43.250445907Z I0922 21:47:43.250352    2766 trace.go:205] Trace[2113592216]: "iptables restore" (22-Sep-2021 21:47:40.359) (total time: 2891ms):
2021-09-22T21:47:43.250511955Z I0922 21:47:43.250452    2766 proxier.go:826] syncProxyRules took 6.967296745s
2021-09-22T21:55:51.034810403Z I0922 21:55:51.034740    2766 proxier.go:871] Syncing iptables rules
2021-09-22T21:55:55.018615376Z I0922 21:55:55.018510    2766 trace.go:205] Trace[1757753544]: "iptables save" (22-Sep-2021 21:55:51.208) (total time: 3810ms):
2021-09-22T21:55:58.086542163Z I0922 21:55:58.086163    2766 trace.go:205] Trace[863049554]: "iptables restore" (22-Sep-2021 21:55:55.087) (total time: 2998ms):
2021-09-22T21:55:58.086542163Z I0922 21:55:58.086248    2766 proxier.go:826] syncProxyRules took 7.053989394s
2021-09-22T22:04:01.197413777Z I0922 22:04:01.197202    2766 proxier.go:871] Syncing iptables rules
2021-09-22T22:04:04.529682876Z I0922 22:04:04.529615    2766 trace.go:205] Trace[1645870906]: "iptables save" (22-Sep-2021 22:04:01.326) (total time: 3203ms):
2021-09-22T22:04:07.517306482Z I0922 22:04:07.517190    2766 trace.go:205] Trace[1902657597]: "iptables restore" (22-Sep-2021 22:04:04.584) (total time: 2932ms):
2021-09-22T22:04:07.517306482Z I0922 22:04:07.517264    2766 proxier.go:826] syncProxyRules took 6.322735506s
2021-09-22T22:12:15.424965620Z I0922 22:12:15.424710    2766 proxier.go:871] Syncing iptables rules
2021-09-22T22:12:19.502690440Z I0922 22:12:19.502611    2766 trace.go:205] Trace[559611378]: "iptables save" (22-Sep-2021 22:12:15.572) (total time: 3929ms):
2021-09-22T22:12:23.745428697Z I0922 22:12:23.745321    2766 trace.go:205] Trace[939371679]: "iptables restore" (22-Sep-2021 22:12:19.546) (total time: 4198ms):
2021-09-22T22:12:23.745488332Z I0922 22:12:23.745420    2766 proxier.go:826] syncProxyRules took 8.323349137s



Expected results:

It should not take this amount of time post pod restart to sync iptables rules, we have tested this in lab environment.


Additional info:

We have increased the log level for some of the sdn pods to check this issue.But after restarting the sdn pods this issue get resolved.

I am attaching the mustgather before and after restarting the sdn pods to check this issue.

Comment 17 Mohamed Mahmoud 2021-10-07 13:58:54 UTC
based on iptables dump this issue is most likely dup of https://bugzilla.redhat.com/show_bug.cgi?id=2002291 which is fixed in 4.7.z release

Comment 21 Mohamed Mahmoud 2021-10-07 15:14:43 UTC

*** This bug has been marked as a duplicate of bug 2002291 ***


Note You need to log in before you can comment on or make changes to this bug.