Description of problem:
Delay in IPtables sync rules on the worker nodes 

Version-Release number of selected component (if applicable):
OCP 4.7.19


How reproducible:
I have tried to reproduce this issue in my lab environment but its not reproducible, In lab environment.


Steps to Reproduce:
1.
2.
3.

Actual results:
Customer is using OCP 4.7, whenever customer restart a pod, It take up to 10 mins to reach that endpoint using service IP, while checking the sdn pod logs, we found below messages.

2021-09-22T21:39:25.292283539Z I0922 21:39:25.292214    2766 proxier.go:871] Syncing iptables rules
2021-09-22T21:39:29.310260092Z I0922 21:39:29.310012    2766 trace.go:205] Trace[1178748399]: "iptables save" (22-Sep-2021 21:39:25.438) (total time: 3871ms):
2021-09-22T21:39:32.368260083Z I0922 21:39:32.368095    2766 trace.go:205] Trace[1113058674]: "iptables restore" (22-Sep-2021 21:39:29.355) (total time: 3012ms):
2021-09-22T21:39:32.368260083Z I0922 21:39:32.368174    2766 proxier.go:826] syncProxyRules took 7.07847288s
2021-09-22T21:47:36.286802199Z I0922 21:47:36.286755    2766 proxier.go:871] Syncing iptables rules
2021-09-22T21:47:40.280708718Z I0922 21:47:40.280623    2766 trace.go:205] Trace[1833141062]: "iptables save" (22-Sep-2021 21:47:36.444) (total time: 3836ms):
2021-09-22T21:47:43.250445907Z I0922 21:47:43.250352    2766 trace.go:205] Trace[2113592216]: "iptables restore" (22-Sep-2021 21:47:40.359) (total time: 2891ms):
2021-09-22T21:47:43.250511955Z I0922 21:47:43.250452    2766 proxier.go:826] syncProxyRules took 6.967296745s
2021-09-22T21:55:51.034810403Z I0922 21:55:51.034740    2766 proxier.go:871] Syncing iptables rules
2021-09-22T21:55:55.018615376Z I0922 21:55:55.018510    2766 trace.go:205] Trace[1757753544]: "iptables save" (22-Sep-2021 21:55:51.208) (total time: 3810ms):
2021-09-22T21:55:58.086542163Z I0922 21:55:58.086163    2766 trace.go:205] Trace[863049554]: "iptables restore" (22-Sep-2021 21:55:55.087) (total time: 2998ms):
2021-09-22T21:55:58.086542163Z I0922 21:55:58.086248    2766 proxier.go:826] syncProxyRules took 7.053989394s
2021-09-22T22:04:01.197413777Z I0922 22:04:01.197202    2766 proxier.go:871] Syncing iptables rules
2021-09-22T22:04:04.529682876Z I0922 22:04:04.529615    2766 trace.go:205] Trace[1645870906]: "iptables save" (22-Sep-2021 22:04:01.326) (total time: 3203ms):
2021-09-22T22:04:07.517306482Z I0922 22:04:07.517190    2766 trace.go:205] Trace[1902657597]: "iptables restore" (22-Sep-2021 22:04:04.584) (total time: 2932ms):
2021-09-22T22:04:07.517306482Z I0922 22:04:07.517264    2766 proxier.go:826] syncProxyRules took 6.322735506s
2021-09-22T22:12:15.424965620Z I0922 22:12:15.424710    2766 proxier.go:871] Syncing iptables rules
2021-09-22T22:12:19.502690440Z I0922 22:12:19.502611    2766 trace.go:205] Trace[559611378]: "iptables save" (22-Sep-2021 22:12:15.572) (total time: 3929ms):
2021-09-22T22:12:23.745428697Z I0922 22:12:23.745321    2766 trace.go:205] Trace[939371679]: "iptables restore" (22-Sep-2021 22:12:19.546) (total time: 4198ms):
2021-09-22T22:12:23.745488332Z I0922 22:12:23.745420    2766 proxier.go:826] syncProxyRules took 8.323349137s



Expected results:

It should not take this amount of time post pod restart to sync iptables rules, we have tested this in lab environment.


Additional info:

We have increased the log level for some of the sdn pods to check this issue.But after restarting the sdn pods this issue get resolved.

I am attaching the mustgather before and after restarting the sdn pods to check this issue.