Description of problem:

We have a customer suffering strange egressIP outages when vxlan_monitor performs the "egressVXLANMonitor" checks, *look at the timestamps* of the following checks (PFA full log):

~~~
I0601 16:02:34.045175   14443 vxlan_monitor.go:231] Node 192.168.52.153 may be offline... retrying
I0602 02:56:09.040928   14443 vxlan_monitor.go:231] Node 192.168.52.153 may be offline... retrying
W0602 05:26:09.042415   14443 vxlan_monitor.go:226] Node 192.168.52.153 is offline
I0602 05:26:14.040664   14443 vxlan_monitor.go:213] Node 192.168.52.153 is back online
I0602 05:31:39.039763   14443 vxlan_monitor.go:231] Node 192.168.52.153 may be offline... retrying
I0602 16:13:44.039254   14443 vxlan_monitor.go:231] Node 192.168.52.153 may be offline... retrying
W0603 03:56:34.039591   14443 vxlan_monitor.go:226] Node 192.168.52.153 is offline
I0603 03:56:39.039668   14443 vxlan_monitor.go:213] Node 192.168.52.153 is back online
~~~

As per source code, those retry checks should be performed within seconds, right?

Version-Release number of selected component (if applicable):

OCP v3.11.98

How reproducible:

As per customer comment:

--------------------------
EgressIP needs to be set up with a blocking egressnetworkpolicy. Then, from a pod using an egressIP try to send a packet to a blocked destination. This will end up in just increasing the outgoing packet counter, but will not receive any. Also, per the code, just the counters are checked again, and the check will still fail, and after maxRetries failures the egress router node will be treated as being down. I suspect this will remove routing towards that node. And after this procedure will the code start pinging the egress router at all, which will find that the node is back online. I think at the very first occurrence where we suspect the node might be down, we should start pinging it. That would avoid this behavior. Or to be simpler, a background process sending pings every second may be set up. That would make the code more simple.

This is a rare case, when the source node has light egress traffic, and a simple packet without response can trigger this.
--------------------------

Steps to Reproduce:
1.
2.
3.

Actual results:

Very unusual long delay between egressVXLANMonitor retries.

Expected results:

Maybe a node.retries counter reset as follows?

~~~
diff --git a/pkg/network/node/vxlan_monitor.go b/pkg/network/node/vxlan_monitor.go
index b7b1c8c2f1..37b329efcc 100644
--- a/pkg/network/node/vxlan_monitor.go
+++ b/pkg/network/node/vxlan_monitor.go
@@ -232,6 +232,8 @@ func (evm *egressVXLANMonitor) check(retryOnly bool) bool {
                                        retry = true
                                        continue
                                }
+                       } else {
+                               node.retries = 0
                        }
                }
~~~

Additional info:

https://github.com/openshift/origin/blob/master/pkg/network/node/vxlan_monitor.go