Bug 2068303

* Bare metal IPI installation
* Each node has 2 NICs connected to the external/baremetal network - enp2s0 and enp3s0
* Node IPs assigned to enp2s0 on each node
* MetalLB deployed in layer 2 mode

MetalLB load balancer works when first created.  Can ping load balancer IP, connect to service, etc.  Some time later, MetalLB IP stops working; can't ping or connect.  Disconnecting enp3s0 from the network (link down) restores functionality.

Some details:

* Environment is KVM VMs (using vBMC for IPI).  Virtual network is a Linux bridge.

* Load balancer IP is 192.168.123.23

* Load balancer IP is assigned to ocp4-worker3
  - enp2s0 (52:54:00:00:00:06) w/ node IP 192.168.123.106
  - enp3s0 (52:54:00:00:01:06) "unused" (but connected) interface

* Provisioning node has IP 192.168.123.100 & MAC 52:54:00:00:00:00

After deleting the ARP entry for 192.168.123.23 and attempting to ping that IP from the provisioner, tcpdump (on the hypervisor) shows:

13:18:31.132696 52:54:00:00:00:00 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.123.23 tell 192.168.123.100, length 28
13:18:31.132962 52:54:00:00:00:06 > 52:54:00:00:00:00, ethertype ARP (0x0806), length 60: Reply 192.168.123.23 is-at 52:54:00:00:00:06, length 46
13:18:31.133089 52:54:00:00:00:00 > 52:54:00:00:00:06, ethertype IPv4 (0x0800), length 98: 192.168.123.100 > 192.168.123.23: ICMP echo request, id 11, seq 1, length 64
13:18:31.133092 52:54:00:00:01:06 > 52:54:00:00:00:00, ethertype ARP (0x0806), length 60: Reply 192.168.123.23 is-at 52:54:00:00:01:06, length 46
13:18:31.133348 52:54:00:00:00:06 > 52:54:00:00:01:06, ethertype IPv4 (0x0800), length 98: 192.168.123.100 > 192.168.123.23: ICMP echo request, id 11, seq 1, length 64
13:18:36.436691 52:54:00:00:00:06 > 52:54:00:00:01:06, ethertype ARP (0x0806), length 42: Request who-has 192.168.123.23 tell 192.168.123.106, length 28
13:18:36.438117 52:54:00:00:01:06 > 52:54:00:00:00:06, ethertype ARP (0x0806), length 60: Reply 192.168.123.23 is-at 52:54:00:00:01:06, length 46

This shows that an ARP response is being sent from both enp2s0 and enp3s0, with each containing the MAC of the interface from which it was sent.  It also shows (oddly?) that echo requests are actually sent to *both* MACs, but no echo reply is sent.  Finally, the last two lines actually show an ARP request and reply between the 2 interfaces in the worker node.

It's probably also worth noting that a static ARP entry does not fix the issue.  Here is the tcpdump output (on the hypervisor) when I attempt the ping after creating a static ARP entry (192.168.123.23 --> 52:54:00:00:00:06):

17:08:58.835651 52:54:00:00:00:00 > 52:54:00:00:00:06, ethertype IPv4 (0x0800), length 98: 192.168.123.100 > 192.168.123.23: ICMP echo request, id 18, seq 1, length 64
17:08:58.835863 52:54:00:00:00:06 > 52:54:00:00:01:06, ethertype IPv4 (0x0800), length 98: 192.168.123.100 > 192.168.123.23: ICMP echo request, id 18, seq 1, length 64
17:09:04.020764 52:54:00:00:00:06 > 52:54:00:00:01:06, ethertype ARP (0x0806), length 42: Request who-has 192.168.123.23 tell 192.168.123.106, length 28
17:09:04.021144 52:54:00:00:01:06 > 52:54:00:00:00:06, ethertype ARP (0x0806), length 60: Reply 192.168.123.23 is-at 52:54:00:00:01:06, length 46

I will attach logs from the MetalLB controller pod and the speaker pod on the worker to which the LB IP is assigned (ocp4-worker3).