The basic issue here is that there by nature exists a race condition between when the cluster is able to re-assign the service endpoint and when the migrating VM switches its seat of identity (I just coined that phrase for this conversation) between the source and destination node. I don't know if there's anything we could do to try and coordinate these two events better, but I'm re-assigning the component to Networking as my guess is they might have better insight into the matter. Please feel free to re-assign if this isn't the best move--this one really is a gray area in terms of ownership.

(In reply to sgott from comment #3)
> The basic issue here is that there by nature exists a race condition between
> when the cluster is able to re-assign the service endpoint and when the
> migrating VM switches its seat of identity (I just coined that phrase for
> this conversation) between the source and destination node.

To clarify, this is when using Linux bridge where Kubernetes Service is not involved.

This may be an issue caused by us having a NIC in the destination virt-launcher with a MAC identical to the NIC of the source guest VM for a brief moment after the virt-launcher is started, but before virt-handler reassigns the MAC.

The team will look into this. Thanks for providing such a detailed analysis of what's happening!

(In reply to Petr Horáček from comment #5)
> This may be an issue caused by us having a NIC in the destination
> virt-launcher with a MAC identical to the NIC of the source guest VM for a
> brief moment after the virt-launcher is started, but before virt-handler
> reassigns the MAC.

Yup, it is caused by creating the network plumbing on the destination host, the linux kernel automatically sends an IPv6 ND NS/NA on device creation.
1

This goes out on pod creation (not just virt-launchers, any pod does this)
00:59:17.488074 02:3b:54:00:00:01 > 33:33:00:00:00:02, ethertype IPv6 (0x86dd), length 70: fe80::3b:54ff:fe00:1 > ff02::2: ICMP6, router solicitation, length 16

Then all switches update their tables with the "new" 02:3b:54:00:00:01 location.

That is not a problem for most pods, but unfortunate for VM live migration. A frame to update the network should only go out when the VM runs on destination. This is supposed to be done by the qemu GARP/RARP, but it only goes out much later, during announce_self() when the VM really runs on the destination.

00:59:19.451322 02:3b:54:00:00:01 > ff:ff:ff:ff:ff:ff, ethertype Reverse ARP (0x8035), length 60: Reverse Request who-is 02:3b:54:00:00:01 tell 02:3b:54:00:00:01, length 46

However, the network was already updated by the IPv6 ND earlier, and traffic redirected.

Can reproduce the behaviour of the pod going up by doing this:

# ip link add address 00:11:22:33:44:55 name test0 type veth peer name test1 address 00:11:22:33:44:56
# ip link set dev test0 master virt.home.arpa
# ip link set dev test0 up 
# ip link set dev test1 up 

Here goes the ND NS out which update the switches :(

01:25:55.150023 00:11:22:33:44:56 > 33:33:ff:33:44:56, ethertype IPv6 (0x86dd), length 86: :: > ff02::1:ff33:4456: ICMP6, neighbor solicitation, who has fe80::211:22ff:fe33:4456, length 32

You probably need to have multus/CNI set NOARP on the interfaces when creating the plumbing:

# ip link add address 00:11:22:33:44:55 name test0 type veth peer name test1 address 00:11:22:33:44:56
# ip link set dev test0 arp off
# ip link set dev test1 arp off
# ip link set dev test1 up 
# ip link set dev test0 up 

<NO IPv6 ND is seen>

And after this the update to the network will be done by qemu, when the VM runs using virtio announce.

Thank you Germano for the detail investigation.

We had an initial discussion to evaluated this issue.
Here are the main ideas raised:
- Send IPv6 ND NS/NA frequently during migration from the source virt-launcher.
  - A workaround before a formal version, can be tried to do using a sidecar.
- Use the tuning CNI meta plugin to set the NOARP option for "all" and "default" interfaces.
  It needs to be added to the plugin chain as the top plugin in the NetworkAttachementDefinition.
- Change the specific CNI plugin (e.g. the bridge CNI) to accept the initial link state or the NOARP option.

Action Items:
- Test the tuning CNI meta plugin and see if it behaves as expected.

(In reply to Edward Haas from comment #8)
> Thank you Germano for the detail investigation.
My pleasure.
> 
> We had an initial discussion to evaluated this issue.
> Here are the main ideas raised:
> - Send IPv6 ND NS/NA frequently during migration from the source
> virt-launcher.

Or a GARP, so its more clear about the intentions in case someone is looking at a tcpdump and wondering what is that additional IPv6 ND spam coming from 2 different places now :)

>   - A workaround before a formal version, can be tried to do using a sidecar.
> - Use the tuning CNI meta plugin to set the NOARP option for "all" and
> "default" interfaces.
>   It needs to be added to the plugin chain as the top plugin in the
> NetworkAttachementDefinition.
> - Change the specific CNI plugin (e.g. the bridge CNI) to accept the initial
> link state or the NOARP option.
> 
> Action Items:
> - Test the tuning CNI meta plugin and see if it behaves as expected.

Frankly, I don't think this is really a CNV specific problem, it just affects CNV more than others. 
It seems to be a general bad OCP behaviour, checking my network I have dozens of random MAC spammed per day on a pretty much idle cluster, all coming from OCP nodes. It just a matter of time and luck until it spams a MAC that already exists on the network, and in CNV case the same spamming breaks traffic on live migration as it spams the VM MAC. Maybe OCP should be fixed.

(In reply to Germano Veit Michel from comment #9)
> > - Send IPv6 ND NS/NA frequently during migration from the source
> > virt-launcher.
> 
> Or a GARP, so its more clear about the intentions in case someone is looking
> at a tcpdump and wondering what is that additional IPv6 ND spam coming from
> 2 different places now :)

Yes, although unlike IPv6, that would have to be sent with a broadcast source address.
I am not even sure it is "legal" to send such packets (they may be dropped due to the
non unicast source).

> > Action Items:
> > - Test the tuning CNI meta plugin and see if it behaves as expected.
> 
> Frankly, I don't think this is really a CNV specific problem, it just
> affects CNV more than others. 
> It seems to be a general bad OCP behaviour, checking my network I have
> dozens of random MAC spammed per day on a pretty much idle cluster, all
> coming from OCP nodes. It just a matter of time and luck until it spams a
> MAC that already exists on the network, and in CNV case the same spamming
> breaks traffic on live migration as it spams the VM MAC. Maybe OCP should be
> fixed.

Each pod is creating a network end-point device which is announcing itself.
This seems to be the default behavior of the kernel, unrelated to OCP.
For IP traffic, this seems indeed a waste, as address resolution is expected
anyway, with it comes the learning.

I am not really sure why this default has been decided on, we should probably ask
the kernel network devs about it.

We can pursue this in all direction, it all depends on the response time.

(In reply to Edward Haas from comment #10)
> (In reply to Germano Veit Michel from comment #9)
> > > - Send IPv6 ND NS/NA frequently during migration from the source
> > > virt-launcher.
> > 
> > Or a GARP, so its more clear about the intentions in case someone is looking
> > at a tcpdump and wondering what is that additional IPv6 ND spam coming from
> > 2 different places now :)
> 
> Yes, although unlike IPv6, that would have to be sent with a broadcast
> source address.
> I am not even sure it is "legal" to send such packets (they may be dropped
> due to the
> non unicast source).

The source address (L2) is unicast and should be the MAC of the VM, otherwise
it won't have the effects we want.

Note QEMU sends these at the end of migration, VMware and others do the exact same.
So its unlikely they will be dropped.

> 
> > > Action Items:
> > > - Test the tuning CNI meta plugin and see if it behaves as expected.
> > 
> > Frankly, I don't think this is really a CNV specific problem, it just
> > affects CNV more than others. 
> > It seems to be a general bad OCP behaviour, checking my network I have
> > dozens of random MAC spammed per day on a pretty much idle cluster, all
> > coming from OCP nodes. It just a matter of time and luck until it spams a
> > MAC that already exists on the network, and in CNV case the same spamming
> > breaks traffic on live migration as it spams the VM MAC. Maybe OCP should be
> > fixed.
> 
> Each pod is creating a network end-point device which is announcing itself.
> This seems to be the default behavior of the kernel, unrelated to OCP.
> For IP traffic, this seems indeed a waste, as address resolution is expected
> anyway, with it comes the learning.

Right, but this behaviour is there for years, for much longer than OCP exists.
I'd say OCP should have set more sane defaults when creating all the mesh
of devices it uses for the network plumbing. Well note really OCP but the
related subcomponents.

Still, it seems that network spamming wasn't really a problem until we had VMs.

Also, it appears that when running on other platforms (i.e. vSphere) we don't get that spam, maybe dropped at the hypervisor by MAC filtering.

> I am not really sure why this default has been decided on, we should
> probably ask
> the kernel network devs about it.
> 
> We can pursue this in all direction, it all depends on the response time.

Just my 2c, but I think changing a decades old default may break other things and this should be fixed somewhere in OCP.

(In reply to Edward Haas from comment #8)
> We had an initial discussion to evaluated this issue.
> Here are the main ideas raised:
> - Send IPv6 ND NS/NA frequently during migration from the source
> virt-launcher.
>   - A workaround before a formal version, can be tried to do using a sidecar.
> - Use the tuning CNI meta plugin to set the NOARP option for "all" and
> "default" interfaces.
>   It needs to be added to the plugin chain as the top plugin in the
> NetworkAttachementDefinition.
> - Change the specific CNI plugin (e.g. the bridge CNI) to accept the initial
> link state or the NOARP option.

A possible workaround is to make sure the guest is generating egress traffic
out of the relevant interface.

We likely do not see this problem widely because during live migration
the guest is still sending traffic out, causing all the network to re-learn
the path to the mac quickly.

(In reply to Edward Haas from comment #16)
> We likely do not see this problem widely because during live migration
> the guest is still sending traffic out, causing all the network to re-learn
> the path to the mac quickly.

In a simple network with a single switch, yes. But that's not the case in data-centers.

Some common data-center network topologies like spine-leaf/trees will only update the MAC address port in the switch table if the guest sends some broadcast frame.
It may take a while for it to send a broadcast one, and if it only sends unicast to a particular system it will only update the switches on a particular path end to end, leaving many others with the wrong entry.
So even with the guest sending traffic, it can still cause some network outage, to a subset of systems.
It would take broadcast or many unicast in different directions to update the network.

Note the IPv6 ND goes out with the mac group bit set, and floods the wrong info to all switches, it would take something similar to repair the entries until the VM is fully migrated.

Nijin could you help us assess the urgency of this? Is the workaround of the guest continuously emitting ARP an acceptable temporary solution?

(In reply to Edward Haas from comment #8)
> Thank you Germano for the detail investigation.
> 
> We had an initial discussion to evaluated this issue.
> Here are the main ideas raised:
> - Send IPv6 ND NS/NA frequently during migration from the source
> virt-launcher.
>   - A workaround before a formal version, can be tried to do using a sidecar.
> - Use the tuning CNI meta plugin to set the NOARP option for "all" and
> "default" interfaces.
>   It needs to be added to the plugin chain as the top plugin in the
> NetworkAttachementDefinition.
> - Change the specific CNI plugin (e.g. the bridge CNI) to accept the initial
> link state or the NOARP option.
> 
> Action Items:
> - Test the tuning CNI meta plugin and see if it behaves as expected.

Tuning CNI cannot be added to the plugin chain as the top plugin since it doesn't return any result (the following error is returned when trying to put the tuning at the top of the chain "Required prevResult missing").
Even if it was possible it wouldn't necessary be a good idea.

I"m not sure why "ip link set dev ifaceName arp off" disables "neighbor solicitation" messages (ipv6 is not using ARP), but even with the "arp off" I still see other ipv6 traffic when setting the interface to "up"-

12:29:38.260546 test1 Out IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48
12:29:38.260552 test0 M   IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48

I couldn't find any "sysctl" that disables "neighbor solicitation" the same way the "arp off" does. The only option I could find is to disable ipv6 by sysctl "net.ipv6.conf.all.disable_ipv6".
I don't think it is a good idea to disable IPv6 for all the interfaces in the pod. Maybe some actually need and use it, especially the primary one.
But as mentioned before, the tuning cannot be the first one in the chain, so even this option cannot be done with the tuning.

I think the best option is to ask the bridge CNI to create the interface with down state and for kubvirt to move it to up only once the qemu is ready.
I will open an RFE in the CNI plugins repo and link it here.

(In reply to Edward Haas from comment #8)
> Thank you Germano for the detail investigation.
> 
> We had an initial discussion to evaluated this issue.
> Here are the main ideas raised:
> - Send IPv6 ND NS/NA frequently during migration from the source
> virt-launcher.
>   - A workaround before a formal version, can be tried to do using a sidecar.
> - Use the tuning CNI meta plugin to set the NOARP option for "all" and
> "default" interfaces.
>   It needs to be added to the plugin chain as the top plugin in the
> NetworkAttachementDefinition.
> - Change the specific CNI plugin (e.g. the bridge CNI) to accept the initial
> link state or the NOARP option.
> 
> Action Items:
> - Test the tuning CNI meta plugin and see if it behaves as expected.

Tuning CNI cannot be added to the plugin chain as the top plugin since it doesn't return any result (the following error is returned when trying to put the tuning at the top of the chain "Required prevResult missing").
Even if it was possible it wouldn't necessary be a good idea.

I"m not sure why "ip link set dev ifaceName arp off" disables "neighbor solicitation" messages (ipv6 is not using ARP), but even with the "arp off" I still see other ipv6 traffic when setting the interface to "up"-

12:29:38.260546 test1 Out IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48
12:29:38.260552 test0 M   IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48

I couldn't find any "sysctl" that disables "neighbor solicitation" the same way the "arp off" does. The only option I could find is to disable ipv6 by sysctl "net.ipv6.conf.all.disable_ipv6".
I don't think it is a good idea to disable IPv6 for all the interfaces in the pod. Maybe some actually need and use it, especially the primary one.
But as mentioned before, the tuning cannot be the first one in the chain, so even this option cannot be done with the tuning.

I think the best option is to ask the bridge CNI to create the interface with down state and for kubvirt to move it to up only once the qemu is ready.
I will open an RFE in the CNI plugins repo and link it here.

Bridge plugin CNI issue to introduce "activateInterface" option - https://github.com/containernetworking/plugins/issues/951