Laine, would you please triage this? Thanks.

My understanding is that the root cause of this problem is that the RARP packets (that are always sent by qemu when the guest is unpaused on the destination) are dropped/lost, and so incoming traffic goes to the wrong place until a packet is sent from the newly unpaused guest?

The first thing that needs saying is that, as Sean says, adding a command to trigger qemu's self-announce would would require a new API in libvirt, and new APIs cannot be backported - they require a rebase of the package. So if this new feature is only needed for a temporary mitigation until OVN is properly fixed, and if the mitigation is needed in a downstream libvirt that is not going to be rebased anytime soon, then the avenue of adding a libvirt API is a non-starter.

Possibly some creativity could result in a workable solution, e.g. if libvirt were to just always do a periodic self-announce for several seconds after a migrated guest was unplugged on the destination. (I mention this because it's pretty much the only thing I can think of that could be done without adding a new API). I had thought that QEMU already retransmitted the RARPS though - Dave, can you confirm or refute this? Maybe it just needs to continue a bit longer?

There was an RFE filed against libvirt years ago (before qemu exposed the self-announce thing) asking for something like this, but it has been lost to the sands of the "stale bug" policy.

(In reply to Laine Stump from comment #7)
> My understanding is that the root cause of this problem is that the RARP
> packets (that are always sent by qemu when the guest is unpaused on the
> destination) are dropped/lost, and so incoming traffic goes to the wrong
> place until a packet is sent from the newly unpaused guest?

That's my understanding too.  (Aside on terminology: "Neutron" is the
networking component of OpenStack)

> The first thing that needs saying is that, as Sean says, adding a command to
> trigger qemu's self-announce would would require a new API in libvirt, and
> new APIs cannot be backported - they require a rebase of the package. So if
> this new feature is only needed for a temporary mitigation until OVN is
> properly fixed, and if the mitigation is needed in a downstream libvirt that
> is not going to be rebased anytime soon, then the avenue of adding a libvirt
> API is a non-starter.

I agree, it's fine if libvirt does not backport the API (and I don't
think it should, either).  This is more for the long-term.

Temporarily using the QMP passthrough for 'announce-self' is fine — 
because "this doesn't affect QEMU state / lifecycle in any way that 
impacts libvirt", as DanPB noted on IRC.

[...]

(In reply to Laine Stump from comment #7)
> My understanding is that the root cause of this problem is that the RARP
> packets (that are always sent by qemu when the guest is unpaused on the
> destination) are dropped/lost, and so incoming traffic goes to the wrong
> place until a packet is sent from the newly unpaused guest?
> 
> The first thing that needs saying is that, as Sean says, adding a command to
> trigger qemu's self-announce would would require a new API in libvirt, and
> new APIs cannot be backported - they require a rebase of the package. So if
> this new feature is only needed for a temporary mitigation until OVN is
> properly fixed, and if the mitigation is needed in a downstream libvirt that
> is not going to be rebased anytime soon, then the avenue of adding a libvirt
> API is a non-starter.
> 
> Possibly some creativity could result in a workable solution, e.g. if
> libvirt were to just always do a periodic self-announce for several seconds
> after a migrated guest was unplugged on the destination. (I mention this
> because it's pretty much the only thing I can think of that could be done
> without adding a new API). I had thought that QEMU already retransmitted the
> RARPS though - Dave, can you confirm or refute this? Maybe it just needs to
> continue a bit longer?

Right it does send them a few times;  You can set the migration parameters:
   announce-initial  announce-max      announce-rounds   announce-step     

to increase the number and change the timing of the announcments.

> There was an RFE filed against libvirt years ago (before qemu exposed the
> self-announce thing) asking for something like this, but it has been lost to
> the sands of the "stale bug" policy.

It would be good to get it exposed - that's why it was added!

(In reply to Dr. David Alan Gilbert from comment #9)
> (In reply to Laine Stump from comment #7)
> > My understanding is that the root cause of this problem is that the RARP
> > packets (that are always sent by qemu when the guest is unpaused on the
> > destination) are dropped/lost, and so incoming traffic goes to the wrong
> > place until a packet is sent from the newly unpaused guest?
> > 
> > The first thing that needs saying is that, as Sean says, adding a command to
> > trigger qemu's self-announce would would require a new API in libvirt, and
> > new APIs cannot be backported - they require a rebase of the package. So if
> > this new feature is only needed for a temporary mitigation until OVN is
> > properly fixed, and if the mitigation is needed in a downstream libvirt that
> > is not going to be rebased anytime soon, then the avenue of adding a libvirt
> > API is a non-starter.

I do not expect this to actully get fixed in OVN for a long time, perhaps that team
will find time to work on it but i expect this wont happen for a year or more.

i have tried to see if we can get an ETA https://bugzilla.redhat.com/show_bug.cgi?id=2012179#c1
But i suspect this will be used even after ovn has support for live migration both as a safety measure
and for other networking solutions.

> > 
> > Possibly some creativity could result in a workable solution, e.g. if
> > libvirt were to just always do a periodic self-announce for several seconds
> > after a migrated guest was unplugged on the destination. (I mention this
> > because it's pretty much the only thing I can think of that could be done
> > without adding a new API). I had thought that QEMU already retransmitted the
> > RARPS though - Dave, can you confirm or refute this? Maybe it just needs to
> > continue a bit longer?
> 
> Right it does send them a few times;
i belive it send them 3 times 

>  You can set the migration parameters:
>    announce-initial  announce-max      announce-rounds   announce-step 

interesting is this something we can set today via libvirt?

we could perhaps have it annouch once a second for 60 seconds by default and make this configureable in nova.
that might be sufficent to mitigate the issue.
so anoucne-initial=3 anounce-max=63 announce-rounds=60 announce-step=1 something like that?

> 
> to increase the number and change the timing of the announcments.
> 
> > There was an RFE filed against libvirt years ago (before qemu exposed the
> > self-announce thing) asking for something like this, but it has been lost to
> > the sands of the "stale bug" policy.
> 
> It would be good to get it exposed - that's why it was added!

ah that answer my first question yes it would be good to expose it and we woudl proably use that as our primary api
but it would be good to aslo have a way to manually tirger the annouch. i know that qemu annouches when we migrate automaticly
but im not sure it does it when we hot attach a netwrok interface and likely we should. granted for hot attach the assumtion is
that the guest will liekly do a dhcp requst aftger it brings up the nic or similar but it might make sense to trigger an annouce
on device attachment to futer reduce latency of mac propagation in this case.

(In reply to smooney from comment #10)
> (In reply to Dr. David Alan Gilbert from comment #9)
> > (In reply to Laine Stump from comment #7)
> > > My understanding is that the root cause of this problem is that the RARP
> > > packets (that are always sent by qemu when the guest is unpaused on the
> > > destination) are dropped/lost, and so incoming traffic goes to the wrong
> > > place until a packet is sent from the newly unpaused guest?
> > > 
> > > The first thing that needs saying is that, as Sean says, adding a command to
> > > trigger qemu's self-announce would would require a new API in libvirt, and
> > > new APIs cannot be backported - they require a rebase of the package. So if
> > > this new feature is only needed for a temporary mitigation until OVN is
> > > properly fixed, and if the mitigation is needed in a downstream libvirt that
> > > is not going to be rebased anytime soon, then the avenue of adding a libvirt
> > > API is a non-starter.
> 
> I do not expect this to actully get fixed in OVN for a long time, perhaps
> that team
> will find time to work on it but i expect this wont happen for a year or
> more.
> 
> i have tried to see if we can get an ETA
> https://bugzilla.redhat.com/show_bug.cgi?id=2012179#c1
> But i suspect this will be used even after ovn has support for live
> migration both as a safety measure
> and for other networking solutions.
> 
> > > 
> > > Possibly some creativity could result in a workable solution, e.g. if
> > > libvirt were to just always do a periodic self-announce for several seconds
> > > after a migrated guest was unplugged on the destination. (I mention this
> > > because it's pretty much the only thing I can think of that could be done
> > > without adding a new API). I had thought that QEMU already retransmitted the
> > > RARPS though - Dave, can you confirm or refute this? Maybe it just needs to
> > > continue a bit longer?
> > 
> > Right it does send them a few times;
> i belive it send them 3 times 
> 
> >  You can set the migration parameters:
> >    announce-initial  announce-max      announce-rounds   announce-step 
> 
> interesting is this something we can set today via libvirt?

Not that I know of (ask a libvirt person though)

> we could perhaps have it annouch once a second for 60 seconds by default and
> make this configureable in nova.
> that might be sufficent to mitigate the issue.
> so anoucne-initial=3 anounce-max=63 announce-rounds=60 announce-step=1
> something like that?
> 
> > 
> > to increase the number and change the timing of the announcments.
> > 
> > > There was an RFE filed against libvirt years ago (before qemu exposed the
> > > self-announce thing) asking for something like this, but it has been lost to
> > > the sands of the "stale bug" policy.
> > 
> > It would be good to get it exposed - that's why it was added!
> 
> ah that answer my first question yes it would be good to expose it and we
> woudl proably use that as our primary api
> but it would be good to aslo have a way to manually tirger the annouch. i
> know that qemu annouches when we migrate automaticly
> but im not sure it does it when we hot attach a netwrok interface and likely
> we should. granted for hot attach the assumtion is
> that the guest will liekly do a dhcp requst aftger it brings up the nic or
> similar but it might make sense to trigger an annouce
> on device attachment to futer reduce latency of mac propagation in this case.

I think in the case of virtio-net it triggers some reannouncments from the guest in some cases
(including the migration case above), not sure what else does.

Note: Be careful when doing all this to make sure when you fail/cancel a migration you
do an announce again on the source to make sure you haven't partially swung it over to the
destination.

> Note: Be careful when doing all this to make sure when you fail/cancel a
> migration you
> do an announce again on the source to make sure you haven't partially swung
> it over to the
> destination.


that is a very good point the current work in progress patch that was proposed only calls self anoucne in post_live_migrate from which we cannot cancel or revert.
if we were to revert however we might want to annouch on the souce again although i dont think currently libvirt/qemu will annouch until its about to unpause the
guest on the dest which will never result in us reverting.

we do need to be carful however but i think doing this only in post_live_migrate will be sufficet for now.

Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days