Bug 2090739

+++ This bug was initially created as a clone of Bug #2046340 +++

While forcefully transferring leadership, leader completes all the current
'in-flight' commands, so the higher layers can re-try corresponding transactions
when the new leader is elected.  However, most of these commands are already
sent to followers, hence they will actually be committed by the majority of
the cluster members, i.e. will be treated as committed by the new leader, unless
there is an actual network problem between servers.  However, the old leader
will decline append replies, since it's not the leader anymore and commands
are already completed with RAFT_CMD_LOST_LEADERSHIP status.

New leader will replicate the commit index back to the old leader.
Old leader, will re-try the previously completed transaction anyway, because
the "cluster error"s are temporary.  If transaction had some prerequisites
that doesn't allow double committing or there are other database
constraints (like indexes) that will not allow transaction to be committed
twice, the server will reply to the client with a false-negative transaction
result.  If there are no prerequisites or additional database constraints,
the server will execute the same transaction second time as a follower.
E.g. in the OVN case, this may result in creation of duplicated logical
switches / routers / load balancers.  I.e. resources with the same non-indexed
name.  That may cause issues later where ovn-nbctl will not be able to add
ports to these switches or some other unpleasant consequences.

This BZ is to investigate and, probably, fix the most common case, where the
leadership is transferred voluntarily under normal operation.  The idea is
to not complete all commands in case of of compaction-triggered leadership
transfer and let them timeout or be committed by advancing the commit index
after receiving messages from a new leader ("Command completed without reply").

Similar issue was fixed before for the transaction executed through the
follower during a leader failover:
  5a9b53a51ec9 ("ovsdb raft: Fix duplicated transaction execution when leader failover.")

There are sill, probably, rare cases where 'at-least-once' semantics might
still be a thing. E.g. in corner cases involving the client re-connection
and actual network / communication issues between raft servers.  But this
needs much deeper analysis.  The change, AFAIU, should eliminate duplicated
transaction commits and false-negatives in normal operation.