Description of problem:

While using a bridge additional network, if something (like a nmstate applying a nodenetworkpolicy) forces a rebuild of the bridge used by the additional bridge network, the veths that were previously attached to the bridge are not re-attached so the secondary network stops working.

Version-Release number of selected component (if applicable):

4.8.19

How reproducible:

Whenever the bridge used by Multus additional network is rebuilt.

Steps to Reproduce:
1. Something to cause a rebuild of the bridge used by Multus. In our case, it was something (likely nmstate) causing a setting change in NetworkManager that, in turn, caused NetworkManager to bring down and up the connection used to create the bridge.

Actual results:

Bridge doesn't get the veths used to wire the pods to it re-attached.

Expected results:

Bridge gets the veths used to wire the pods to it re-attached.

Additional info:

The exact trigger that makes it easy to reproduce is to have the bridge defined via nmstate and then bring down and up the networkmanager connection created by nmstate manually.

The exact way that causes this connection to be brought down and up is yet undetermined and will be investigated in a separate BZ to nmstate team.

However, this does not remove the fact that Multus bridge networks should not be leaving orphaned veths around.