+++ This bug was initially created as a clone of Bug #2132754 +++

nm-cloud-setup reconfigures the network. It does so frequently and automatically. It is also enabled by default in RHEL cloud images.

When doing that, it removes IP addresses that are not expected. That causes problems, for services that add IP addresses on the interface (by not using NetworkManager API but netlink/iproute2 directly).

The solution will be two fold:

1) bug 2132754: nm-cloud-setup uses Reapply() D-Bus call. The Reapply() call will get a new flag "preserved-external-ip", so that NetworkManager leaves externally configured IP addresses/routes on the interface alone.

2) this rhbz. When the device/connection activates, it should stay in "activating" state until nm-cloud-setup completed and configured additional addresses. This will also delay "NetworkManager-wait-online.service" and "network-online.target" and allows the user to order themselves after nm-cloud-setup is done.



Note that we could solve the ordering partially by only delaying network-online.target. For that, there could be a service which prevents "network-online.target" to be reached while nm-cloud-setup is still running. However, it would be nicer if you can also see in `nmcli device` that the device is still in "activating" state. We have a mechanism for that, the dispatcher "pre-up" event. We need to install a pre-up script which blocks activation while nm-cloud-setup is running. The pre-up script runs pretty late in the activation, when the interface is otherwise mostly configured.

One problem is that nm-cloud-setup does not run per-interface. It runs and configures all interfaces. If then a pre-up event comes for eth0 and eth1, then they both need to block until nm-cloud-setup is complete. That might be slightly compliated to get right (but maybe not. A `systemctl restart nm-cloud-setup.service` in the pre-up dispatcher script might be sufficient).

Another downside is that eth0 and eth1 both need to wait for nm-cloud-setup to complete. Maybe eth0 could complete faster. But that seems not a big problem.


I think all this needs is a blocking "pre-up" script that calls `systemctl restart nm-cloud-setup.service` and waits for it to complete.