one possible solution we discussed:


nm-cloud-setup fetches the configuration from HTTP meta data.
It should (SHA2)hash the interface per-interface, and write those hashes to a state file in /run/nm-cloud-setup.

If the existing hash for an interface matches the one on disk, then nm-cloud-setup would skip the update on the interface.

This means, nm-cloud-setup would only touch an interface the first time a device activates.

To get also re-activation of the interface right, the hash would also contain the D-Bus path of the NMActiveConnection. That one changes, when a device goes down an up again.

Also, there could be a new environment variable to opt out from that, to always, on every call, update the configuration in NetworkManager. To get the previous behavior. nm-cloud-setup in general gets configured via environment variables, so the user could write `Environment=NM_CLOUD_SETUP_ALWAYS_UPDATE=1` to nm-cloud-setup.service override.

Also, with this there is still a race. nm-cloud-setup gets triggered by a timer (nm-cloud-setup.timer) and a dispatcher script (/usr/lib/NetworkManager/dispatcher.d/90-nm-cloud-setup.sh). That means, if a device activates and concurrently a user is modifying the interface outside of NetworkManager, then nm-cloud-setup might still replace the user's configuration. I think that should be solved by installing 90-nm-cloud-setup.sh as a dispatcher script for the "pre-up" event. The "pre-up" event blocks the "activated" state in NetworkManager to be reached, which in turn blocks "NetworkManager-wait-online.service" and "network-online.target". So the race could be avoided during boot, if a user orders their scripts after network-online.target. The race won't be avoided, if the user issues `nmcli connection up eth0` at a later point.


How does that sound?

This is supposed to be used in the IPaddr2 agent when we create Virtual IPs in Pacemaker (High Availability) cluster.

So it'd be great if you can provide a way we can reserve the IP before doing "ip addr add ...":
https://github.com/ClusterLabs/resource-agents/blob/main/heartbeat/IPaddr2#L683

and a way to remove the reservation as well that we can use when we stop the Virtual IP resource.

(In reply to Thomas Haller from comment #3)
> 
> Also, with this there is still a race. [...]

this race seems to be quite a problem. A "solution" that still sometimes reconfigure the interface is likely to still cause the same problems (albeit "less" frequently). That doesn't seem like a proper solution...

(In reply to Thomas Haller from comment #3)
> one possible solution we discussed:
> 
> 
> nm-cloud-setup fetches the configuration from HTTP meta data.
> It should (SHA2)hash the interface per-interface, and write those hashes to
> a state file in /run/nm-cloud-setup.

what I suggested earlier (to hash the fetched data, persist to /run and skip update if the hash still matches) would probably work.

But there seems to be a simpler solution. "nm-cloud-setup" can just construct the profile that it would like to configure (as it currently does). Then it calls `GetAppliedConnection()` on the D-Bus API. If the returned connection is identical to what nm-cloud-setup plans to configure, it can skip the update/reapply call. No need to persist anything to disk.

(In reply to Thomas Haller from comment #8)
> (In reply to Thomas Haller from comment #3)
> > one possible solution we discussed:
> > 
> > 
> > nm-cloud-setup fetches the configuration from HTTP meta data.
> > It should (SHA2)hash the interface per-interface, and write those hashes to
> > a state file in /run/nm-cloud-setup.
> 
> what I suggested earlier (to hash the fetched data, persist to /run and skip
> update if the hash still matches) would probably work.
> 
> But there seems to be a simpler solution. "nm-cloud-setup" can just
> construct the profile that it would like to configure (as it currently
> does). Then it calls `GetAppliedConnection()` on the D-Bus API. If the
> returned connection is identical to what nm-cloud-setup plans to configure,
> it can skip the update/reapply call. No need to persist anything to disk.

Hm. That is not gonna work. nm-cloud-setup already avoids updates that don't bring anything new. So the problem are nm-cloud-setup runs in general.

nm-cloud-setup uses the Reapply() API of NetworkManager to configure the interface. We need to add a flag "preserve-external-ip" to the API, so that NetworkManager ignores IP addresses/routes that were added outside of NetworkManager.

fixed upstream by https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/1476

(In reply to Thomas Haller from comment #13)
> fixed upstream by
> https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/
> merge_requests/1476

The relevant parts are backported to nm-1-40 for rhel-8:
https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/caff311fd6a7a1021d2021f65d69a66283a1ac7e

> Redirecting this question to Thomas. Can you confirm that this is fixed in 8.8?

Yes, it's fixed.

FYI: you can also see that because this bug is a rhel-8 bug and it's state is > MODIFIED. RHEL-8.8 is currently in development, so it means the fix is in the latest nightlies (or in centos-stream-8). But thanks for double checking.

> ----
> .The `nm-cloud-setup` service no longer removes routes and manually-configured secondary IP addresses from interfaces
> 
> Based on the information received from the cloud environment, the `nm-cloud-setup` service configures network interfaces. Previously, administrators had
> to disable `nm-cloud-setup` to manually configure routes and secondary IP addresses on interfaces to avoid that the service removes them. This update adds 
> a flag to the `Reapply()` function to preserve externally added addresses and routes. As a result, administrators
> no longer need to disable the `nm-cloud-setup` service in the mentioned scenario.
> ----

This text is very good. If I had one suggestion, maybe the sentence "This update adds" can be dropped. It talks about how it's implemented internally, it is not important for the nm-cloud-setup user. But either way a good text, IMO. Thanks.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (NetworkManager bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:2968