Here's the package version we were  using:

Name 	Version 	Release/Dist 	Arch
NetworkManager 	1.36.0 	2.el8 	x86_64
NetworkManager-cloud-setup 	1.36.0 	2.el8 	x86_64
NetworkManager-libnm 	1.36.0 	2.el8 	x86_64
NetworkManager-ovs 	1.36.0 	2.el8 	x86_64
NetworkManager-team 	1.36.0 	2.el8 	x86_64
NetworkManager-tui 	1.36.0 	2.el8 	x86_64

Skimming things here, https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/80c9e2d9eca328ee69e9046648bb08f02e05696e may be related?

BTW, it should be possible to boot an 8.5 node, use `rpm-ostree usroverlay` and then `rpm -Uvh` on the 8.6 builds to try verifying that direction.

there are two problems here.

1) when NetworkManager restarts, it leaves the interface up (so that you could restart remotely, without cutting yourself off). It also means, after restart, it currently will not add IP addresses/routes that are not there, and not delete IP adresses/routes which are unexpectedly there.

The intend is, that you shouldn't notice that NetworkManager restarted.  It also means, restarting the daemon is usually the wrong thing to do, because NetworkManager actively tries that there are not changes by doing that. It only makes sense when you update the rpm, or you change a fundamental setting in NetworkManager.conf (many settings can be reloaded without restart). In those cases, are reboot is the safe option. Of course, restart must work well, so we have a problem here that we need to fix.

In this case, the restart happened while the interface was not fully configured (at 1650549654.8070). NetworkManager doesn't remember that the IP addresses are not there, because when it restarted the profile was only half up. So after restart it assumes that the address is missing for legitimate reasons, and will not re-add it(*).

This behavior is flawed. When restarting, NetworkManager needs to take full control of the interface again, and re-configure it. That makes the restart more noticeable, but it's necessary. It's probably still a good advise: don't restart NetworkManager unless you really need to and know what you do.

This aspect is a duplicate of bug 2050216.


2) not adding the IP address (*) is a problem, because it means we can also not ACCEPT the lease. Note that after restart, NM will do DHCP, but it would not have configured the IP address. Then, accepting the lease fails (at 1650549655.0502). Even if it wouldn't fail (to accept the lease), due to problem (1) it would not configure the IP address from DHCP. The second problem here is that DHCP fails, and that NM doesn't restart it. DHCP needs to be restarted on fatal errors to retry. That is a bug, and was partly discussed on bug 2001986. Usually, you wouldn't notice, because DHCP does not fail in such a bad way.

about problem 2, I opened bug 2077752.

This rhbz depends on the two mentioned bugs.

The `systemctl restart NetworkManager` is in https://github.com/openshift/machine-config-operator/blob/2a1e3a6e53214b6470191389ba3c0e16a5c14ee7/templates/common/_base/files/configure-ovs-network.yaml#L391
and indeed it seems likely to me it could just be `systemctl reload NetworkManager`.

So https://github.com/openshift/machine-config-operator/pull/2992 is a bit related to this I think but I just tossed up https://github.com/openshift/machine-config-operator/pull/3119

ok,  I closed bug 2077752 as duplicate of this one. Because otherwise this rhbz had no purpose and was only waiting for the other two bugs. Also, I did it this way, because this bug has some discussion that might be important, so I'd rather not close this as a duplicate of the bug with no discussion, but the other way around).


Let's interpret this bug (bug 2077605) to be about problem 2) from comment 4.
For problem 1), please refer to bug 2050216.

Note that the bigger problem of this issue is actually bug 2050216 (not bug 2077605). So you probably want to follow up there...

*** Bug 2077752 has been marked as a duplicate of this bug. ***

@thaller
What about having NM use its existing SIGTERM handler to block shutdown until it's reached an "idle" or stable state?

(In reply to Colin Walters from comment #10)
> @thaller
> What about having NM use its existing SIGTERM handler to block shutdown
> until it's reached an "idle" or stable state?

I think that would not work.

For one, trying to get a DHCP lease could take a long time (or even indefinitely, depending on `ipv4.dhcp-timeout`). This could block shutdown indefinitely, and if you give up waiting after a timeout, you still can run into problem 1.

Also, the combined state of the device (`nmcli device`) is composed of several things, among others IPv4 and IPv6 configuration. Usually, we consider a device ready, when it reaches "activated" state. That then would for example unblock `NetworkManager-wait-online.service`). But with `ipv4.may-fail=yes`, the device can be in activated state when only IPv6 is ready, while DHCPv4 is still pending. So it would be hard to determine when all configuration (IP and other) is ready, so that some form of problem 1) doesn't happen during restart.

OK, I think another alternative is serializing to e.g. `/run` the state that we want to keep.

    @rhbz1086906
    @delete_testeth0 @restart_if_needed
    @wait-online-for-both-ips
    Scenario: NM - general - wait-online - for both ipv4 and ipv6
    * Prepare simulated test "testG" device
    * Add "ethernet" connection named "con_general" for device "testG" with options "ipv4.may-fail no ipv6.may-fail no"
    * Restart NM
    * Execute "/usr/bin/nm-online -s -q --timeout=30"
    When "inet .* global" is visible with command "ip a s testG"
    Then "inet6 .* global" is visible with command "ip a s testG"

tested 100 times w/o any issue with main branch copr package

We have a new test in NMCI
https://gitlab.freedesktop.org/NetworkManager/NetworkManager-ci/-/merge_requests/1050

It randomly slows down the DHCP server and does a service restart. This covers both situations when DHCPv4 is done or not.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (NetworkManager bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:7680