Bug 2175919

Summary: When starting, NetworkManager signals ready too early (causing a race with After=NetworkManager.service)
Product: Red Hat Enterprise Linux 9 Reporter: Thomas Haller <thaller>
Component: NetworkManagerAssignee: Thomas Haller <thaller>
Status: CLOSED ERRATA QA Contact: Filip Pokryvka <fpokryvk>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 9.2CC: bgalvani, lrintel, rkhan, sfaye, sukulkar, till, vbenes
Target Milestone: rcKeywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: NetworkManager-1.43.5-1.el9 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-11-07 08:38:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Thomas Haller 2023-03-06 19:43:26 UTC
A service that orders itself `After=NetworkManager.service` should reasonably expect that NetworkManager is up and running. Likewise, if `systemctl status NetworkManager` indicates "Active: active (running)", we would expect NM to be fully functional and have the D-Bus interface ready and accessible.

NetworkManager.service is `Type=dbus`, which means that systemd will automatically declare the service as started, once BusName=org.freedesktop.NetworkManager is acquired (`man systemd.service`). While there are other ways to indicate to systemd when a service is started, this is a convenient and sensible behavior for NetworkManager.

However, NetworkManager acquires the D-Bus name pretty early, when most of the initialization is not yet done. So `After=NetworkManager.service` the D-Bus name exists and NetworkManager is connected, but it's not yet fully usable.

NetworkManager does this in purpose (wrongly), to be faster at being. However, being fast is wrong, if we need to lie about being ready.

Fix that.


Note that acquiring a D-Bus name is an atomic operation. So NetworkManager uses that to detect whether another instance is running in parallel. That is useful, because then NetworkManager errors out early without doing anything. If this changes, we will first start (and mess with the system) before we notice that another NetworkManager instance is running. That can be a problem, when you try to start NetworkManager from the terminal (and forget to stop the running daemon). In production that however does not happen, because systemd ensures the service is only running once. So that problem is acceptable and there is no workaround to implement. E.g. first checking whether the service is running, which has a small overhead and is still a bit racy (but probably would would work to catch common cases where the daemon is running already).


Note that NM-ci does a workaround for that race here: https://gitlab.freedesktop.org/NetworkManager/NetworkManager-ci/-/commit/3a545c2a8df4fab80c71c13e76c8cd38fda48c19
To get CI coverage, something like:
```
diff --git i/nmci/nmutil.py w/nmci/nmutil.py
index 9908445b0d69..7575ffe2d8a9 100644
--- i/nmci/nmutil.py
+++ w/nmci/nmutil.py
@@ -62,29 +62,37 @@ class _NMUtil:
     def wait_for_nm_bus(self, timeout=DEFAULT_TIMEOUT, do_assert=True):
         busctl_argv = [
             "busctl",
             "call",
             "org.freedesktop.NetworkManager",
             "/org/freedesktop/NetworkManager",
             "org.freedesktop.NetworkManager",
             "GetAllDevices",
         ]
         timeout = nmci.util.start_timeout(timeout)
+        was_ready_at_first = True
         while timeout.loop_sleep(0.1):
             if nmci.process.run_search_stdout(
                 busctl_argv,
                 "/org/freedesktop/NetworkManager",
                 ignore_stderr=True,
                 ignore_returncode=True,
                 timeout=timeout.remaining_time(),
             ):
+                if do_assert and not was_ready_at_first:
+                    _, nm_ver = nmci.misc.nm_version_detect()
+                    assert nm_ver >= [
+                        1,
+                        43,
+                        2,
+                    ], f"NetworkManager 1.34 is expected to be ready right away"
                 return True
+            was_ready_at_first = False
         if do_assert:
             raise nmci.util.ExpectedException(
                 f"NetworkManager bus not running in {timeout.elapsed_time()} seconds"
             )
         return False
 
     def nm_size_kb(self):
         pid = self.nm_pid()
         if not pid:
             raise nmci.util.ExpectedException(
```
might work.

Comment 2 Thomas Haller 2023-03-30 11:08:06 UTC
*** Bug 2177188 has been marked as a duplicate of this bug. ***

Comment 8 errata-xmlrpc 2023-11-07 08:38:04 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (NetworkManager bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:6585