Bug 2175919 - When starting, NetworkManager signals ready too early (causing a race with After=NetworkManager.service)
Summary: When starting, NetworkManager signals ready too early (causing a race with Af...
Keywords:
Status: VERIFIED
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: NetworkManager
Version: 9.2
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Thomas Haller
QA Contact: Filip Pokryvka
URL:
Whiteboard:
: 2177188 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-03-06 19:43 UTC by Thomas Haller
Modified: 2023-08-02 04:29 UTC (History)
7 users (show)

Fixed In Version: NetworkManager-1.43.5-1.el9
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker NMT-363 0 None None None 2023-03-06 19:44:28 UTC
Red Hat Issue Tracker RHELPLAN-150861 0 None None None 2023-03-06 19:44:31 UTC
freedesktop.org Gitlab NetworkManager NetworkManager-ci merge_requests 1410 0 None opened nmutil: add `ready_at_first_check` to `wait_for_nm_bus` 2023-04-28 14:22:29 UTC
freedesktop.org Gitlab NetworkManager NetworkManager merge_requests 1560 0 None opened [th/dbus-request-name-later] "RequestName" of NetworkManager D-Bus API later to fix race 2023-03-06 19:45:36 UTC

Description Thomas Haller 2023-03-06 19:43:26 UTC
A service that orders itself `After=NetworkManager.service` should reasonably expect that NetworkManager is up and running. Likewise, if `systemctl status NetworkManager` indicates "Active: active (running)", we would expect NM to be fully functional and have the D-Bus interface ready and accessible.

NetworkManager.service is `Type=dbus`, which means that systemd will automatically declare the service as started, once BusName=org.freedesktop.NetworkManager is acquired (`man systemd.service`). While there are other ways to indicate to systemd when a service is started, this is a convenient and sensible behavior for NetworkManager.

However, NetworkManager acquires the D-Bus name pretty early, when most of the initialization is not yet done. So `After=NetworkManager.service` the D-Bus name exists and NetworkManager is connected, but it's not yet fully usable.

NetworkManager does this in purpose (wrongly), to be faster at being. However, being fast is wrong, if we need to lie about being ready.

Fix that.


Note that acquiring a D-Bus name is an atomic operation. So NetworkManager uses that to detect whether another instance is running in parallel. That is useful, because then NetworkManager errors out early without doing anything. If this changes, we will first start (and mess with the system) before we notice that another NetworkManager instance is running. That can be a problem, when you try to start NetworkManager from the terminal (and forget to stop the running daemon). In production that however does not happen, because systemd ensures the service is only running once. So that problem is acceptable and there is no workaround to implement. E.g. first checking whether the service is running, which has a small overhead and is still a bit racy (but probably would would work to catch common cases where the daemon is running already).


Note that NM-ci does a workaround for that race here: https://gitlab.freedesktop.org/NetworkManager/NetworkManager-ci/-/commit/3a545c2a8df4fab80c71c13e76c8cd38fda48c19
To get CI coverage, something like:
```
diff --git i/nmci/nmutil.py w/nmci/nmutil.py
index 9908445b0d69..7575ffe2d8a9 100644
--- i/nmci/nmutil.py
+++ w/nmci/nmutil.py
@@ -62,29 +62,37 @@ class _NMUtil:
     def wait_for_nm_bus(self, timeout=DEFAULT_TIMEOUT, do_assert=True):
         busctl_argv = [
             "busctl",
             "call",
             "org.freedesktop.NetworkManager",
             "/org/freedesktop/NetworkManager",
             "org.freedesktop.NetworkManager",
             "GetAllDevices",
         ]
         timeout = nmci.util.start_timeout(timeout)
+        was_ready_at_first = True
         while timeout.loop_sleep(0.1):
             if nmci.process.run_search_stdout(
                 busctl_argv,
                 "/org/freedesktop/NetworkManager",
                 ignore_stderr=True,
                 ignore_returncode=True,
                 timeout=timeout.remaining_time(),
             ):
+                if do_assert and not was_ready_at_first:
+                    _, nm_ver = nmci.misc.nm_version_detect()
+                    assert nm_ver >= [
+                        1,
+                        43,
+                        2,
+                    ], f"NetworkManager 1.34 is expected to be ready right away"
                 return True
+            was_ready_at_first = False
         if do_assert:
             raise nmci.util.ExpectedException(
                 f"NetworkManager bus not running in {timeout.elapsed_time()} seconds"
             )
         return False
 
     def nm_size_kb(self):
         pid = self.nm_pid()
         if not pid:
             raise nmci.util.ExpectedException(
```
might work.

Comment 2 Thomas Haller 2023-03-30 11:08:06 UTC
*** Bug 2177188 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.