RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2175919 - When starting, NetworkManager signals ready too early (causing a race with After=NetworkManager.service)
Summary: When starting, NetworkManager signals ready too early (causing a race with Af...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: NetworkManager
Version: 9.2
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Thomas Haller
QA Contact: Filip Pokryvka
URL:
Whiteboard:
: 2177188 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-03-06 19:43 UTC by Thomas Haller
Modified: 2023-11-07 10:11 UTC (History)
7 users (show)

Fixed In Version: NetworkManager-1.43.5-1.el9
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-11-07 08:38:04 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker NMT-363 0 None None None 2023-03-06 19:44:28 UTC
Red Hat Issue Tracker RHELPLAN-150861 0 None None None 2023-03-06 19:44:31 UTC
Red Hat Product Errata RHBA-2023:6585 0 None None None 2023-11-07 08:38:30 UTC
freedesktop.org Gitlab NetworkManager NetworkManager-ci merge_requests 1410 0 None opened nmutil: add `ready_at_first_check` to `wait_for_nm_bus` 2023-04-28 14:22:29 UTC
freedesktop.org Gitlab NetworkManager NetworkManager merge_requests 1560 0 None opened [th/dbus-request-name-later] "RequestName" of NetworkManager D-Bus API later to fix race 2023-03-06 19:45:36 UTC

Description Thomas Haller 2023-03-06 19:43:26 UTC
A service that orders itself `After=NetworkManager.service` should reasonably expect that NetworkManager is up and running. Likewise, if `systemctl status NetworkManager` indicates "Active: active (running)", we would expect NM to be fully functional and have the D-Bus interface ready and accessible.

NetworkManager.service is `Type=dbus`, which means that systemd will automatically declare the service as started, once BusName=org.freedesktop.NetworkManager is acquired (`man systemd.service`). While there are other ways to indicate to systemd when a service is started, this is a convenient and sensible behavior for NetworkManager.

However, NetworkManager acquires the D-Bus name pretty early, when most of the initialization is not yet done. So `After=NetworkManager.service` the D-Bus name exists and NetworkManager is connected, but it's not yet fully usable.

NetworkManager does this in purpose (wrongly), to be faster at being. However, being fast is wrong, if we need to lie about being ready.

Fix that.


Note that acquiring a D-Bus name is an atomic operation. So NetworkManager uses that to detect whether another instance is running in parallel. That is useful, because then NetworkManager errors out early without doing anything. If this changes, we will first start (and mess with the system) before we notice that another NetworkManager instance is running. That can be a problem, when you try to start NetworkManager from the terminal (and forget to stop the running daemon). In production that however does not happen, because systemd ensures the service is only running once. So that problem is acceptable and there is no workaround to implement. E.g. first checking whether the service is running, which has a small overhead and is still a bit racy (but probably would would work to catch common cases where the daemon is running already).


Note that NM-ci does a workaround for that race here: https://gitlab.freedesktop.org/NetworkManager/NetworkManager-ci/-/commit/3a545c2a8df4fab80c71c13e76c8cd38fda48c19
To get CI coverage, something like:
```
diff --git i/nmci/nmutil.py w/nmci/nmutil.py
index 9908445b0d69..7575ffe2d8a9 100644
--- i/nmci/nmutil.py
+++ w/nmci/nmutil.py
@@ -62,29 +62,37 @@ class _NMUtil:
     def wait_for_nm_bus(self, timeout=DEFAULT_TIMEOUT, do_assert=True):
         busctl_argv = [
             "busctl",
             "call",
             "org.freedesktop.NetworkManager",
             "/org/freedesktop/NetworkManager",
             "org.freedesktop.NetworkManager",
             "GetAllDevices",
         ]
         timeout = nmci.util.start_timeout(timeout)
+        was_ready_at_first = True
         while timeout.loop_sleep(0.1):
             if nmci.process.run_search_stdout(
                 busctl_argv,
                 "/org/freedesktop/NetworkManager",
                 ignore_stderr=True,
                 ignore_returncode=True,
                 timeout=timeout.remaining_time(),
             ):
+                if do_assert and not was_ready_at_first:
+                    _, nm_ver = nmci.misc.nm_version_detect()
+                    assert nm_ver >= [
+                        1,
+                        43,
+                        2,
+                    ], f"NetworkManager 1.34 is expected to be ready right away"
                 return True
+            was_ready_at_first = False
         if do_assert:
             raise nmci.util.ExpectedException(
                 f"NetworkManager bus not running in {timeout.elapsed_time()} seconds"
             )
         return False
 
     def nm_size_kb(self):
         pid = self.nm_pid()
         if not pid:
             raise nmci.util.ExpectedException(
```
might work.

Comment 2 Thomas Haller 2023-03-30 11:08:06 UTC
*** Bug 2177188 has been marked as a duplicate of this bug. ***

Comment 8 errata-xmlrpc 2023-11-07 08:38:04 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (NetworkManager bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:6585


Note You need to log in before you can comment on or make changes to this bug.