Bug 2173196
| Summary: | Warn in nmcli if NetworkManager daemon version doesn't match | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | Jeffrey Cutter <jcutter> |
| Component: | NetworkManager | Assignee: | Íñigo Huguet <ihuguet> |
| Status: | CLOSED ERRATA | QA Contact: | David Jaša <djasa> |
| Severity: | medium | Docs Contact: | |
| Priority: | low | ||
| Version: | 9.3 | CC: | bgalvani, ferferna, ihuguet, jcutter, lrintel, rkhan, sfaye, sukulkar, thaller, till, vbenes |
| Target Milestone: | rc | Keywords: | Triaged |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | NetworkManager-1.44.0-1.el9 | Doc Type: | No Doc Update |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-11-07 08:37:57 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Jeffrey Cutter
2023-02-24 17:12:59 UTC
```
[root@rhel02 ~]# yum list NetworkManager --showduplicates
Updating Subscription Management repositories.
Last metadata expiration check: 0:09:33 ago on Fri 24 Feb 2023 11:50:33 AM EST.
Installed Packages
NetworkManager.x86_64 1:1.30.0-7.el8 @System
Available Packages
NetworkManager.x86_64 1:1.14.0-14.el8 rhel-8-for-x86_64-baseos-rpms
NetworkManager.x86_64 1:1.20.0-3.el8 rhel-8-for-x86_64-baseos-rpms
NetworkManager.x86_64 1:1.20.0-5.el8_1 rhel-8-for-x86_64-baseos-rpms
NetworkManager.x86_64 1:1.22.8-4.el8 rhel-8-for-x86_64-baseos-rpms
NetworkManager.x86_64 1:1.22.8-5.el8_2 rhel-8-for-x86_64-baseos-rpms
NetworkManager.x86_64 1:1.26.0-8.el8 rhel-8-for-x86_64-baseos-rpms
NetworkManager.x86_64 1:1.26.0-9.el8_3 rhel-8-for-x86_64-baseos-rpms
NetworkManager.x86_64 1:1.26.0-12.el8_3 rhel-8-for-x86_64-baseos-rpms
NetworkManager.x86_64 1:1.26.0-13.el8_3 rhel-8-for-x86_64-baseos-rpms
NetworkManager.x86_64 1:1.26.0-14.el8_3 rhel-8-for-x86_64-baseos-rpms
NetworkManager.x86_64 1:1.30.0-7.el8 rhel-8-for-x86_64-baseos-rpms
NetworkManager.x86_64 1:1.30.0-9.el8_4 rhel-8-for-x86_64-baseos-rpms
NetworkManager.x86_64 1:1.30.0-10.el8_4 rhel-8-for-x86_64-baseos-rpms
NetworkManager.x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms
NetworkManager.x86_64 1:1.32.10-4.el8 rhel-8-for-x86_64-baseos-rpms
NetworkManager.x86_64 1:1.32.10-5.el8_5 rhel-8-for-x86_64-baseos-rpms
NetworkManager.x86_64 1:1.36.0-4.el8 rhel-8-for-x86_64-baseos-rpms
NetworkManager.x86_64 1:1.36.0-7.el8_6 rhel-8-for-x86_64-baseos-rpms
NetworkManager.x86_64 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms
NetworkManager.x86_64 1:1.40.0-1.el8 rhel-8-for-x86_64-baseos-rpms
NetworkManager.x86_64 1:1.40.0-2.el8_7 rhel-8-for-x86_64-baseos-rpms
NetworkManager.x86_64 1:1.40.0-5.el8_7 rhel-8-for-x86_64-baseos-rpms
[root@rhel02 ~]# yum update NetworkManager
Updating Subscription Management repositories.
Last metadata expiration check: 0:09:46 ago on Fri 24 Feb 2023 11:50:33 AM EST.
Dependencies resolved.
=======================================================================================================================================================================================================================================
Package Architecture Version Repository Size
=======================================================================================================================================================================================================================================
Upgrading:
NetworkManager x86_64 1:1.40.0-5.el8_7 rhel-8-for-x86_64-baseos-rpms 2.3 M
NetworkManager-libnm x86_64 1:1.40.0-5.el8_7 rhel-8-for-x86_64-baseos-rpms 1.9 M
NetworkManager-team x86_64 1:1.40.0-5.el8_7 rhel-8-for-x86_64-baseos-rpms 158 k
NetworkManager-tui x86_64 1:1.40.0-5.el8_7 rhel-8-for-x86_64-baseos-rpms 353 k
jansson x86_64 2.14-1.el8 rhel-8-for-x86_64-baseos-rpms 47 k
Installing dependencies:
NetworkManager-initscripts-updown noarch 1:1.40.0-5.el8_7 rhel-8-for-x86_64-baseos-rpms 141 k
Transaction Summary
=======================================================================================================================================================================================================================================
Install 1 Package
Upgrade 5 Packages
Total download size: 4.8 M
Is this ok [y/N]: y
Downloading Packages:
(1/6): jansson-2.14-1.el8.x86_64.rpm 114 kB/s | 47 kB 00:00
(2/6): NetworkManager-initscripts-updown-1.40.0-5.el8_7.noarch.rpm 340 kB/s | 141 kB 00:00
(3/6): NetworkManager-team-1.40.0-5.el8_7.x86_64.rpm 339 kB/s | 158 kB 00:00
(4/6): NetworkManager-1.40.0-5.el8_7.x86_64.rpm 7.7 MB/s | 2.3 MB 00:00
(5/6): NetworkManager-libnm-1.40.0-5.el8_7.x86_64.rpm 5.6 MB/s | 1.9 MB 00:00
(6/6): NetworkManager-tui-1.40.0-5.el8_7.x86_64.rpm 297 kB/s | 353 kB 00:01
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 2.9 MB/s | 4.8 MB 00:01
warning: /var/cache/dnf/rhel-8-for-x86_64-baseos-rpms-51b3b78d5698246b/packages/NetworkManager-initscripts-updown-1.40.0-5.el8_7.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs) 442 kB/s | 5.0 kB 00:00
Importing GPG key 0xFD431D51:
Userid : "Red Hat, Inc. (release key 2) <security>"
Fingerprint: 567E 347A D004 4ADE 55BA 8A5F 199E 2F91 FD43 1D51
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Is this ok [y/N]: y
Key imported successfully
Importing GPG key 0xD4082792:
Userid : "Red Hat, Inc. (auxiliary key) <security>"
Fingerprint: 6A6A A7C9 7C88 90AE C6AE BFE2 F76F 66C3 D408 2792
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Is this ok [y/N]: y
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Running scriptlet: NetworkManager-libnm-1:1.40.0-5.el8_7.x86_64 1/1
Upgrading : NetworkManager-libnm-1:1.40.0-5.el8_7.x86_64 1/11
Running scriptlet: NetworkManager-libnm-1:1.40.0-5.el8_7.x86_64 1/11
Running scriptlet: NetworkManager-1:1.40.0-5.el8_7.x86_64 2/11
Upgrading : NetworkManager-1:1.40.0-5.el8_7.x86_64 2/11
Running scriptlet: NetworkManager-1:1.40.0-5.el8_7.x86_64 2/11
Installing : NetworkManager-initscripts-updown-1:1.40.0-5.el8_7.noarch 3/11
Running scriptlet: NetworkManager-initscripts-updown-1:1.40.0-5.el8_7.noarch 3/11
Upgrading : jansson-2.14-1.el8.x86_64 4/11
Upgrading : NetworkManager-team-1:1.40.0-5.el8_7.x86_64 5/11
Upgrading : NetworkManager-tui-1:1.40.0-5.el8_7.x86_64 6/11
Cleanup : NetworkManager-tui-1:1.30.0-7.el8.x86_64 7/11
Cleanup : NetworkManager-team-1:1.30.0-7.el8.x86_64 8/11
Running scriptlet: NetworkManager-1:1.30.0-7.el8.x86_64 9/11
Cleanup : NetworkManager-1:1.30.0-7.el8.x86_64 9/11
Running scriptlet: NetworkManager-1:1.30.0-7.el8.x86_64 9/11
Cleanup : NetworkManager-libnm-1:1.30.0-7.el8.x86_64 10/11
Running scriptlet: NetworkManager-libnm-1:1.30.0-7.el8.x86_64 10/11
Cleanup : jansson-2.11-3.el8.x86_64 11/11
Running scriptlet: jansson-2.11-3.el8.x86_64 11/11
Verifying : NetworkManager-initscripts-updown-1:1.40.0-5.el8_7.noarch 1/11
Verifying : jansson-2.14-1.el8.x86_64 2/11
Verifying : jansson-2.11-3.el8.x86_64 3/11
Verifying : NetworkManager-team-1:1.40.0-5.el8_7.x86_64 4/11
Verifying : NetworkManager-team-1:1.30.0-7.el8.x86_64 5/11
Verifying : NetworkManager-1:1.40.0-5.el8_7.x86_64 6/11
Verifying : NetworkManager-1:1.30.0-7.el8.x86_64 7/11
Verifying : NetworkManager-libnm-1:1.40.0-5.el8_7.x86_64 8/11
Verifying : NetworkManager-libnm-1:1.30.0-7.el8.x86_64 9/11
Verifying : NetworkManager-tui-1:1.40.0-5.el8_7.x86_64 10/11
Verifying : NetworkManager-tui-1:1.30.0-7.el8.x86_64 11/11
Installed products updated.
Upgraded:
NetworkManager-1:1.40.0-5.el8_7.x86_64 NetworkManager-libnm-1:1.40.0-5.el8_7.x86_64 NetworkManager-team-1:1.40.0-5.el8_7.x86_64 NetworkManager-tui-1:1.40.0-5.el8_7.x86_64 jansson-2.14-1.el8.x86_64
Installed:
NetworkManager-initscripts-updown-1:1.40.0-5.el8_7.noarch
Complete!
[root@rhel02 ~]# nmcli c s
NAME UUID TYPE DEVICE
eth0 5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03 ethernet eth0
[root@rhel02 ~]# nmcli c m 5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03 ipv4.dns-search example.org
Error: Failed to modify connection 'eth0': ipv6.addr-gen-mode: property is invalid
[root@rhel02 ~]# systemctl status NetworkManager
● NetworkManager.service - Network Manager
Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2023-02-24 11:06:12 EST; 54min ago
Docs: man:NetworkManager(8)
Main PID: 746 (NetworkManager)
Tasks: 3 (limit: 23484)
Memory: 7.3M
CGroup: /system.slice/NetworkManager.service
└─746 /usr/sbin/NetworkManager --no-daemon
Feb 24 11:06:12 rhel02 NetworkManager[746]: <info> [1677254772.9982] device (eth0): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
Feb 24 11:06:13 rhel02 NetworkManager[746]: <info> [1677254773.0070] device (eth0): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
Feb 24 11:06:13 rhel02 NetworkManager[746]: <info> [1677254773.0073] device (eth0): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
Feb 24 11:06:13 rhel02 NetworkManager[746]: <info> [1677254773.0083] manager: NetworkManager state is now CONNECTED_LOCAL
Feb 24 11:06:13 rhel02 NetworkManager[746]: <info> [1677254773.0097] manager: NetworkManager state is now CONNECTED_SITE
Feb 24 11:06:13 rhel02 NetworkManager[746]: <info> [1677254773.0111] policy: set 'eth0' (eth0) as default for IPv4 routing and DNS
Feb 24 11:06:13 rhel02 NetworkManager[746]: <info> [1677254773.0135] device (eth0): Activation: successful, device activated.
Feb 24 11:06:13 rhel02 NetworkManager[746]: <info> [1677254773.0151] manager: NetworkManager state is now CONNECTED_GLOBAL
Feb 24 11:06:13 rhel02 NetworkManager[746]: <info> [1677254773.0182] manager: startup complete
Feb 24 12:00:45 rhel02 NetworkManager[746]: <info> [1677258045.1513] audit: op="connection-update" uuid="5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03" name="eth0" result="fail" reason="ipv6.addr-gen-mode: property is invalid"
[root@rhel02 ~]# date
Fri Feb 24 12:00:59 EST 2023
[root@rhel02 ~]# systemctl restart NetworkManager
[root@rhel02 ~]# nmcli c m 5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03 ipv4.dns-search example.org
[root@rhel02 ~]#
```
Thanks Jeffrey for the report. Just wondering, do you have any expectation/idea what to do about this? Making nmcli not require NM to be restarted is unfeasible, automatically restarting NM seems to be too dangerous. Is there anything how we should signal that NM might need to be manually restarted? Hi Till, I'm working from the (possibly wrong) assumption that all packages being updated shy of the kernel should refresh services as necessary to keep them working as expected and running the latest code. Hence my reporting of this issue. My workaround to the issue was to automate restarting NetworkManager prior to using the nmcli command. I just found the following solutions document: https://access.redhat.com/solutions/5943031 Perhaps there is something short of a complete restart of NetworkManager that can be done instead to get nmcli working again? Shy of some other way to ensure nmcli is functional, at the very least, if nmcli is run in this state, signaling that a restart of NetworkManager is required to be able to use nmcli would be a lot more user friendly than the confusing errors provided currently. Thanks, -Jeff It might be feasible to add a warning to nmcli in case NetworKManager is newer than nmcli (or libnm I guess) to consider restarting NM. If we are doing this, we would target RHEL 9. What are your thoughts about this? That sounds reasonable to me. Btw, Doing major upgrades (rhel8.x->8.7) without offline-update or reboot, seems a bad idea (on any mildly important system). Also, installing NetworkManager package does not restart the daemon (for certain reasons). Usually you would avoid problems by rebooting/offline-update, or at least have `yum install` followed by `systemctl restart NetworkManager`. But sure, that is not entirely obvious. A warning in nmcli would have helper with that. --- In the past, nmcli checked the version and errored out if it differed. That was too strict, because usually it is supposed to work. That check was then dropped ([1]). However, merely printing a warning might be a good idea still. [1] https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=3c67a1ec5e8ef35169bba961faff6bfdfacc1cfa. --- The actual bug here is > Actual results: > [root@rhel02 ~]# nmcli c m 5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03 ipv4.dns-search example.org > Error: Failed to modify connection 'eth0': ipv6.addr-gen-mode: property is invalid > > > Expected results: > [root@rhel02 ~]# nmcli c m 5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03 ipv4.dns-search example.org > [root@rhel02 ~]# While doing it this way is questionable, the expectation that this should work is justified. NetworkManager clients and the daemon are supposed to be backward compatible in both ways, precisely because it's easy to run into a situation where the version differs. It's a bug that it doesn't work, and it should be found out what exactly went wrong. (In reply to Thomas Haller from comment #6) > > [root@rhel02 ~]# nmcli c m 5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03 ipv4.dns-search example.org > > Error: Failed to modify connection 'eth0': ipv6.addr-gen-mode: property is invalid > NetworkManager clients and the daemon are supposed to be backward compatible > in both ways, precisely because it's easy to run into a situation where the > version differs. It's a bug that it doesn't work, and it should be found out > what exactly went wrong. This is fixed by commit [1] in NM 1.40.2 and later. [1] https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/70060d570b4b35e029a5d6ab8aa8026563ad580c (In reply to Thomas Haller from comment #6) > NetworkManager clients and the daemon are supposed to be backward compatible > in both ways, precisely because it's easy to run into a situation where the > version differs. It's a bug that it doesn't work, and it should be found out > what exactly went wrong. If this is a basic requirement for NM, this needs to be tested by integration tests. Do we have these? (In reply to Till Maas from comment #8) > If this is a basic requirement for NM, this needs to be tested by > integration tests. Do we have these? no. VERIFIED in 1.44.0-1.el9: - when nmcli is of higher version than the running daemon, warning is printed (upgrade scenario) - when nmcli is of lower version than the running daemon, warning is not issued (downgrade scenario) > (In reply to Till Maas from comment #8) > > If this is a basic requirement for NM, this needs to be tested by > integration tests. Do we have these? We don't and this would be quite hard to add: - one of basic assumptions of NM CI is that the NM is installed at the beginning and it's not touched with an exception of additional packages of the same version - testing of git main would only be possible with some mocking in place (daemon pretending to be of a lower version or nmcli of higher) - in principle, it should be possible to test this in downstream OS tooling however the tooling would have to be probably extended to accommodate custom tests Hi David, (In reply to David Jaša from comment #12) > - when nmcli is of lower version than the running daemon, warning is not > issued (downgrade scenario) Are you sure of this? It should warn if the version is not exactly the same, it doesn't matter if it's lower or higher. (In reply to Íñigo Huguet from comment #13) > Hi David, > > (In reply to David Jaša from comment #12) > > - when nmcli is of lower version than the running daemon, warning is not > > issued (downgrade scenario) > > Are you sure of this? It should warn if the version is not exactly the same, > it doesn't matter if it's lower or higher. When you ask, I'm actually not. It's not possible to downgrade to a version that would have this fix right now (essentially the same problem as my 2nd bullet of CI difficulties) and I didn't realize it when testing (and then writing). (In reply to David Jaša from comment #12) > ... > > (In reply to Till Maas from comment #8) > > > > If this is a basic requirement for NM, this needs to be tested by > > integration tests. Do we have these? > > We don't and this would be quite hard to add: > - one of basic assumptions of NM CI is that the NM is installed at the > beginning and it's not touched with an exception of additional packages of > the same version > - testing of git main would only be possible with some mocking in place > (daemon pretending to be of a lower version or nmcli of higher) > - in principle, it should be possible to test this in downstream OS tooling > however the tooling would have to be probably extended to accommodate custom > tests It will be covered using mocking (to test both lower and higher version) (In reply to David Jaša from comment #14) > (In reply to Íñigo Huguet from comment #13) > > Hi David, > > > > (In reply to David Jaša from comment #12) > > > - when nmcli is of lower version than the running daemon, warning is not > > > issued (downgrade scenario) > > > > Are you sure of this? It should warn if the version is not exactly the same, > > it doesn't matter if it's lower or higher. > > When you ask, I'm actually not. It's not possible to downgrade to a version > that would have this fix right now (essentially the same problem as my 2nd > bullet of CI difficulties) and I didn't realize it when testing (and then > writing). It works both ways, sorry for the noise. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (NetworkManager bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2023:6585 |