Bug 1971527

on AWS images, there is nm-cloud-setup running that reconfigures the network.


Does

  systemctl disable nm-cloud-setup.service
  systemctl disable nm-cloud-setup.timer

help?

The above suggestion to disable nm-cloud-setup is working. I am posting my replication results in RHEL 8.4 AMI here. I launched an instance with RHEL 8.4 AMI with 2 network interfaces attached

        # ip a
	1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
	    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
	    inet 127.0.0.1/8 scope host lo
	       valid_lft forever preferred_lft forever
	    inet6 ::1/128 scope host
	       valid_lft forever preferred_lft forever
	2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc fq_codel state UP group default qlen 1000
	    link/ether 0a:6b:6a:73:8e:0c brd ff:ff:ff:ff:ff:ff
	    inet 172.31.45.194/20 brd 172.31.47.255 scope global noprefixroute eth0
	       valid_lft forever preferred_lft forever
	3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc fq_codel state UP group default qlen 1000
	    link/ether 0a:49:e2:8c:9f:b2 brd ff:ff:ff:ff:ff:ff
	    inet 172.31.32.249/20 brd 172.31.47.255 scope global noprefixroute eth1
	       valid_lft forever preferred_lft forever
	    inet6 fe80::4787:47f3:14c0:4d20/64 scope link noprefixroute
	       valid_lft forever preferred_lft forever


	# ip r
	default via 172.31.32.1 dev eth0 proto dhcp metric 100
	default via 172.31.32.1 dev eth1 proto dhcp metric 101
	172.31.32.0/20 dev eth0 proto kernel scope link src 172.31.45.194 metric 100
	172.31.32.0/20 dev eth1 proto kernel scope link src 172.31.32.249 metric 101

	# nmcli connection show
	NAME                UUID                                  TYPE      DEVICE
	System eth0         5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03  ethernet  eth0
	Wired connection 1  2736fb27-69b3-3307-8e57-f0fef05a7638  ethernet  eth1
	ens3                02cc87e6-dc7c-4c9e-a530-f908611c7f5e  ethernet  --

	# sudo systemctl disable nm-cloud-setup.service
	Removed /etc/systemd/system/NetworkManager.service.wants/nm-cloud-setup.service.


	# sudo systemctl disable nm-cloud-setup.timer
	Removed /etc/systemd/system/timers.target.wants/nm-cloud-setup.timer.


	# nmcli connection modify "Wired connection 1" +ipv4.routes "10.20.38.0/24 172.31.32.1"

	# nmcli connection up "Wired connection 1"
	Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/3)

	# ip r
	default via 172.31.32.1 dev eth0 proto dhcp metric 100
	default via 172.31.32.1 dev eth1 proto dhcp metric 101
	10.20.38.0/24 via 172.31.32.1 dev eth1 proto static metric 101		<---- The Route is successfully added
	172.31.32.0/20 dev eth0 proto kernel scope link src 172.31.45.194 metric 100
	172.31.32.0/20 dev eth1 proto kernel scope link src 172.31.32.249 metric 101

> However, I wonder whether both the services were introduced in RHEL8.4 AMIs since customer did not face the issue in RHEL8.3 AMI.

Yes, this is new in RHEL-8.4 AMI.



This happens here: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/blob/654bb92a6968d9cee03b0c70d7797b057734e2a3/src/nm-cloud-setup/main.c#L349
I guess, nm-cloud-setup should merge the routes/addresses/rules from the profile too, before adding the automatically added ones. That would make sense.

Possible fix: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/893

nm-cloud-setup is supposed to configure networking in a cloud environment automatically.

When a user has special requirements (like configuing manual routes), then they are already in a position where they explicitly reconfigure the networking. In such a case it is completely OK to disable nm-cloud-setup, if the provided automatism doesn't do what the user wants.

Still,

- The out of the box behavior should be better, which I think https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/893 should improve. This change probably will land in rhel-8.6, but likely won't land in rhel-8.5 (do 8.5 Z stream releases result in new AMI images to be generated?). Anyway, while the patch probably would help with this particular case, it's not possible that the automatic behavior is optimal for everybody.

I am not familiar how cloud images are provided to AWS. Could you please ask your question to the cloud-team?

- I am not aware which KB articles exist (if any). In general, this is something that should be well documented. It should also be documented that it is fully OK to disable nm-cloud-setup and configure whatever special requirements the user has. I wouldn't call that a "workaround". `man nm-cloud-setup` ([1]) should also explain it, but if something is unclear there, we'd like to improve the manual. 


[1] https://networkmanager.pages.freedesktop.org/NetworkManager/NetworkManager/nm-cloud-setup.html

> This change probably will land in rhel-8.6, but likely won't land in rhel-8.5 (do 8.5 Z stream releases result in new AMI images to be generated?).

Sorry, I confused the release numbers :(

https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/893 likely will make it into rhel-8.5, but not 8.4 (or 8.4.z).

fixed on upstream nm-1-32 branch (https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/893).

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: NetworkManager security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4361