Bug 1970021

Description of problem:

RHCOS performs an overlay mount at NetworkManager configuration directory so that changes at NetworkManager level are never persisted.

The overlay mount looks like this:

$ mount | grep NetworkManager
overlay on /etc/NetworkManager/system-connections-merged type overlay (rw,relatime,seclabel,lowerdir=/etc/NetworkManager/system-connections,upperdir=/run/nm-system-connections,workdir=/run/nm-system-connections-work)

And NetworkManager is configured to store its connections configurations at this directory as per this config drop-in:

$ cat /etc/NetworkManager/conf.d/99-keyfiles.conf 
[keyfile]
path=/etc/NetworkManager/system-connections-merged

The problem is that if networking configuration made by nmstate is not persisted and such configuration is required the next time that a node boots in order for it to be able to reach the kube-apiserver, then that node won't be able to start the nmstate handler pod so that its networking gets confirured.

Version-Release number of selected component (if applicable):

4.7

How reproducible:

Always if your node requires the networking configuration provided via nmstate to reach the kube-apiserver

Steps to Reproduce:
1. Reboot the node if it requires the networking configuration provided via nmstate to reach the kube-apiserver
2.
3.

Actual results:

No network connectivity, node not ready

Expected results:

Network connectivity, node ready

Additional info:

As per this BZ (https://bugzilla.redhat.com/show_bug.cgi?id=1916363), it seems that other components like OVN faced situations where copying from /etc/NetworkManager/system-connections-merged back to /etc/NetworkManager/system-connections was required.