> Expected results:
> Only one connection for ens3.


When NetworkManager runs in initrd, then nm-initrd-generator tool generates in-memory connections (stored under /run).
It does not mangle existing connections, but creates a new one.

You say, only one profile is expected. But it's not clear how you imagine this to work. If the kernel command line specifies a certain configuration, why should another existing configuration be changed (merged? Or hidden altogether?). Or how is this supposed to work?


What a usercould do, is generate an initrd and add a connection profile there. That could then be used instead of the one generated from the kernel command line.


> Applying specific settings won't work.
> # nmcli con edit ens3

You have two profiles with the same name. In that case, it would be better to refer to the profile by UUID (which is unique).



> Workaround: revert to network-legacy dracut module.

The dracut module also generates a new profile (in ifcfg format). So it's not clear, why that avoids the problem. Well, maybe because at the end of the early boot, the ifcfg file gets copied over to real-root, thereby overwriting the file that is there. Overwriting user configuration in /etc seems not a good idea.



> The main issue is the fact they are not able to apply specific properties to the right connection.

In your example, both profiles have the same name. Either, rename the profile to have distinct names:

  nmcli connection modify uuid c3050dff-5bed-4e36-9b09-3b2ca37732b4 con-name ens3-2

or, refer to the desired profile by UUID, which is unique.

> But it's not clear how you imagine this to work.

I imagine this to work the same way than RHEL 8.2 and previous versions. 

> The dracut module also generates a new profile (in ifcfg format). So it's not clear, why that avoids the problem.

I don't know the why, but this change of behaviour is a regression.

> or, refer to the desired profile by UUID, which is unique.

This UUID changes at each boot (see also rhbz#1918002), hence it would be impossible to get the new settings persistent.

> What a usercould do, is generate an initrd and add a connection profile there.

AFAIK it is not documented, and I understand a 2nd profile will be kept.
Is it possible to include an ifcfg file as a profile inside the initrd?

Please note this case concerns all early net boots, clevis/tang but also iSCSI or any diskless system.

I have a very similar issue on my own laptop running Fedora and getting unlocked by Clevis.
I filed BZ #1918002 against Fedora for this.

For me, using the "omit ifcfg" workaround doesn't work because I need to reconfigure the network interface with specific settings after switching root (to set my own domain name "libvirt" + set autoneg to off because the dock from Lenovo is not stable).

In this end, I can workaround this by creating a hacky service that cleans up the boot interface before calling NetworkManager-wait-online.service, something like this:

/etc/systemd/system/network-switch-to-dock.service:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
[Unit]
Description=Switch to Dock connection if booted on 'enp0s20f0u2u1u2'
Requires=NetworkManager.service
After=NetworkManager.service
Before=NetworkManager-wait-online.service
ConditionPathExists=/sys/class/net/enp0s20f0u2u1u2

[Service]
Type=oneshot
ExecStart=/bin/sh -c "nmcli connection show --active | grep -qw enp0s20f0u2u1u2 && nmcli connection up 'Ethernet Dock' && nmcli connection delete enp0s20f0u2u1u2 || true"

[Install]
WantedBy=multi-user.target
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

It's **very** ugly but I didn't find something better at all.
Indeed, omitting ifcfg module doesn't help because NetworkManager will not reapply the persistent ifcfg file after switching root.

I think we need to come up with a good solution: to me, the only way is to be able to have the UUID of the boot interface be somehow "persistent" and match what is set in the ifcfg file on the real root file system.

Please check BZ #1918002 as well.

The current understanding of the problem is that

1) initrd with NM creates in-memory profiles in /run
2) Users would like to adjust the settings and save the profiles as a persistent profile in /etc
3) After reboot, initrd creates a new in-memory profile in /run, therefore the persistent profile in /etc is not activated

A systemd service to rectify this is probably the best approach at this momemt to workaround this:


A script like this:

profile=ens3
persistent_profile="/etc/sysconfig/network-scripts/ifcfg-$profile"
var_run_profile="/run/NetworkManager/system-connections/$profile.nmconnection"

if [[ -f "${persistent_profile}" ]] && [[ -f "${var_run_profile}" ]]
then
    nmcli con up "${persistent_profile}"
    nmcli con delete "${var_run_profile}"
fi

as a systemd service as proposed in comment 12 might solve the reported issue.

What is still unclear, is the auto-connect priority the actual setting that needs to be changed with nmcli or is this just an example?

> What is still unclear, is the auto-connect priority the actual setting that needs to be changed with nmcli or is this just an example?

Yes, it was just given as an example.
Currently they reverted to network-legacy as a workaround, waiting for a fix.

(In reply to Christophe Besson from comment #16)
> > What is still unclear, is the auto-connect priority the actual setting that needs to be changed with nmcli or is this just an example?
> 
> Yes, it was just given as an example.
> Currently they reverted to network-legacy as a workaround, waiting for a fix.

There is special handling in NM to match a network interface configuration based on the IP configuration to profiles on disk that match this. For example if you configure the same IP address in initramfs and a profile on disk, NM will show the profile as activated. From what I understood, NM will not apply any additional settings from the profile on disk, also with the network-legacy module. Therefore it would be good to have an example where you could apply a change from the on-disk profile after switching from initrd to the real root system that worked with network-legacy but does not work now. Currently it seems to me that the possibility to automatically enable a different profile after switching from initrd to real root is a missing feature for both network-legacy and network-manager.
You can try this by changing the IP configuration with nmcli: 
nmcli con modify ens3 ipv4.addressses 192.168.122.222/24

Then you should see two profiles after reboot both with legacy and network-manager modules.


Changing the auto-connect priority works as expected with then network-manager module. When deactivating all profiles, the one on disk is activated with a higher priority than the one from initrd.

> There is special handling in NM to match a network interface configuration based on the IP configuration to profiles on disk that match this. For example if you configure the same IP address in initramfs and a profile on disk, NM will show the profile as activated.

Note sure to see what you mean, as I can see two profiles/connections, and again the one where on-disk settings have been applied is not the one which is activated.


> From what I understood, NM will not apply any additional settings from the profile on disk, also with the network-legacy module. Therefore it would be good to have an example where you could apply a change from the on-disk profile after switching from initrd to the real root system that worked with network-legacy but does not work now. Currently it seems to me that the possibility to automatically enable a different profile after switching from initrd to real root is a missing feature for both network-legacy and network-manager.

Ok, it could be considered as a missing feature in both cases (NM and network-legacy).
BUT... in the network-legacy / RHEL 8.2 use case, omitting the dracut's ifcfg module leads to the expected behaviour, and this is widely used for users needing early networking with custom settings (like the one we gave as an example). This is documented here:
https://access.redhat.com/solutions/3017441

Based on this, I still consider that behaviour as a regression.


> Changing the auto-connect priority works as expected with then network-manager module. When deactivating all profiles, the one on disk is activated with a higher priority than the one from initrd.

Indeed. I checked by creating the following service and then it works as expected:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# cat /etc/systemd/system/early-networking.service
[Unit]
Description=Early networking workaround
Requires=NetworkManager.service
After=NetworkManager.service
Before=NetworkManager-wait-online.service
ConditionKernelCommandLine=rd.neednet

[Service]
Type=oneshot
ExecStart=/bin/sh -c "nmcli con down $(nmcli -f NAME con show --active) || true"

[Install]
WantedBy=multi-user.target
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Could it be handled directly by NM in this case?

(In reply to Christophe Besson from comment #19)
> > There is special handling in NM to match a network interface configuration based on the IP configuration to profiles on disk that match this. For example if you configure the same IP address in initramfs and a profile on disk, NM will show the profile as activated.
> 
> Note sure to see what you mean, as I can see two profiles/connections, and
> again the one where on-disk settings have been applied is not the one which
> is activated.


I was unclear. I described what you see with the legacy module since it only configures the interfaces but does not create an NM profile. Therefore, if there is a NM profile on disk, NM will show the profile as activated if the IP configuration matches. But NM will not activate the profile on disk (as far as I understood). Therefore, any settings that are only on disk but not configured in the kernel are not picked up.

With the network-manager module, a in-memory profile is created in initrd which is shown as activated after booting. If there is another profile on disk, it is shown as inactive. So the reporting is more accurate here.

The configuration examples that are given here regarding the auto connect priority do not affect an activated profile or interface. They only matter for inactive profiles since they describe which profile should be activated if one needs to be activated. Therefore, from a functional point of view, it works.


> > From what I understood, NM will not apply any additional settings from the profile on disk, also with the network-legacy module. Therefore it would be good to have an example where you could apply a change from the on-disk profile after switching from initrd to the real root system that worked with network-legacy but does not work now. Currently it seems to me that the possibility to automatically enable a different profile after switching from initrd to real root is a missing feature for both network-legacy and network-manager.
> 
> Ok, it could be considered as a missing feature in both cases (NM and
> network-legacy).
> BUT... in the network-legacy / RHEL 8.2 use case, omitting the dracut's
> ifcfg module leads to the expected behaviour, and this is widely used for

What do you mean by expected behavior? As far as I understand, with legacy/RHEL 8.2, it is also not possible to re-configure the network interface after initrd using a on-disk profile. Maybe there is an error in this analysis but this is the current understanding.

> users needing early networking with custom settings (like the one we gave as
> an example). This is documented here:
> https://access.redhat.com/solutions/3017441
> 
> Based on this, I still consider that behaviour as a regression.
> 
> 
> > Changing the auto-connect priority works as expected with then network-manager module. When deactivating all profiles, the one on disk is activated with a higher priority than the one from initrd.
> 
> Indeed. I checked by creating the following service and then it works as
> expected:
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> # cat /etc/systemd/system/early-networking.service
> [Unit]
> Description=Early networking workaround
> Requires=NetworkManager.service
> After=NetworkManager.service
> Before=NetworkManager-wait-online.service
> ConditionKernelCommandLine=rd.neednet
> 
> [Service]
> Type=oneshot
> ExecStart=/bin/sh -c "nmcli con down $(nmcli -f NAME con show --active) ||
> true"
> 
> [Install]
> WantedBy=multi-user.target
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> Could it be handled directly by NM in this case?

If the on-disk profile configure a different IP address than the initrd, this service will change the systems behavior by changing the IP address from the dracut/initrd one to the one on disk. This would be a problem if the on-disk profiles are stale and the kernel commandline is current. Therefore it does not seem to be so easy. Having a possibility to update the network configuration during the switch from initrd to real root seems like a valid feature request at this point but it still needs to be refined.

> BUT... in the network-legacy / RHEL 8.2 use case, omitting the
> dracut's ifcfg module leads to the expected behaviour, and this is
> widely used for users needing early networking with custom settings
> (like the one we gave as an example). This is documented here:
> https://access.redhat.com/solutions/3017441

From what I see, using 'network-legacy' and omitting the 'ifcfg' works
only in some cases. It doesn't work, for example, if the user changes
the interface from static to dynamic addressing (or vice versa), or if
there are different routes.

This happens because when NM starts with an interface configured, it
generates an in-memory connection and tries to match it to a on-disk
one. If it finds a matching on-disk connection, NM will activate it,
otherwise a external-volatile connection is created.

The match is based on the IP method, routes, MAC addresses, and other
properties:

 https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/blob/1.30.4/src/core/NetworkManagerUtils.c#L700

So, yes, there is a regression in some cases. But this use case of
having a different network configuration in real root compared to
initrd doesn't seem fully supported even with the network-legacy
module.

To support this use case, we could add a new configuration option in
NetworkManager.conf to tell NM to ignore existing configuration on one
or multiple interfaces. For example:

  [device]
  match-device=interface-name:enp1s0
  preserve-configuration=no

The default value would be 'yes' to keep the old behavior. 'no' means
that NM doesn't generate connections for interfaces already
configured.

If the user has the network-legacy module without the ifcfg module in
initrd, setting "preserve-configuration=no" is enough.

If NM runs in the initrd, it's not enough, because the connections
generated in the initrd are copied to the real root. Even if NM
doesn't generate new connections, there are still those from the
initrd that conflict with user's ones.

Another mechanism is needed to delete or ignore initrd
connections. One possibility is that nm-initrd-generator adds a key to
the 'user' setting of generated connection to indicate the source the
connection is 'initrd'. In NM then there is a new property to ignore
such connections.

But perhaps we don't need this complexity, and it's enough to have a
configuration option to wipe /run/NetworkManager/system-connections at
first start in real root.

The advantage of configuration options compared to e.g a systemd
service is that in the future the NetworkManager initrd module could
also write those options based on some kernel command line.

Hi Cristophe,

connections created in the initrd from kernel command line can't
be modified because they are ephemeral and re-generated every time.

To support having a network configuration in the real root different
from the one in the initrd, we plan to add a couple of new
NetworkManager configuration properties to be added to
/etc/NetworkManager/NetworkManager.conf:

 [device-enp1s0]
 match-device=interface-name:enp1s0
 keep-configuration=no                                           (1)
 allowed-connections=except:origin:nm-initrd-generator           (2)

(1) is to ignore any existing configuration on device 'enp1s0' when NM
starts. That is, to not create an in-memory connection based on IP
addresses present on the interface;

(2) tells NM to ignore connections created by the initrd-generator from
the kernel command line for device 'enp1s0'.

In this way, users only need to create a persistent connection with
nmcli and it will be honored in the real root, even if the interface
was configured differently in initrd.

Does this solve the customer's problem?

Hi Beniamino,

yes it sounds like a good approach, which should solve both cases attached to this BZ.
It should also work in the fedora case mentioned by Renaud in c#12.

I'll be able to check the behavior of a test package in my environment if needed.

+1

(In reply to Christophe Besson from comment #25)
> I'll be able to check the behavior of a test package in my environment if
> needed.

For which RHEL version should I build it? Is 8.5 ok?

I will add a link to the RHEL 8.5 build once it's available.

NetworkManager-1.32.6-1.el8 supports the 2 new configuration options.

Added tests, that NM prefers persistent profiles from rootfs for configured interfaces (both keyfile and ifcfg format).

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: NetworkManager security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4361