Bug 2031385

Could you please provide us logs from the installation environment? You can find them in /tmp/*log during the installation or /var/log/anaconda in the installed system.

Created attachment 1850785 [details]
install logs

Logs from install

So it seems that there if we have these three connections there is no deterministic expectation for which of br0 or ens3 would be activated and configured by NM on boot:

[connection]
id=br0
uuid=c8834cc9-84e2-4ff0-9fa8-73d485b9062c
type=bridge
interface-name=br0

[bridge]
stp=false

[ipv4]
method=auto

[ipv6]
addr-gen-mode=eui64
method=auto

[proxy]
[connection]
id=br0_slave_1
uuid=c96618d4-b9f6-4148-9cfb-83842da0aea0
type=ethernet
interface-name=ens3
master=br0
slave-type=bridge

[ethernet]

[bridge-port]
[connection]
id=ens3
uuid=d271cc67-3024-4a5e-a1c9-5f98310cc59f
type=ethernet
interface-name=ens3

[ethernet]

[ipv4]
method=auto

[ipv6]
addr-gen-mode=eui64
method=auto

[proxy]

We have two cases in this BZ I was able to reproduce consistently where the keyfiles differ only in UUIDs while the device configured by dhcp differ:

1) bridge defined in kickstart

network --device=br0 --bridgeslaves=ens3 --bridgeopts=stp=no --bootproto=dhcp --onboot=on

In this case br0 device is configured with dhcp after boot.

2) bridge defined in kickstart via including configuration generated in %pre section

%include /tmp/ksinclude
%pre
echo network --device=br0 --bridgeslaves=ens3 --bridgeopts=stp=no --bootproto=dhcp --onboot=on >> /tmp/ksinclude
%end

In this case ens3 is configured with dhcp after boot.

I'll attach logs for both cases in following comments.

Beniamino, is this behavior expected from the NM point of view ?
If so, we'll perhaps need to stop creating the ens3 connection, or make it autoconnect=False (or set autoactivation priorities in Anaconda).

(In reply to Radek Vykydal from comment #4)
>
> Beniamino, is this behavior expected from the NM point of view ?

If two connections have autoconnect enabled and the same autoconnect-priority, then the one with a more recent last-activation timestamp gets selected. In case of tie (as when none of them was activated before), there is alphabetical comparison between the UUIDs.

So, in case of generated connections where the UUID changes every time, yes, I think this behavior is expected.

(In reply to Chris Adams from comment #0)

> This works for install (I get br0 and a physical device as expected), but at
> the end of setup, anaconda generates 3 connections for NetworkManager - the
> bridge, the bridge slave, and the base interface (duplicating the bridge
> slave, except set to DHCP). This causes NetworkManager to start the base
> interface instead of the bridge on boot.
> 
> If I replace the above %pre/%include with the same network line as what's
> generated, this doesn't happen; anaconda just generates the expected 2
> connections in NM.

As explained in comment #4 Anaconda actually creates the same (3) connections in both cases, but in the %pre case for some reason ens3 wins over br0 (see comment #7).

Anaconda creates default connections for all ethernet devices for legacy reasons, but in this case the connection for ens3 should have autoconnect=False set. This can be done (1) by this patch:

https://github.com/rvykydal/anaconda/commit/b3ffea53fe5455941c9a48b540c8be272ea4e2e0

(2) More thorough solution that may be applied in the future would be to stop creating default configuration files for devices which was not explicitly configured (the ethernet device in this case which was activated in intramfs to fetch installer image, but was not explicitly configured). Tracked in https://issues.redhat.com/browse/INSTALLER-1026

(3) The issue can be also resolved in kickstart configuration by configuring the wired device with onboot=no explicitly, for the case from the description:

%pre --erroronfail --interpreter=/bin/bash
for dpath in /sys/class/net/*; do
        [ $(cat $dpath/carrier) = 1 -a $(cat $dpath/type) = 1 ] || continue
        echo "network --device=br0 --bridgeslave=${dpath##*/} --bridgeopts=stp=no --bootproto=dhcp --onboot=on" > /tmp/ks-include-net
        echo "network --device=${dpath##*/} --bootproto=dhcp --onboot=off" >> /tmp/ks-include-net
        break
done
%end
%include /tmp/ks-include-net

I don't think this is a problem with NM at all, it's an Anaconda bug purely related to putting networking config in %include. If I use a kickstart that has the network config directly in the KS file, anaconda does not generate the extraneous interface config. So I don't think it's the case that "Anaconda creates default connections for all ethernet devices for legacy reasons", it's purely an issue when the network config comes from an %include section.

(In reply to Chris Adams from comment #9)
> I don't think this is a problem with NM at all, it's an Anaconda bug purely
> related to putting networking config in %include. If I use a kickstart that
> has the network config directly in the KS file, anaconda does not generate
> the extraneous interface config. So I don't think it's the case that
> "Anaconda creates default connections for all ethernet devices for legacy
> reasons", it's purely an issue when the network config comes from an
> %include section.

That seems strange to me. Both cases should go through the same code path in RHEL 9.
In my reproducer Anaconda creates the third interface config (/etc/NetworkManager/system-connections/enp3s0.nmconnection in your case) also in case of direct network configuration but I may be missing something. Could you please attach the logs from /tmp (same as in comment #2) also for the case with direct network configuration?

I just re-confirmed this (with CentOS 9-stream). I ran my kickstart twice - the first time with the network config generated in the %pre from the first comment, the second time with that commented out and its output just put directly in the kickstart:

network --device=br0 --bridgeslave=enp3s0 --bridgeopts=stp=no --bootproto=dhcp --onboot=on

The run with the %pre-config generated three NM configs, while the run with the directly-set config generated only two NM configs (as desired).

I've run into a couple of other issues with things generated in a %pre that I think were related to how %includes get handled (like a second run through some handled bits or something along those lines?).

(In reply to Chris Adams from comment #11)
> I just re-confirmed this (with CentOS 9-stream). I ran my kickstart twice -
> the first time with the network config generated in the %pre from the first
> comment, the second time with that commented out and its output just put
> directly in the kickstart:
> 
> network --device=br0 --bridgeslave=enp3s0 --bridgeopts=stp=no
> --bootproto=dhcp --onboot=on
> 
> The run with the %pre-config generated three NM configs, while the run with
> the directly-set config generated only two NM configs (as desired).

Could you still please add the logs for this run (two NM configs), I wonder where is the difference from my reproducer. Maybe it is inst.ks=hd: vs inst.ks=http: in my case. Perhaps the network configuration was applied already in initramfs for inst.ks=hd (I mean for the regular network config, the %pre section is processed only after switch root). There are some limited cases when network can be applied already in initramfs (only single network command). 

> 
> I've run into a couple of other issues with things generated in a %pre that
> I think were related to how %includes get handled (like a second run through
> some handled bits or something along those lines?).

This could point to network being applied already in initramfs or in general in processing of kickstart in initramfs.

Created attachment 1871519 [details]
install logs for non-%pre-generated config

Logs from a run with a kickstart that just has the single bridge network line in the KS, not generated from %pre. This generated only the two expected NM connections.

(In reply to Chris Adams from comment #13)
> Created attachment 1871519 [details]
> install logs for non-%pre-generated config
> 
> Logs from a run with a kickstart that just has the single bridge network
> line in the KS, not generated from %pre. This generated only the two
> expected NM connections.

Thank you for the logs. The difference from my reproducer is really inst.ks coming from hd: which causes activation of br0 already in initramfs.

The %pre section is processed after switch root so in that case a wired device is activated in intramfs to fetch installer image and the bridge is created after anaconda switch root.
The fix of the %pre case (and also the regular case where kickstart is not fetched from hd:) would be one of the solutions in comment #8.

After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.