Created attachment 1892499 [details] Installer image packages from failing daily Anaconda boot.iso Description of problem: Update of a connection created in initramfs by kickstart with network --onboot value is not working. It corresponds to failing onboot-activate kickstart test. Version-Release number of selected component (if applicable): anaconda-37.10-3.fc37 It started to fail with this kstests daily run (daily-iso scenario): https://github.com/rhinstaller/kickstart-tests/actions/runs/2552549057 The last known working daily kstests run from the day before (https://github.com/rhinstaller/kickstart-tests/actions/runs/2545726883) has anaconda-37.10-1.20220621173531791046.master.58.gc163c3c908.fc37.x86_64 How reproducible: Always Steps to Reproduce: 1. Run PXE installation with ip=ens3:dhcp option and kickstart network command network --device ens3 --onboot no 2. 3. Actual results: The /etc/NetworkManager/system-connections/ens3.nmconnection does not have value of [connection].autoconnect set (it is True by default) Expected results: The /etc/NetworkManager/system-connections/ens3.nmconnection has value of [connection].autoconnect set to False Additional info: The onboot-activate kickstart test is failing also in this PR (kickstart-test --testtype coverage): https://github.com/rhinstaller/anaconda/pull/4207 https://github.com/rhinstaller/anaconda/actions/runs/2550139098 It can be related to the way network connection is created in initramfs, but I need to investigate more.
Adam, I still need to investigate the issue, but I was surprised by the diff making the python packages and anaconda with the initramfs python patch the most susceptible candidates to me (while I was expecting NM). So I am just making you aware of this BZ (and maybe also of https://bugzilla.redhat.com/show_bug.cgi?id=2100741 - perhaps initramfs issue as well).
I wonder if this and #2101527 have the same cause?
Created attachment 1893170 [details] Journal from reproducer with added DDDDD debug messages I narrowed the issue down to storing connection created in initramfs as persistent after switch root by Anaconda. The case described here is boot options configuration ip=ens3:dhcp and kickstart configuration: network --device ens3 --onboot no which is supposed to set autoconnect of ens3 to False. The system has one more device ens10. After initramfs there are these configuration files (created in intramfs) on the system: /run/NetworkManager/system-connections/Wired connection 1.nmconnection /run/NetworkManager/system-connections/ens3.nmconnection Anaconda applies the kickstart configuration on the ens3 connection, making the connection persistent in https://github.com/rhinstaller/anaconda/blob/16bcf2f019745b52e6c95ee0ab35da727b4de020/pyanaconda/network.py#L241 running https://github.com/rhinstaller/anaconda/blob/16bcf2f019745b52e6c95ee0ab35da727b4de020/pyanaconda/modules/network/initialization.py#L73 calling NM update2 method, making the connection persistent After this task the configuration files are (ens3 made persistent): /run/NetworkManager/system-connections/Wired connection 1.nmconnection /etc/NetworkManager/system-connections/ens3.nmconnection In the next task https://github.com/rhinstaller/anaconda/blob/16bcf2f019745b52e6c95ee0ab35da727b4de020/pyanaconda/network.py#L242 https://github.com/rhinstaller/anaconda/blob/16bcf2f019745b52e6c95ee0ab35da727b4de020/pyanaconda/modules/network/initialization.py#L213 Anaconda is looking at persistent configuration of devices: https://github.com/rhinstaller/anaconda/blob/16bcf2f019745b52e6c95ee0ab35da727b4de020/pyanaconda/modules/network/initialization.py#L231 The get_config_file_connection_of_device function started to behave differently. It is calling NMRemoteConnection.get_filename() for the connections obtained by NMClient.get_connections() and as can be seen in the attached log (look for the DDDDD string in added debug messages) it is getting: /run/NetworkManager/system-connections/Wired connection 1.nmconnection /run/NetworkManager/system-connections/ens3.nmconnection ie the files from the state before the application of kickstart. while formerly it was getting (I'll attach the log in the next comment) /run/NetworkManager/system-connections/Wired connection 1.nmconnection /etc/NetworkManager/system-connections/ens3.nmconnection as expected. Creating new configuration for ens3 with default autoconnect (True) is a consequence of not finding existing persistent configuration (/etc/NetworkManager/system-connections/ens3.nmconnection). Here is the package diff for the working and non-working case (package file lists from Description and comment #1): 11,17c11,17 < anaconda-37.10-3.fc37.x86_64 < anaconda-core-37.10-3.fc37.x86_64 < anaconda-dracut-37.10-3.fc37.x86_64 < anaconda-gui-37.10-3.fc37.x86_64 < anaconda-install-env-deps-37.10-3.fc37.x86_64 < anaconda-install-img-deps-37.10-3.fc37.x86_64 < anaconda-tui-37.10-3.fc37.x86_64 --- > anaconda-37.10-1.20220621173531791046.master.58.gc163c3c908.fc37.x86_64 > anaconda-core-37.10-1.20220621173531791046.master.58.gc163c3c908.fc37.x86_64 > anaconda-dracut-37.10-1.20220621173531791046.master.58.gc163c3c908.fc37.x86_64 > anaconda-gui-37.10-1.20220621173531791046.master.58.gc163c3c908.fc37.x86_64 > anaconda-install-env-deps-37.10-1.20220621173531791046.master.58.gc163c3c908.fc37.x86_64 > anaconda-install-img-deps-37.10-1.20220621173531791046.master.58.gc163c3c908.fc37.x86_64 > anaconda-tui-37.10-1.20220621173531791046.master.58.gc163c3c908.fc37.x86_64 19c19 < anaconda-widgets-37.10-3.fc37.x86_64 --- > anaconda-widgets-37.10-1.20220621173531791046.master.58.gc163c3c908.fc37.x86_64 25,26c25,26 < audit-3.0.8-2.fc37.x86_64 < audit-libs-3.0.8-2.fc37.x86_64 --- > audit-3.0.8-1.fc37.x86_64 > audit-libs-3.0.8-1.fc37.x86_64 37,39c37,39 < bind-libs-9.16.30-1.fc37.x86_64 < bind-license-9.16.30-1.fc37.noarch < bind-utils-9.16.30-1.fc37.x86_64 --- > bind-libs-9.16.29-1.fc37.x86_64 > bind-license-9.16.29-1.fc37.noarch > bind-utils-9.16.29-1.fc37.x86_64 42,43c42,43 < blivet-data-3.4.4-3.fc37.noarch < blivet-gui-runtime-2.3.0-6.fc37.noarch --- > blivet-data-3.4.4-2.20220517gitb4e0356b.fc37.noarch > blivet-gui-runtime-2.3.0-57.20220621git57e33a0.fc37.noarch 45c45 < boost-regex-1.78.0-4.fc37.x86_64 --- > boost-regex-1.78.0-2.fc37.x86_64 47,49c47,49 < brlapi-0.8.4-4.fc37.x86_64 < brltty-6.5-4.fc37.x86_64 < btrfs-progs-5.18-2.fc37.x86_64 --- > brlapi-0.8.4-1.fc37.x86_64 > brltty-6.5-1.fc37.x86_64 > btrfs-progs-5.18-1.fc37.x86_64 68,69c68,69 < createrepo_c-0.20.0-20220623010005.4.gd1d12dd.fc37.x86_64 < createrepo_c-libs-0.20.0-20220623010005.4.gd1d12dd.fc37.x86_64 --- > createrepo_c-0.20.0-20220621010446.3.gc61caaa.fc37.x86_64 > createrepo_c-libs-0.20.0-20220621010446.3.gc61caaa.fc37.x86_64 88,91c88,91 < device-mapper-1.02.175-8.fc37.x86_64 < device-mapper-event-1.02.175-8.fc37.x86_64 < device-mapper-event-libs-1.02.175-8.fc37.x86_64 < device-mapper-libs-1.02.175-8.fc37.x86_64 --- > device-mapper-1.02.175-7.fc36.x86_64 > device-mapper-event-1.02.175-7.fc36.x86_64 > device-mapper-event-libs-1.02.175-7.fc36.x86_64 > device-mapper-libs-1.02.175-7.fc36.x86_64 99,100c99,100 < dnf-4.13.0-20220623010007.0.gef09e48f.fc37.noarch < dnf-data-4.13.0-20220623010007.0.gef09e48f.fc37.noarch --- > dnf-4.13.0-20220621010448.0.gef09e48f.fc37.noarch > dnf-data-4.13.0-20220621010448.0.gef09e48f.fc37.noarch 107c107 < drpm-0.5.1-20220623010001.1.g39a106e.fc37.x86_64 --- > drpm-0.5.1-20220621010441.1.g39a106e.fc37.x86_64 130,131c130,131 < file-5.41-6.fc37.x86_64 < file-libs-5.41-6.fc37.x86_64 --- > file-5.41-5.fc37.x86_64 > file-libs-5.41-5.fc37.x86_64 150,152c150,152 < gdb-12.1-3.fc37.x86_64 < gdb-gdbserver-12.1-3.fc37.x86_64 < gdb-headless-12.1-3.fc37.x86_64 --- > gdb-12.1-1.fc37.x86_64 > gdb-gdbserver-12.1-1.fc37.x86_64 > gdb-headless-12.1-1.fc37.x86_64 163c163 < glib2-2.73.1-1.fc37.x86_64 --- > glib2-2.72.2-1.fc37.x86_64 174,175c174,175 < gnutls-3.7.6-4.fc37.x86_64 < gobject-introspection-1.72.0-2.fc37.x86_64 --- > gnutls-3.7.6-1.fc37.x86_64 > gobject-introspection-1.72.0-1.fc37.x86_64 187c187 < gpgme-1.17.0-3.fc37.x86_64 --- > gpgme-1.17.0-2.fc37.x86_64 201c201 < grubby-8.40-61.fc37.x86_64 --- > grubby-8.40-60.fc37.x86_64 223c223 < ibus-libs-1.5.26-10.fc37.x86_64 --- > ibus-libs-1.5.26-9.fc37.x86_64 234,235c234,235 < iscsi-initiator-utils-6.2.1.4-5.git2a8f9d8.fc37.x86_64 < iscsi-initiator-utils-iscsiuio-6.2.1.4-5.git2a8f9d8.fc37.x86_64 --- > iscsi-initiator-utils-6.2.1.4-4.git2a8f9d8.fc36.x86_64 > iscsi-initiator-utils-iscsiuio-6.2.1.4-4.git2a8f9d8.fc36.x86_64 238c238 < isomd5sum-1.2.3-16.fc37.x86_64 --- > isomd5sum-1.2.3-15.fc36.x86_64 264,265c264,265 < kbd-2.5.1-1.fc37.x86_64 < kbd-misc-2.5.1-1.fc37.noarch --- > kbd-2.5.0-1.fc37.x86_64 > kbd-misc-2.5.0-1.fc37.noarch 267,270c267,270 < kernel-5.19.0-0.rc3.20220621git78ca55889a54.28.fc37.x86_64 < kernel-core-5.19.0-0.rc3.20220621git78ca55889a54.28.fc37.x86_64 < kernel-modules-5.19.0-0.rc3.20220621git78ca55889a54.28.fc37.x86_64 < kernel-modules-extra-5.19.0-0.rc3.20220621git78ca55889a54.28.fc37.x86_64 --- > kernel-5.19.0-0.rc2.20220617git47700948a4ab.24.fc37.x86_64 > kernel-core-5.19.0-0.rc2.20220617git47700948a4ab.24.fc37.x86_64 > kernel-modules-5.19.0-0.rc2.20220617git47700948a4ab.24.fc37.x86_64 > kernel-modules-extra-5.19.0-0.rc2.20220617git47700948a4ab.24.fc37.x86_64 281c281 < langtable-0.0.58-2.fc37.noarch --- > langtable-0.0.58-1.fc37.noarch 317c317 < libbabeltrace-1.5.8-11.fc37.x86_64 --- > libbabeltrace-1.5.8-9.fc36.x86_64 319,327c319,327 < libblkid-2.38-4.fc37.x86_64 < libblockdev-2.27-2.fc37.x86_64 < libblockdev-btrfs-2.27-2.fc37.x86_64 < libblockdev-crypto-2.27-2.fc37.x86_64 < libblockdev-dm-2.27-2.fc37.x86_64 < libblockdev-fs-2.27-2.fc37.x86_64 < libblockdev-kbd-2.27-2.fc37.x86_64 < libblockdev-loop-2.27-2.fc37.x86_64 < libblockdev-lvm-2.27-2.fc37.x86_64 --- > libblkid-2.38-3.fc37.x86_64 > libblockdev-2.99-130.20220607git73eafd0.fc37.x86_64 > libblockdev-btrfs-2.99-130.20220607git73eafd0.fc37.x86_64 > libblockdev-crypto-2.99-130.20220607git73eafd0.fc37.x86_64 > libblockdev-dm-2.99-130.20220607git73eafd0.fc37.x86_64 > libblockdev-fs-2.99-130.20220607git73eafd0.fc37.x86_64 > libblockdev-kbd-2.99-130.20220607git73eafd0.fc37.x86_64 > libblockdev-loop-2.99-130.20220607git73eafd0.fc37.x86_64 > libblockdev-lvm-2.99-130.20220607git73eafd0.fc37.x86_64 329,335c329,336 < libblockdev-mdraid-2.27-2.fc37.x86_64 < libblockdev-mpath-2.27-2.fc37.x86_64 < libblockdev-nvdimm-2.27-2.fc37.x86_64 < libblockdev-part-2.27-2.fc37.x86_64 < libblockdev-plugins-all-2.27-2.fc37.x86_64 < libblockdev-swap-2.27-2.fc37.x86_64 < libblockdev-utils-2.27-2.fc37.x86_64 --- > libblockdev-mdraid-2.99-130.20220607git73eafd0.fc37.x86_64 > libblockdev-mpath-2.99-130.20220607git73eafd0.fc37.x86_64 > libblockdev-nvdimm-2.99-130.20220607git73eafd0.fc37.x86_64 > libblockdev-nvme-2.99-130.20220607git73eafd0.fc37.x86_64 > libblockdev-part-2.99-130.20220607git73eafd0.fc37.x86_64 > libblockdev-plugins-all-2.99-130.20220607git73eafd0.fc37.x86_64 > libblockdev-swap-2.99-130.20220607git73eafd0.fc37.x86_64 > libblockdev-utils-2.99-130.20220607git73eafd0.fc37.x86_64 337,338c338,339 < libbrotli-1.0.9-8.fc37.x86_64 < libbytesize-2.7-2.fc37.x86_64 --- > libbrotli-1.0.9-7.fc36.x86_64 > libbytesize-2.7-2.20220526gitd6dae10.fc37.x86_64 342c343 < libcap-ng-0.8.3-2.fc37.x86_64 --- > libcap-ng-0.8.3-1.fc37.x86_64 347c348 < libcomps-0.1.18-20220623010002.1.gae903ff.fc37.x86_64 --- > libcomps-0.1.18-20220621010442.1.gae903ff.fc37.x86_64 353c354 < libdnf-0.67.0-20220623010006.8.geff7e688.fc37.x86_64 --- > libdnf-0.67.0-20220621010447.7.g9f0c0d20.fc37.x86_64 367c368 < libfdisk-2.38-4.fc37.x86_64 --- > libfdisk-2.38-3.fc37.x86_64 386c387 < libibverbs-39.0-2.fc37.x86_64 --- > libibverbs-39.0-1.fc36.x86_64 401,402c402,403 < libmodulemd-2.14.0-20220623010000.13.ga0741ff.fc37.x86_64 < libmount-2.38-4.fc37.x86_64 --- > libmodulemd-2.14.0-20220621010440.13.ga0741ff.fc37.x86_64 > libmount-2.38-3.fc37.x86_64 406,407c407,408 < libnl3-3.6.0-4.fc37.x86_64 < libnl3-cli-3.6.0-4.fc37.x86_64 --- > libnl3-3.6.0-3.fc37.x86_64 > libnl3-cli-3.6.0-3.fc37.x86_64 410a412 > libnvme-1.0-1.fc37.x86_64 416c418 < libproxy-0.4.18-1.fc37.x86_64 --- > libproxy-0.4.17-5.fc37.x86_64 418c420 < libpwquality-1.4.4-10.fc37.x86_64 --- > libpwquality-1.4.4-7.fc36.x86_64 420,428c422,430 < librepo-1.14.3-20220623010001.2.g6985f3f.fc37.x86_64 < libreport-2.17.1-2.fc37.x86_64 < libreport-anaconda-2.17.1-2.fc37.x86_64 < libreport-cli-2.17.1-2.fc37.x86_64 < libreport-filesystem-2.17.1-2.fc37.noarch < libreport-gtk-2.17.1-2.fc37.x86_64 < libreport-plugin-bugzilla-2.17.1-2.fc37.x86_64 < libreport-plugin-reportuploader-2.17.1-2.fc37.x86_64 < libreport-web-2.17.1-2.fc37.x86_64 --- > librepo-1.14.3-20220621010441.2.g6985f3f.fc37.x86_64 > libreport-2.17.1-1.fc37.x86_64 > libreport-anaconda-2.17.1-1.fc37.x86_64 > libreport-cli-2.17.1-1.fc37.x86_64 > libreport-filesystem-2.17.1-1.fc37.noarch > libreport-gtk-2.17.1-1.fc37.x86_64 > libreport-plugin-bugzilla-2.17.1-1.fc37.x86_64 > libreport-plugin-reportuploader-2.17.1-1.fc37.x86_64 > libreport-web-2.17.1-1.fc37.x86_64 433,435c435,437 < libselinux-3.4-4.fc37.x86_64 < libselinux-utils-3.4-4.fc37.x86_64 < libsemanage-3.4-3.fc37.x86_64 --- > libselinux-3.4-3.fc37.x86_64 > libselinux-utils-3.4-3.fc37.x86_64 > libsemanage-3.4-2.fc37.x86_64 438c440 < libsmartcols-2.38-4.fc37.x86_64 --- > libsmartcols-2.38-3.fc37.x86_64 440c442 < libsolv-0.7.22-20220623005957.1.g5e1ac6aa.fc37.x86_64 --- > libsolv-0.7.22-20220621010437.1.g5e1ac6aa.fc37.x86_64 449c451 < libtdb-1.4.7-2.fc37.x86_64 --- > libtdb-1.4.7-1.fc37.x86_64 463c465 < libuuid-2.38-4.fc37.x86_64 --- > libuuid-2.38-3.fc37.x86_64 469,470c471,472 < libwacom-2.3.0-1.fc37.x86_64 < libwacom-data-2.3.0-1.fc37.noarch --- > libwacom-2.2.0-1.fc37.x86_64 > libwacom-data-2.2.0-1.fc37.noarch 483c485 < libxml2-2.9.14-2.fc37.x86_64 --- > libxml2-2.9.14-1.fc37.x86_64 488c490 < lilv-libs-0.24.14-2.fc37.x86_64 --- > lilv-libs-0.24.14-1.fc37.x86_64 505,507c507,509 < lvm2-2.03.11-8.fc37.x86_64 < lvm2-dbusd-2.03.11-8.fc37.noarch < lvm2-libs-2.03.11-8.fc37.x86_64 --- > lvm2-2.03.11-7.fc36.x86_64 > lvm2-dbusd-2.03.11-7.fc36.noarch > lvm2-libs-2.03.11-7.fc36.x86_64 571c573 < parted-3.5-3.fc37.x86_64 --- > parted-3.5-2.fc37.x86_64 586,587c588,589 < policycoreutils-3.4-3.fc37.x86_64 < policycoreutils-python-utils-3.4-3.fc37.noarch --- > policycoreutils-3.4-2.fc37.x86_64 > policycoreutils-python-utils-3.4-2.fc37.noarch 599,651c601,653 < python-pip-wheel-22.0.4-4.fc37.noarch < python-setuptools-wheel-60.9.3-5.fc37.noarch < python3-3.11.0~b3-4.fc37.x86_64 < python3-audit-3.0.8-2.fc37.x86_64 < python3-blivet-3.4.4-3.fc37.noarch < python3-blockdev-2.27-2.fc37.x86_64 < python3-bytesize-2.7-2.fc37.x86_64 < python3-cairo-1.21.0-2.fc37.x86_64 < python3-cffi-1.15.0-6.fc37.x86_64 < python3-charset-normalizer-2.1.0-1.fc37.noarch < python3-cryptography-37.0.2-3.fc37.x86_64 < python3-dasbus-1.6-5.fc37.noarch < python3-dbus-1.2.18-4.fc37.x86_64 < python3-dnf-4.13.0-20220623010007.0.gef09e48f.fc37.noarch < python3-gobject-3.42.1-2.fc37.x86_64 < python3-gobject-base-3.42.1-2.fc37.x86_64 < python3-gobject-base-noarch-3.42.1-2.fc37.noarch < python3-gpg-1.17.0-3.fc37.x86_64 < python3-hawkey-0.67.0-20220623010006.8.geff7e688.fc37.x86_64 < python3-idna-3.3-3.fc37.noarch < python3-kickstart-3.41-2.fc37.noarch < python3-langtable-0.0.58-2.fc37.noarch < python3-libcomps-0.1.18-20220623010002.1.gae903ff.fc37.x86_64 < python3-libdnf-0.67.0-20220623010006.8.geff7e688.fc37.x86_64 < python3-libreport-2.17.1-2.fc37.x86_64 < python3-libs-3.11.0~b3-4.fc37.x86_64 < python3-libselinux-3.4-4.fc37.x86_64 < python3-libsemanage-3.4-3.fc37.x86_64 < python3-meh-0.50-6.fc37.noarch < python3-meh-gui-0.50-6.fc37.noarch < python3-packaging-21.3-4.fc37.noarch < python3-pid-2.2.3-14.fc37.noarch < python3-ply-3.11-16.fc37.noarch < python3-policycoreutils-3.4-3.fc37.noarch < python3-productmd-1.33-5.fc37.noarch < python3-pwquality-1.4.4-10.fc37.x86_64 < python3-pyatspi-2.38.2-3.fc37.noarch < python3-pycparser-2.20-7.fc37.noarch < python3-pyparsing-2.4.7-12.fc37.noarch < python3-pyparted-3.12.0-5.fc37.x86_64 < python3-pysocks-1.7.1-14.fc37.noarch < python3-pyudev-0.23.2-2.fc37.noarch < python3-requests-2.27.1-5.fc37.noarch < python3-requests-file-1.5.1-6.fc37.noarch < python3-requests-ftp-0.3.1-25.fc37.noarch < python3-rpm-4.18.0-0.alpha2.2.fc37.x86_64 < python3-setools-4.4.0-8.fc37.x86_64 < python3-setuptools-60.9.3-5.fc37.noarch < python3-simpleline-1.9.0-3.fc37.noarch < python3-six-1.16.0-7.fc37.noarch < python3-systemd-234-21.fc37.x86_64 < python3-urllib3-1.26.9-3.fc37.noarch < python3-virt-firmware-1.0-1.fc37.noarch --- > python-pip-wheel-22.0.4-2.fc37.noarch > python-setuptools-wheel-60.9.3-2.fc37.noarch > python3-3.10.5-2.fc37.x86_64 > python3-audit-3.0.8-1.fc37.x86_64 > python3-blivet-3.4.4-2.20220517gitb4e0356b.fc37.noarch > python3-blockdev-2.99-130.20220607git73eafd0.fc37.x86_64 > python3-bytesize-2.7-2.20220526gitd6dae10.fc37.x86_64 > python3-cairo-1.21.0-1.fc37.x86_64 > python3-cffi-1.15.0-5.fc37.x86_64 > python3-charset-normalizer-2.0.12-1.fc37.noarch > python3-cryptography-37.0.2-1.fc37.x86_64 > python3-dasbus-1.6-4.fc36.noarch > python3-dbus-1.2.18-3.fc36.x86_64 > python3-dnf-4.13.0-20220621010448.0.gef09e48f.fc37.noarch > python3-gobject-3.42.1-1.fc37.x86_64 > python3-gobject-base-3.42.1-1.fc37.x86_64 > python3-gobject-base-noarch-3.42.1-1.fc37.noarch > python3-gpg-1.17.0-2.fc37.x86_64 > python3-hawkey-0.67.0-20220621010447.7.g9f0c0d20.fc37.x86_64 > python3-idna-3.3-2.fc36.noarch > python3-kickstart-3.41-1.fc37.noarch > python3-langtable-0.0.58-1.fc37.noarch > python3-libcomps-0.1.18-20220621010442.1.gae903ff.fc37.x86_64 > python3-libdnf-0.67.0-20220621010447.7.g9f0c0d20.fc37.x86_64 > python3-libreport-2.17.1-1.fc37.x86_64 > python3-libs-3.10.5-2.fc37.x86_64 > python3-libselinux-3.4-3.fc37.x86_64 > python3-libsemanage-3.4-2.fc37.x86_64 > python3-meh-0.50-5.fc36.noarch > python3-meh-gui-0.50-5.fc36.noarch > python3-packaging-21.3-2.fc36.noarch > python3-pid-2.2.3-13.fc36.noarch > python3-ply-3.11-15.fc36.noarch > python3-policycoreutils-3.4-2.fc37.noarch > python3-productmd-1.33-4.fc36.noarch > python3-pwquality-1.4.4-7.fc36.x86_64 > python3-pyatspi-2.38.2-2.fc36.noarch > python3-pycparser-2.20-6.fc36.noarch > python3-pyparsing-2.4.7-10.fc36.noarch > python3-pyparted-3.12.0-2.fc37.x86_64 > python3-pysocks-1.7.1-12.fc36.noarch > python3-pyudev-0.23.2-1.fc37.noarch > python3-requests-2.27.1-2.fc36.noarch > python3-requests-file-1.5.1-5.fc36.noarch > python3-requests-ftp-0.3.1-24.fc36.noarch > python3-rpm-4.18.0-0.alpha2.1.fc37.x86_64 > python3-setools-4.4.0-7.fc37.x86_64 > python3-setuptools-60.9.3-2.fc37.noarch > python3-simpleline-1.9.0-2.fc36.noarch > python3-six-1.16.0-5.fc36.noarch > python3-systemd-234-20.fc36.x86_64 > python3-urllib3-1.26.9-1.fc37.noarch > python3-virt-firmware-0.98-1.fc37.noarch 654c656 < rdma-core-39.0-2.fc37.x86_64 --- > rdma-core-39.0-1.fc36.x86_64 661,663c663,665 < rpm-4.18.0-0.alpha2.2.fc37.x86_64 < rpm-build-libs-4.18.0-0.alpha2.2.fc37.x86_64 < rpm-libs-4.18.0-0.alpha2.2.fc37.x86_64 --- > rpm-4.18.0-0.alpha2.1.fc37.x86_64 > rpm-build-libs-4.18.0-0.alpha2.1.fc37.x86_64 > rpm-libs-4.18.0-0.alpha2.1.fc37.x86_64 666,667c668,669 < rpm-plugin-selinux-4.18.0-0.alpha2.2.fc37.x86_64 < rpm-sign-libs-4.18.0-0.alpha2.2.fc37.x86_64 --- > rpm-plugin-selinux-4.18.0-0.alpha2.1.fc37.x86_64 > rpm-sign-libs-4.18.0-0.alpha2.1.fc37.x86_64 672c674 < satyr-0.39-5.fc37.x86_64 --- > satyr-0.39-4.fc36.x86_64 697c699 < system-storage-manager-1.3-13.fc37.noarch --- > system-storage-manager-1.3-12.fc36.noarch 719,720c721,722 < util-linux-2.38-4.fc37.x86_64 < util-linux-core-2.38-4.fc37.x86_64 --- > util-linux-2.38-3.fc37.x86_64 > util-linux-core-2.38-3.fc37.x86_64 724,725c726,727 < volume_key-0.3.12-16.fc37.x86_64 < volume_key-libs-0.3.12-16.fc37.x86_64 --- > volume_key-0.3.12-15.fc36.x86_64 > volume_key-libs-0.3.12-15.fc36.x86_64 Maybe the behavior change is related to Python update ? Or maybe we've been doing something wrong handling the connetions in Anaconda and just hit some race ?
Created attachment 1893171 [details] Journal from working case with added DDDDD debug messages
Thomas, do you can you please look at comment #4 ? We seem to be failing to get filenams of a NMRemoteConnection correctly. If there is nothing apparent, I can try to create a reproducer on a regular installed system, but I am rather asking here first. Thank you.
So a few observations: * We call the client's `get_devices()` early in `initialize_network`, when we log "Devices found:". `get_supported_devices()` ultimately calls it, over in pyanaconda/modules/network/network.py, through a few layers of indirection. This call is *before* the `ApplyKickstartTask` task runs. * We then call it again in `DumpMissingConfigFilesTask`, the thing that's going wrong. It's what we use to get the list of devices we look for the config files of. We are using the same NM client instance for both these calls. * `get_devices()` does have at least *some* level of caching. If you write a simple test script that calls it twice in succession and do an identity check on the results, they are the same object. * The NM client has a `reload_connections()` method (and an async version which is now preferred): https://networkmanager.dev/docs/libnm/latest/NMClient.html#nm-client-reload-connections So...maybe we should be using that here? I will try and reproduce the bug and see if a call to `reload_connections()` before we get the connection list for the second time helps.
That doesn't seem to help. Darn.
I don't actually know what the problem is. There are no known bugs in libnm or known regressions that would cause this. I think there is a threading problem. --- Sorry for the verbosity... feel free to ignore my analysis :) Notes About NMClient, GMainContext and D-Bus -------------------------------------------- `nm_client_reload_connections()` basically calls the `ReloadConnections` D-Bus method (asynchronously). That would load profiles from disk, but that would only be necessary if you modify the files on disk directly. So that would not matter here. Libnm's central object is the NMClient. This basically caches objects from the D-Bus API. And... it only updates while running/iterating the GMainContext, at which point it also can emit any events. Aside from being called by the main context (aka "mainloop") when it iterates, it is not actively doing anything. When you make an async method calls like `nm_remote_connection_update2()`, you basically send a D-Bus message (and register a callback for the response), but that doesn't change the cache's state. Note that if you call any of libnm's async methods (like `nm_remote_connection_update2()` and the `_finish()` part), then clearly you are already iterating the main context. Because that is how async methods in glib work, being "driven" by the GMainContext. If the async call completes, you get a callback from the GMainContext at which point you'd call `nm_remote_connection_update2_finish()`. All D-Bus events are always processed in order. I think it's helpful to realize that on D-Bus you just send messages back and forth. There are D-Bus methods (send request and get response message) and asynchronous D-Bus events (also messages). But it's all just just an ordered stream of messages that get dispatched by iterating the GMainContext. For example at https://github.com/rhinstaller/anaconda/blob/16bcf2f019745b52e6c95ee0ab35da727b4de020/pyanaconda/modules/network/initialization.py#L105 we get a reference to a `connection` from the cache. Then there is a call `update_connection_from_ksdata()` which needs to end up iterating the main context (it must, because it calls "Update2" and waits for the reponse). Whenever you iterate the main context, then *everything* in the cache might change, for example the very next event could be that NetworkManager exited and dropped off the bus. That means, if you keep a reference to "connection", then afterwards it may be very different. That alone is not necessarily a problem. The reference you hold is still valid, worst case the object is no longer inside the NMClient cache (in which case `nm_object_get_client(connection)` returns NULL). But is something to be aware of. Multithreaded NMClient in Anaconda ---------------------------------- OK, more to the point. I don't understand how anaconda runs the GMainContext and how https://github.com/rhinstaller/anaconda/blob/16bcf2f019745b52e6c95ee0ab35da727b4de020/pyanaconda/modules/network/nm_client.py#L1042 is supposed to work. I think it's not correct. We see the call to `return queue.Queue().get()`, basically blocking that somebody puts the result inside the queue. That can only be another thread (because the current thread is blocked). That approach seems very odd to me, I suspect it's not correct, in seems difficult to get right. libnm's NMClient is not threadsafe. Meaning, you cannot access it at the same time from multiple threads without some synchronization... well, as said, NMClient does not change unless somebody is iterating the GMainContext. So multiple threads could read the same NMClient (if they ensure that the main context is not being iterated at that time!). But -- since iterating the GMainContext causes things to change, and you can only iterate a GMainContext on one thread at a time -- it means, you cannot iterate the GMainContext while you access it. In other words, the way to synchronize multithreadded access to the NMClient is by always ensuring that only the thread that "owns" (and iterates) the GMainContext accesses the NMClient. The by far simplest way to achieve that, is only to have one thread that uses the NMClient in the first place :) Basically queue.Queue().get() blocks and expects another thread to iterate the GMainContext to get the result. This would only be correct if you 1) ensure that before queue.Queue().get() no other thread is iterating the GMainContext (since you access `connection.update2()`. But I don't see that happening, because the call `return queue.Queue().get()` does not unblock a worker thread which starts iterating the GMainContext. Instead, the worker thread is already running concurrently, which is wrong. 2) ensure that after `sync_queue.put(ret)` the other thread (which is about to get unblocked) does not access `NMClient` until the thread the processes the response stops iterating the GMainContext. Basically, after queue.Queue().get() returns, you would have to know when the iterating thread is done, before accessing the NMClient. I don't see any code that would perform such synchronization, so https://github.com/rhinstaller/anaconda/blob/16bcf2f019745b52e6c95ee0ab35da727b4de020/pyanaconda/modules/network/nm_client.py#L272 is a wrong access while worker thread still iterates the GMainContext. Sorry, I think there is no easy fix. But I am very happy to be proven wrong!!
I cry uncle whenever threading comes into things, but, I am working on a different theory. I tested adding this check to `get_config_file_connection_of_device()` on both 36 and Rawhide: for con in nm_client.get_connections(): if con.get_unsaved(): log.info(f"XXX {con.get_interface_name()} has unsaved data!") on 36, it logs nothing. On Rawhide, it logs twice. So now I'm testing this: for con in nm_client.get_connections(): if con.get_unsaved(): log.info(f"XXX {con.get_interface_name()} has unsaved data!") if con.save(): log.info(f"XXX {con.get_interface_name()} saved!")
IMO it's wrong to do Queue.get()/Queue.put() on two threads. There should only be one thread. The thread that needs to make an async call (update2()) but also wants to wait for the result, that thread needs to iterate the GMainContext until the response is here. In other words, - `return sync_queue.get()` gets replaced by `gmainloop.run()` - `sync_queue.put()` gets replaced by `gmainloop.quit()` (and basically pass on the result via some closure... e.g. Something like: loop = glib.MainLoop(nmclient.get_main_context()) result = [] def finish_callback(connection, result): try: res = connection.update2_finish(result) except Exception # TODO: handle result.append(res) loop.quit() connection.update2(..., finish_callback) loop.run() return result[0] (of course, during `loop.run()`, iterating the GMainContext may dispatch any other events that happen in the meantime. So this is no truly blocking)
My new idea doesn't help either. Sigh. One thing to note from the debug output is that, whatever it may be doing wrong, the config file *is* actually getting written by ApplyKickstartTask. There's a `log_configuration_state()` in modules/network/network.py which dumps out the contents of every file in /etc/NetworkManager/system-connections when it's called. It's called right after ApplyKickstartTask is done, and it finds a /etc/NetworkManager/system-connections/ens3.nmconnection with the right contents: Jun 28 07:03:13 dhcp128.anaconda.englab.brq.redhat.com org.fedoraproject.Anaconda.Modules.Network[2093]: INFO:anaconda.threading:Thread Done: AnaTaskThread-ApplyKickstartTask-1 (140575798912576) Jun 28 07:03:13 dhcp128.anaconda.englab.brq.redhat.com NetworkManager[1945]: <debug> [1656399793.8524] active-connection[0x55fddefa9520]: set device "ens3" [0x55fddef89d30] Jun 28 07:03:13 dhcp128.anaconda.englab.brq.redhat.com NetworkManager[1945]: <debug> [1656399793.8526] device[93a92152b4933317] (ens3): add_pending_action (1): 'activation-3' Jun 28 07:03:13 dhcp128.anaconda.englab.brq.redhat.com org.fedoraproject.Anaconda.Modules.Network[2093]: DEBUG:anaconda.modules.network.network:Apply kickstart result: ['ens3'] Jun 28 07:03:13 dhcp128.anaconda.englab.brq.redhat.com org.fedoraproject.Anaconda.Modules.Network[2093]: DEBUG:anaconda.modules.network.network:Dumping configuration state - Apply kickstart Jun 28 07:03:13 dhcp128.anaconda.englab.brq.redhat.com NetworkManager[1945]: <debug> [1656399793.8528] active-connection[0x55fddefa9520]: constructed (NMActRequest, version-id 3, type managed) Jun 28 07:03:13 dhcp128.anaconda.englab.brq.redhat.com NetworkManager[1945]: <info> [1656399793.8531] device (ens3): state change: activated -> deactivating (reason 'new-activation', sys-iface-state: 'managed') Jun 28 07:03:13 dhcp128.anaconda.englab.brq.redhat.com NetworkManager[1945]: <debug> [1656399793.8532] device[93a92152b4933317] (ens3): add_pending_action (2): 'in-state-change' Jun 28 07:03:13 dhcp128.anaconda.englab.brq.redhat.com NetworkManager[1945]: <debug> [1656399793.8543] active-connection[0x55fddefa90a0]: set state deactivating (was activated) Jun 28 07:03:13 dhcp128.anaconda.englab.brq.redhat.com NetworkManager[1945]: <debug> [1656399793.8550] active-connection[0x55fddefa90a0]: check-master-ready: not signalling (state deactivating, no master) Jun 28 07:03:13 dhcp128.anaconda.englab.brq.redhat.com org.fedoraproject.Anaconda.Modules.Network[2093]: DEBUG:anaconda.modules.network.network:/etc/NetworkManager/system-connections/ens3.nmconnection: Jun 28 07:03:13 dhcp128.anaconda.englab.brq.redhat.com org.fedoraproject.Anaconda.Modules.Network[2093]: DEBUG:anaconda.modules.network.network:[connection] Jun 28 07:03:13 dhcp128.anaconda.englab.brq.redhat.com NetworkManager[1945]: <debug> [1656399793.8564] dispatcher: (10) (ens3) dispatching action 'pre-down' (with callback) Jun 28 07:03:13 dhcp128.anaconda.englab.brq.redhat.com org.fedoraproject.Anaconda.Modules.Network[2093]: DEBUG:anaconda.modules.network.network:id=ens3 Jun 28 07:03:13 dhcp128.anaconda.englab.brq.redhat.com org.fedoraproject.Anaconda.Modules.Network[2093]: DEBUG:anaconda.modules.network.network:uuid=69d924df-6493-4c5e-b85b-9d44f80cc532 Jun 28 07:03:13 dhcp128.anaconda.englab.brq.redhat.com org.fedoraproject.Anaconda.Modules.Network[2093]: DEBUG:anaconda.modules.network.network:type=ethernet Jun 28 07:03:13 dhcp128.anaconda.englab.brq.redhat.com NetworkManager[1945]: <debug> [1656399793.8576] device[93a92152b4933317] (ens3): connectivity state changed from FULL to NONE Jun 28 07:03:13 dhcp128.anaconda.englab.brq.redhat.com NetworkManager[1945]: <debug> [1656399793.8582] device[93a92152b4933317] (ens3): connectivity state changed from FULL to NONE Jun 28 07:03:13 dhcp128.anaconda.englab.brq.redhat.com org.fedoraproject.Anaconda.Modules.Network[2093]: DEBUG:anaconda.modules.network.network:autoconnect=false Jun 28 07:03:13 dhcp128.anaconda.englab.brq.redhat.com NetworkManager[1945]: <debug> [1656399793.8592] device[93a92152b4933317] (ens3): remove_pending_action (1): 'in-state-change' Jun 28 07:03:13 dhcp128.anaconda.englab.brq.redhat.com org.fedoraproject.Anaconda.Modules.Network[2093]: DEBUG:anaconda.modules.network.network:autoconnect-retries=1 Jun 28 07:03:13 dhcp128.anaconda.englab.brq.redhat.com org.fedoraproject.Anaconda.Modules.Network[2093]: DEBUG:anaconda.modules.network.network:interface-name=ens3 [etc etc] The lines are mixed in with NetworkManager doing stuff, but you can see that `autoconnect=false` is in there. So the file really does get written, before DumpMissingConfigFilesTask starts up four seconds later. Hmm, I wonder what happens if we run it again at the start of DumpMissingConfigFilesTask...
Created attachment 1893382 [details] journal log with added DDDDD messages @Thomas, yes the Queue solution is bad. I've taken advantage of a new PR (it would be great if you could look at it) we have for working with threads and contexts: https://github.com/rhinstaller/anaconda/pull/4212 updated it with removing of Queue: https://github.com/rhinstaller/anaconda/compare/master...rvykydal:anaconda:t-feng-no-queue I am attaching the output with a few logging messages (DDDDD) added. I added checking the get_filename of the connection right after the update from the same thread (task). Moreover, I've also tried to get the filenames running the client in the installer environment with the same result (/run directory): anaconda root@dhcp205 ~]# ls /etc/NetworkManager/system-connections/ ens10.nmconnection ens3.nmconnection [anaconda root@dhcp205 ~]# ls /run/NetworkManager/system-connections/ [anaconda root@dhcp205 ~]# python Python 3.11.0b3 (main, Jun 24 2022, 00:00:00) [GCC 12.1.1 20220507 (Red Hat 12.1.1-1)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import gi >>> gi.require_version("NM","1.0") >>> from gi.repository import NM >>> client = NM.Client.new() >>> cons = client.get_connections() >>> cons[0].get_filename() '/run/NetworkManager/system-connections/ens3.nmconnection' >>> cons[1].get_filename() '/run/NetworkManager/system-connections/Wired connection 1.nmconnection' I have also logs from a run where I tried to apply the solution proposed in comment #11 (https://github.com/rhinstaller/anaconda/compare/master...rvykydal:anaconda:onboot-sync-by-loop) They result is the same.
rethinking: this issue is most likely unrelated to any potential threadding issue. Sorry about being my offtopic verbosity. (In reply to Radek Vykydal from comment #13) > anaconda root@dhcp205 ~]# ls /etc/NetworkManager/system-connections/ > ens10.nmconnection ens3.nmconnection > [anaconda root@dhcp205 ~]# ls /run/NetworkManager/system-connections/ > [anaconda root@dhcp205 ~]# python > Python 3.11.0b3 (main, Jun 24 2022, 00:00:00) [GCC 12.1.1 20220507 (Red Hat > 12.1.1-1)] on linux > Type "help", "copyright", "credits" or "license" for more information. > >>> import gi > >>> gi.require_version("NM","1.0") > >>> from gi.repository import NM > >>> client = NM.Client.new() > >>> cons = client.get_connections() > >>> cons[0].get_filename() > '/run/NetworkManager/system-connections/ens3.nmconnection' > >>> cons[1].get_filename() > '/run/NetworkManager/system-connections/Wired connection 1.nmconnection' this looks as if the files were moved, without telling NetworkManager about it. Files can be edited/moved, but then it needs to be followed by `nmcli connection reload`, `nmcli connection load "$FILENAME"` or the corresponding libnm/D-Bus API. Could that be the problem? Theretically there could be a bug in the deamon to mess up the filename. But I'd be surprised, also because this code didn't change recently.
Thomas: I think that's what I was investigating in my comments above, isn't it? I tried various variations on called `reload_connections()`, `reload()`, and calling the individual connection's `save()` before trying to read the config files from it. None of that helped...
(In reply to Thomas Haller from comment #14) > rethinking: this issue is most likely unrelated to any potential threadding > issue. Sorry about being my offtopic verbosity. > No problem, thank you for the insights, they are very helpful for me in the context of https://github.com/rhinstaller/anaconda/pull/4212 > > (In reply to Radek Vykydal from comment #13) > > anaconda root@dhcp205 ~]# ls /etc/NetworkManager/system-connections/ > > ens10.nmconnection ens3.nmconnection > > [anaconda root@dhcp205 ~]# ls /run/NetworkManager/system-connections/ > > [anaconda root@dhcp205 ~]# python > > Python 3.11.0b3 (main, Jun 24 2022, 00:00:00) [GCC 12.1.1 20220507 (Red Hat > > 12.1.1-1)] on linux > > Type "help", "copyright", "credits" or "license" for more information. > > >>> import gi > > >>> gi.require_version("NM","1.0") > > >>> from gi.repository import NM > > >>> client = NM.Client.new() > > >>> cons = client.get_connections() > > >>> cons[0].get_filename() > > '/run/NetworkManager/system-connections/ens3.nmconnection' > > >>> cons[1].get_filename() > > '/run/NetworkManager/system-connections/Wired connection 1.nmconnection' > > this looks as if the files were moved, without telling NetworkManager about > it. Files can be edited/moved, but then it needs to be followed by `nmcli > connection reload`, `nmcli connection load "$FILENAME"` or the corresponding > libnm/D-Bus API. Could that be the problem? Well, the files were 'moved' by making a connection persistent in the update2 call as described in comment #4: (In reply to Radek Vykydal from comment #4) > After initramfs there are these configuration files (created in intramfs) on > the system: > /run/NetworkManager/system-connections/Wired connection 1.nmconnection > /run/NetworkManager/system-connections/ens3.nmconnection > > > Anaconda applies the kickstart configuration on the ens3 connection, making > the connection persistent in > https://github.com/rhinstaller/anaconda/blob/16bcf2f019745b52e6c95ee0ab35da727b4de020/pyanaconda/network.py#L241 > running > https://github.com/rhinstaller/anaconda/blob/16bcf2f019745b52e6c95ee0ab35da727b4de020/pyanaconda/modules/network/initialization.py#L73 > calling NM update2 method, making the connection persistent > > > After this task the configuration files are (ens3 made persistent): > /run/NetworkManager/system-connections/Wired connection 1.nmconnection > /etc/NetworkManager/system-connections/ens3.nmconnection After these calls the check calling NM from python in comment #13 returns the old (/run/.. and /run/..) paths. On older composes (before python 3.11) it returns the correct paths (/run.. and /etc..) > Theretically there could be a bug in the deamon to mess up the filename. But > I'd be surprised, also because this code didn't change recently. Yes! And that seems weird to me, the composes started to fail without NM version changed, the changed packages are in comment #4.
Trying on installed system (check_state.sh is listing dirs and running filenames.py): [root@dhcp144 ~]# cat filenames.py. #!/usr/bin/python3 import gi gi.require_version("NM","1.0") from gi.repository import NM from gi.repository import NM client = NM.Client.new() cons = client.get_connections() for con in cons: name = con.get_filename() print(name) -------------------- latest rawhide: [root@dhcp144 ~]# nmcli con NAME UUID TYPE DEVICE. Wired connection 1 95f90694-fb14-3658-a201-ab9dbd4a3c8c ethernet ens3... Wired connection 2 a2d9487e-2411-37f1-a07f-ed3d77a0846f ethernet ens10.. [root@dhcp144 ~]# ./check_state.sh. ls /etc/NetworkManager/system-connections ls /run/NetworkManager/system-connections 'Wired connection 1.nmconnection' 'Wired connection 2.nmconnection' /run/NetworkManager/system-connections/Wired connection 1.nmconnection /run/NetworkManager/system-connections/Wired connection 2.nmconnection [root@dhcp144 ~]# nmcli con e 'Wired connection 1' ===| nmcli interactive connection editor |=== Editing existing '802-3-ethernet' connection: 'Wired connection 1' Type 'help' or '?' for available commands. Type 'print' to show all the connection properties. Type 'describe [<setting>.<prop>]' for detailed property description. You may edit the following settings: connection, 802-3-ethernet (ethernet), 802-1x, dcb, sriov, ethtool, match, ipv4, ipv6, hostname, tc, proxy nmcli> save persistent. Connection 'Wired connection 1' (95f90694-fb14-3658-a201-ab9dbd4a3c8c) successfully updated. nmcli> quit [root@dhcp144 ~]# ./check_state.sh. ls /etc/NetworkManager/system-connections 'Wired connection 1.nmconnection' ls /run/NetworkManager/system-connections 'Wired connection 2.nmconnection' /run/NetworkManager/system-connections/Wired connection 1.nmconnection /run/NetworkManager/system-connections/Wired connection 2.nmconnection .... calling nmcli con reload does not help ----------------------- Installed Fedora 36 without updates (I can't find some 10 days old rawhide repository to be used) [root@dhcp218 ~]# nmcli con NAME UUID TYPE DEVICE Wired connection 1 9bc966c9-6c57-32fa-afed-d1517fdc2888 ethernet ens10 Wired connection 2 6be2fc66-75bd-372d-a44e-899182b7c5c3 ethernet ens3 [root@dhcp218 ~]# ./check_state.sh ls /etc/NetworkManager/system-connections ls /run/NetworkManager/system-connections 'Wired connection 1.nmconnection' 'Wired connection 2.nmconnection' /run/NetworkManager/system-connections/Wired connection 1.nmconnection /run/NetworkManager/system-connections/Wired connection 2.nmconnection [root@dhcp218 ~]# [root@dhcp218 ~]# nmcli con e 'Wired connection 1' ===| nmcli interactive connection editor |=== Editing existing '802-3-ethernet' connection: 'Wired connection 1' Type 'help' or '?' for available commands. Type 'print' to show all the connection properties. Type 'describe [<setting>.<prop>]' for detailed property description. You may edit the following settings: connection, 802-3-ethernet (ethernet), 802-1x, dcb, sriov, ethtool, match, ipv4, ipv6, hostname, tc, proxy nmcli> save persistent Connection 'Wired connection 1' (9bc966c9-6c57-32fa-afed-d1517fdc2888) successfully updated. nmcli> quit [root@dhcp218 ~]# nmcli con NAME UUID TYPE DEVICE Wired connection 1 9bc966c9-6c57-32fa-afed-d1517fdc2888 ethernet ens10 Wired connection 2 6be2fc66-75bd-372d-a44e-899182b7c5c3 ethernet ens3 [root@dhcp218 ~]# ./check_state.sh ls /etc/NetworkManager/system-connections 'Wired connection 1.nmconnection' ls /run/NetworkManager/system-connections 'Wired connection 2.nmconnection' /etc/NetworkManager/system-connections/Wired connection 1.nmconnection /run/NetworkManager/system-connections/Wired connection 2.nmconnection
"Yes! And that seems weird to me, the composes started to fail without NM version changed, the changed packages are in comment #4." My best guess would be that gobject-introspection behaviour changed somehow with Python 3.11.
Created attachment 1898039 [details] Diff of working installer image. According to our kickstart tests seems to be fixed with update from NetworkManager-1.39.8-1.fc37 to NetworkManager-1.39.10-1.fc37. Diff attached.
Thomas do you think the NM update from comment #19 could fix the issue ? Also, in comment #17 there is a reproducer from regular system, may put some light on the cause of the issue.
(In reply to Radek Vykydal from comment #20) > Thomas do you think the NM update from comment #19 could fix the issue ? > > Also, in comment #17 there is a reproducer from regular system, may put some > light on the cause of the issue. the Fedora packages are very close to upstream, and there is only a very small difference between 1.39.8 and 1.39.10. I don't see what would make the difference. Sorry. And sorry for the late reply! But I think this is resolved, right?