I'm running fresh Fedora 41 container on my Incus host (running Arch Linux) and try to upgrade to Fedora 42 using dnf system-upgrade, following the documentation (https://docs.fedoraproject.org/en-US/quick-docs/upgrading-fedora-offline/). When the container is rebooted to upgrade (I use dnf offline reboot rather than dnf system-upgrade reboot), I notice that /usr/sbin/ cannot be merged because of /usr/sbin/iconvconfig. As such, the culprit binary is in both /usr/bin (from F42) and /usr/sbin (from F41). systemctl status also shows unmerged-bin taint. Reproducible: Always Steps to Reproduce: 1. On host, launch Fedora 41 container and login to it: ``` $ incus launch images:fedora/41 test $ incus shell test ``` 2. Inside the container, start and enable `systemd-networkd.service` if necessary in order for IPv4 address from Incus to be assigned. Then, update the system and install bash-completion, less, vim, man-pages, and man-db. 3. Upgrade to Fedora 42 following the documentation in https://docs.fedoraproject.org/en-US/quick-docs/upgrading-fedora-offline/, but use `dnf offline reboot` instead to initiate reboot. Actual Results: Checking the journal when upgrade is performed, I notice the iconvconfig error: ``` ... Apr 07 04:41:07 test.bagas.me dnf5[157]: [ 21/416] Upgrading basesystem-0:11-22. 100% | 3.2 KiB/s | 124.0 B | 00m00s Apr 07 04:41:17 test.bagas.me dnf5[157]: [ 22/416] Upgrading glibc-all-langpacks 100% | 22.0 MiB/s | 226.9 MiB | 00m10s Apr 07 04:41:25 test.bagas.me dnf5[157]: [ 23/416] Upgrading glibc-gconv-extra-0 100% | 942.3 KiB/s | 7.3 MiB | 00m08s Apr 07 04:41:26 test.bagas.me dnf5[157]: [ 24/416] Upgrading glibc-common-0:2.41 100% | 657.0 KiB/s | 1.0 MiB | 00m02s Apr 07 04:41:27 test.bagas.me systemd[1]: Reexecution requested from client PID 166 ('systemctl') (unit dnf5-offline-transaction.service)... Apr 07 04:41:27 test.bagas.me systemd[1]: Reexecuting. Apr 07 04:41:27 test.bagas.me dnf5[157]: [ 25/416] Upgrading glibc-0:2.41-1.fc42 100% | 8.6 MiB/s | 6.7 MiB | 00m01s > <snipped>... Apr 07 04:43:01 test.bagas.me dnf5[157]: [210/416] Upgrading glibc-gconv-extra-0 100% | 6.6 MiB/s | 6.8 MiB | 00m01s Apr 07 04:43:07 test.bagas.me systemd[1]: Reexecution requested from client PID 1224 ('systemctl') (unit dnf5-offline-transaction.service)... Apr 07 04:43:07 test.bagas.me systemd[1]: Reexecuting. Apr 07 04:43:09 test.bagas.me dnf5[157]: [211/416] Upgrading glibc-0:2.41-1.fc42 100% | 732.7 KiB/s | 5.5 MiB | 00m08s > <snipped>... Apr 07 04:43:32 test.bagas.me dnf5[157]: [408/416] Removing glibc-common-0:2.40- 100% | 313.0 B/s | 52.0 B | 00m00s Apr 07 04:43:32 test.bagas.me dnf5[157]: [409/416] Removing glibc-all-langpacks- 100% | 1.9 KiB/s | 41.0 B | 00m00s Apr 07 04:43:32 test.bagas.me dnf5[157]: [410/416] Removing glibc-0:2.40-23.fc41 100% | 1.5 KiB/s | 102.0 B | 00m00s Apr 07 04:43:32 test.bagas.me dnf5[157]: [411/416] Removing basesystem-0:11-21.f 100% | 1.6 KiB/s | 100.0 B | 00m00s Apr 07 04:43:33 test.bagas.me dnf5[157]: [412/416] Removing filesystem-0:3.18-23 100% | 21.3 KiB/s | 16.9 KiB | 00m01s Apr 07 04:43:33 test.bagas.me dnf5[157]: [413/416] Removing setup-0:2.15.0-8.fc4 100% | 3.5 KiB/s | 46.0 B | 00m00s Apr 07 04:43:33 test.bagas.me dnf5[157]: [414/416] Removing fedora-release-conta 100% | 7.5 KiB/s | 100.0 B | 00m00s Apr 07 04:43:33 test.bagas.me dnf5[157]: [415/416] Removing ncurses-base-0:6.5-2 100% | 12.5 KiB/s | 179.0 B | 00m00s Apr 07 04:43:33 test.bagas.me dnf5[157]: warning: posix.fork(): .fork(), .exec(), .wait() and .redirect2null() are deprecated, use rpm.spawn() or rpm.execute() instead Apr 07 04:43:33 test.bagas.me dnf5[157]: warning: posix.wait(): .fork(), .exec(), .wait() and .redirect2null() are deprecated, use rpm.spawn() or rpm.execute() instead Apr 07 04:43:33 test.bagas.me dnf5[1482]: warning: posix.exec(): .fork(), .exec(), .wait() and .redirect2null() are deprecated, use rpm.spawn() or rpm.execute() instead Apr 07 04:43:33 test.bagas.me dnf5[157]: /usr/sbin cannot be merged, found /usr/sbin/iconvconfig Apr 07 04:43:33 test.bagas.me dnf5[157]: All files under /usr/local/sbin are symlinks; linking to ./bin... Apr 07 04:43:33 test.bagas.me dnf5[157]: ...done Apr 07 04:43:37 test.bagas.me systemd[1]: Reexecution requested from client PID 2689 ('systemctl') (unit dnf5-offline-transaction.service)... Apr 07 04:43:37 test.bagas.me systemd[1]: Reexecuting. Apr 07 04:43:37 test.bagas.me systemd[1]: Reload requested from client PID 2813 ('systemctl') (unit dnf5-offline-transaction.service)... Apr 07 04:43:37 test.bagas.me systemd[1]: Reloading... Apr 07 04:43:38 test.bagas.me systemd[1]: Reloading finished in 291 ms. Apr 07 04:43:38 test.bagas.me dnf5[157]: /usr/sbin cannot be merged, found /usr/sbin/iconvconfig Apr 07 04:43:40 test.bagas.me systemd[1]: Started run-p2960-i22863.service - [systemd-run] /usr/bin/systemctl start man-db-cache-update. Apr 07 04:43:40 test.bagas.me systemd[1]: Queuing reload/restart jobs for marked units… > <snipped>... Apr 07 04:43:41 test.bagas.me systemd[1]: man-db-cache-update.service: Deactivated successfully. Apr 07 04:43:41 test.bagas.me systemd[1]: Finished man-db-cache-update.service. Apr 07 04:43:41 test.bagas.me systemd[1]: run-p2960-i22863.service: Deactivated successfully. Apr 07 04:43:43 test.bagas.me systemd[1]: Reload requested from client PID 3037 ('systemctl') (unit dnf5-offline-transaction.service)... Apr 07 04:43:43 test.bagas.me systemd[1]: Reloading... Apr 07 04:43:43 test.bagas.me systemd[1]: Reloading finished in 291 ms. Apr 07 04:43:43 test.bagas.me systemd[1]: Queuing reload/restart jobs for marked units… Apr 07 04:43:45 test.bagas.me systemd[1]: Started run-p3150-i23056.service - [systemd-run] /usr/bin/systemctl start man-db-cache-update. Apr 07 04:43:45 test.bagas.me systemd[1]: Starting man-db-cache-update.service... Apr 07 04:43:49 test.bagas.me systemd[1]: man-db-cache-update.service: Deactivated successfully. Apr 07 04:43:49 test.bagas.me systemd[1]: Finished man-db-cache-update.service. Apr 07 04:43:50 test.bagas.me systemd[1]: run-p3150-i23056.service: Deactivated successfully. Apr 07 04:43:58 test.bagas.me dnf5[157]: [416/416] Removing libgcc-0:14.2.1-7.fc 100% | 0.0 B/s | 10.0 B | 00m25s Apr 07 04:43:58 test.bagas.me dnf5[157]: Warning: skipped OpenPGP checks for 209 packages from repository: fedora Apr 07 04:43:58 test.bagas.me dnf5[157]: Transaction complete! Cleaning up and rebooting... Apr 07 04:43:58 test.bagas.me dnf5[157]: Couldn't connect to D-Bus: [org.freedesktop.DBus.Error.FileNotFound] Failed to open bus (No such file or directory) Apr 07 04:43:58 test.bagas.me systemd[1]: dnf5-offline-transaction.service: Main process exited, code=exited, status=1/FAILURE Apr 07 04:43:58 test.bagas.me systemd[1]: dnf5-offline-transaction.service: Failed with result 'exit-code'. Apr 07 04:43:58 test.bagas.me systemd[1]: Failed to start dnf5-offline-transaction.service - Offline upgrades/transactions using DNF 5. Apr 07 04:43:58 test.bagas.me systemd[1]: dnf5-offline-transaction.service: Consumed 30.920s CPU time, 738.2M memory peak. Apr 07 04:43:58 test.bagas.me systemd[1]: dnf5-offline-transaction.service: Triggering OnFailure= dependencies. ... ``` Expected Results: /usr/sbin/iconvconfig should've been deleted, successfully merging /usr/sbin.
Created attachment 2083639 [details] journal from upgrade boot
Created attachment 2083640 [details] package list post-upgrade
I'm pretty sure this is not a glibc issue. On your system, iconvconfig is just the first directory entry in readdir order (this varies between file systems, but iconvconfig is often the first file created in that directory). Zbigniew, is this expected behavior on upgrades? If so, can the message be changed, so that people do not report this as a glibc bug.
Well, which package is supposed to own /usr/sbin/iconvconfig? On my Fedora 42 system, no package owns it. Can it be removed safely? And how would a packaging update be able to remove a file that no package even owns? (maybe hard-code this as an exception in the migration scriptlet?)
(In reply to Fabio Valentini from comment #4) > Well, which package is supposed to own /usr/sbin/iconvconfig? On my Fedora > 42 system, no package owns it. Can it be removed safely? > And how would a packaging update be able to remove a file that no package > even owns? (maybe hard-code this as an exception in the migration scriptlet?) Fedora 42 does not have /usr/sbin/iconvconfig, it's /usr/bin/iconvconfig instead. It's still owned by glibc.
(In reply to Florian Weimer from comment #5) > (In reply to Fabio Valentini from comment #4) > > Well, which package is supposed to own /usr/sbin/iconvconfig? On my Fedora > > 42 system, no package owns it. Can it be removed safely? > > And how would a packaging update be able to remove a file that no package > > even owns? (maybe hard-code this as an exception in the migration scriptlet?) > > Fedora 42 does not have /usr/sbin/iconvconfig, it's /usr/bin/iconvconfig > instead. It's still owned by glibc. My system disagrees with you :) I have that file. So something must have gone wrong on upgrade from Fedora 41 to 42. The file is still there, but now unowned.
(In reply to Fabio Valentini from comment #6) > (In reply to Florian Weimer from comment #5) > > (In reply to Fabio Valentini from comment #4) > > > Well, which package is supposed to own /usr/sbin/iconvconfig? On my Fedora > > > 42 system, no package owns it. Can it be removed safely? > > > And how would a packaging update be able to remove a file that no package > > > even owns? (maybe hard-code this as an exception in the migration scriptlet?) > > > > Fedora 42 does not have /usr/sbin/iconvconfig, it's /usr/bin/iconvconfig > > instead. It's still owned by glibc. > > My system disagrees with you :) I have that file. So something must have > gone wrong on upgrade from Fedora 41 to 42. The file is still there, but now > unowned. Hmm, so /usr/sbin was not replaced by /usr/bin + symlink for you? Could this be a package installation order issue?
(In reply to Florian Weimer from comment #7) > Hmm, so /usr/sbin was not replaced by /usr/bin + symlink for you? > > Could this be a package installation order issue? No, it wasn't migrated - there are still at least three packages on my F42 Workstation install that install things to /usr/sbin: - dmraid / dmraid-events - iscsi-initiator-utils - vpnc These all appear to be installed by default on Fedora 42 Workstation (through NetworkManager or gnome-boxes / libvirt) so I'm not sure how *anyone* can end up with a migrated system. :(
Hmm, so something is awry here. I repeated a process of the upgrade in a systemd-nspawn container and I can reproduce the issue. (I don't use incus, but I don't think the details of the container manager matter here.) After the upgrade: # find /usr/sbin/ -type f /usr/sbin/iconvconfig /usr/sbin/ldconfig So there is something "special" about the two glibc files, they unexpectedly remain after the upgrade. (But there's some strange behaviour, because rpm both knows and doesn't know about the file: $ rpm -ql glibc-2.41-3.fc42.x86_64|grep bin/ldconfig /usr/bin/ldconfig $ rpm -qf /usr/sbin/ldconfig glibc-2.41-3.fc42.x86_64 But rpm lists a few other symlinks in /usr/sbin as owned by a package, so maybe that's OK.) The upgrade logs looks like this: [434186.662437] dnf5[30]: Symlinking /usr/sbin/halt->/usr/bin/halt [434186.662536] dnf5[30]: Symlinking /usr/sbin/init->/usr/bin/init [434186.662559] dnf5[30]: Symlinking /usr/sbin/mount.ddi->/usr/bin/mount.ddi [434186.662576] dnf5[30]: Symlinking /usr/sbin/poweroff->/usr/bin/poweroff [434186.662594] dnf5[30]: Symlinking /usr/sbin/reboot->/usr/bin/reboot [434186.662619] dnf5[30]: Symlinking /usr/sbin/runlevel->/usr/bin/runlevel [434186.662636] dnf5[30]: Symlinking /usr/sbin/shutdown->/usr/bin/shutdown [434186.662653] dnf5[30]: Symlinking /usr/sbin/telinit->/usr/bin/telinit systemd 257.3-7.fc42 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +IPE +SMACK +SECCOMP -GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +BTF +XKBCOMMON +UTMP +SYSVINIT +LIBARCHIVE) Detected virtualization systemd-nspawn. Detected architecture x86-64. Failed to open libbpf, cgroup BPF features disabled: Operation not supported [434186.810568] dnf5[30]: [156/278] Removing systemd-0:256.12-1.f 100% | 6.0 KiB/s | 989.0 B | 00m00s [434186.829209] dnf5[30]: Symlinking /usr/sbin/addpart->/usr/bin/addpart [434186.829307] dnf5[30]: Symlinking /usr/sbin/blkdiscard->/usr/bin/blkdiscard [434186.829327] dnf5[30]: Symlinking /usr/sbin/blkpr->/usr/bin/blkpr [434186.829350] dnf5[30]: Symlinking /usr/sbin/blkzone->/usr/bin/blkzone [434186.829367] dnf5[30]: Symlinking /usr/sbin/cfdisk->/usr/bin/cfdisk [434186.829396] dnf5[30]: Symlinking /usr/sbin/chcpu->/usr/bin/chcpu [434186.829413] dnf5[30]: Symlinking /usr/sbin/clock->/usr/bin/clock [434186.829431] dnf5[30]: Symlinking /usr/sbin/ctrlaltdel->/usr/bin/ctrlaltdel [434186.829448] dnf5[30]: Symlinking /usr/sbin/delpart->/usr/bin/delpart [434186.829466] dnf5[30]: Symlinking /usr/sbin/fdformat->/usr/bin/fdformat [434186.829482] dnf5[30]: Symlinking /usr/sbin/fdisk->/usr/bin/fdisk [434186.829500] dnf5[30]: Symlinking /usr/sbin/findfs->/usr/bin/findfs [434186.829516] dnf5[30]: Symlinking /usr/sbin/fsck.cramfs->/usr/bin/fsck.cramfs [434186.829533] dnf5[30]: Symlinking /usr/sbin/fsck.minix->/usr/bin/fsck.minix [434186.829551] dnf5[30]: Symlinking /usr/sbin/fsfreeze->/usr/bin/fsfreeze [434186.829567] dnf5[30]: Symlinking /usr/sbin/fstrim->/usr/bin/fstrim [434186.829585] dnf5[30]: Symlinking /usr/sbin/hwclock->/usr/bin/hwclock [434186.829604] dnf5[30]: Symlinking /usr/sbin/ldattach->/usr/bin/ldattach [434186.829620] dnf5[30]: Symlinking /usr/sbin/mkfs->/usr/bin/mkfs [434186.829636] dnf5[30]: Symlinking /usr/sbin/mkfs.cramfs->/usr/bin/mkfs.cramfs [434186.829653] dnf5[30]: Symlinking /usr/sbin/mkfs.minix->/usr/bin/mkfs.minix [434186.829669] dnf5[30]: Symlinking /usr/sbin/nologin->/usr/bin/nologin [434186.829686] dnf5[30]: Symlinking /usr/sbin/pivot_root->/usr/bin/pivot_root [434186.829703] dnf5[30]: Symlinking /usr/sbin/readprofile->/usr/bin/readprofile [434186.829720] dnf5[30]: Symlinking /usr/sbin/resizepart->/usr/bin/resizepart [434186.829734] dnf5[30]: Symlinking /usr/sbin/rfkill->/usr/bin/rfkill [434186.829748] dnf5[30]: Symlinking /usr/sbin/rtcwake->/usr/bin/rtcwake [434186.829765] dnf5[30]: Symlinking /usr/sbin/runuser->/usr/bin/runuser [434186.829782] dnf5[30]: Symlinking /usr/sbin/sfdisk->/usr/bin/sfdisk [434186.829798] dnf5[30]: Symlinking /usr/sbin/swaplabel->/usr/bin/swaplabel [434186.829815] dnf5[30]: Symlinking /usr/sbin/wipefs->/usr/bin/wipefs [434186.829834] dnf5[30]: Symlinking /usr/sbin/zramctl->/usr/bin/zramctl [434186.858146] dnf5[30]: [157/278] Removing util-linux-0:2.40.4- 100% | 9.8 KiB/s | 464.0 B | 00m00s [434186.869320] dnf5[30]: Symlinking /usr/sbin/agetty->/usr/bin/agetty [434186.869409] dnf5[30]: Symlinking /usr/sbin/blkid->/usr/bin/blkid [434186.869447] dnf5[30]: Symlinking /usr/sbin/blockdev->/usr/bin/blockdev [434186.869490] dnf5[30]: Symlinking /usr/sbin/fsck->/usr/bin/fsck [434186.869511] dnf5[30]: Symlinking /usr/sbin/losetup->/usr/bin/losetup [434186.869534] dnf5[30]: Symlinking /usr/sbin/mkswap->/usr/bin/mkswap [434186.869554] dnf5[30]: Symlinking /usr/sbin/partx->/usr/bin/partx [434186.869576] dnf5[30]: Symlinking /usr/sbin/sulogin->/usr/bin/sulogin [434186.869597] dnf5[30]: Symlinking /usr/sbin/swapoff->/usr/bin/swapoff [434186.869617] dnf5[30]: Symlinking /usr/sbin/swapon->/usr/bin/swapon [434186.869638] dnf5[30]: Symlinking /usr/sbin/switch_root->/usr/bin/switch_root [434186.881058] dnf5[30]: [158/278] Removing util-linux-core-0:2. 100% | 6.6 KiB/s | 149.0 B | 00m00s [434186.887742] dnf5[30]: Symlinking /usr/sbin/faillock->/usr/bin/faillock [434186.887838] dnf5[30]: Symlinking /usr/sbin/mkhomedir_helper->/usr/bin/mkhomedir_helper [434186.887866] dnf5[30]: Symlinking /usr/sbin/pam_namespace_helper->/usr/bin/pam_namespace_helper [434186.887889] dnf5[30]: Symlinking /usr/sbin/pam_timestamp_check->/usr/bin/pam_timestamp_check [434186.887911] dnf5[30]: Symlinking /usr/sbin/pwhistory_helper->/usr/bin/pwhistory_helper [434186.887932] dnf5[30]: Symlinking /usr/sbin/unix_chkpwd->/usr/bin/unix_chkpwd [434186.887954] dnf5[30]: Symlinking /usr/sbin/unix_update->/usr/bin/unix_update [434186.894747] dnf5[30]: [159/278] Removing pam-0:1.6.1-7.fc41.x 100% | 24.3 KiB/s | 349.0 B | 00m00s [434186.896855] dnf5[30]: [160/278] Removing systemd-pam-0:256.12 100% | 6.3 KiB/s | 13.0 B | 00m00s [434186.898657] dnf5[30]: [161/278] Removing libfdisk-0:2.40.4-1. 100% | 5.9 KiB/s | 6.0 B | 00m00s [434186.900870] dnf5[30]: [162/278] Removing fmt-0:11.0.2-2.fc41. 100% | 9.8 KiB/s | 10.0 B | 00m00s [434186.913457] dnf5[30]: [163/278] Removing libstdc++-0:14.2.1-7 100% | 2.5 KiB/s | 31.0 B | 00m00s [434186.926924] dnf5[30]: [164/278] Removing authselect-0:1.5.0-8 100% | 2.3 KiB/s | 28.0 B | 00m00s [434186.938890] dnf5[30]: [165/278] Removing glibc-0:2.40-24.fc41 100% | 8.9 KiB/s | 109.0 B | 00m00s [434186.950845] dnf5[30]: [166/278] Removing man-pages-0:6.9.1-2. 100% | 258.3 KiB/s | 2.8 KiB | 00m00s [434186.961820] dnf5[30]: [167/278] Removing vim-common-2:9.1.127 100% | 408.2 KiB/s | 2.4 KiB | 00m00s [434186.967794] dnf5[30]: [168/278] Removing glibc-gconv-extra-0: 100% | 62.3 KiB/s | 638.0 B | 00m00s [434186.969992] dnf5[30]: [169/278] Removing dbus-1:1.14.10-4.fc4 100% | 48.8 KiB/s | 100.0 B | 00m00s [434186.983717] dnf5[30]: [170/278] Removing bash-completion-1:2. 100% | 79.3 KiB/s | 1.0 KiB | 00m00s [434186.997703] dnf5[30]: [171/278] Removing dbus-broker-0:36-4.f 100% | 1.2 KiB/s | 16.0 B | 00m00s [434187.006731] dnf5[30]: [172/278] Removing systemd-libs-0:256.1 100% | 2.5 KiB/s | 23.0 B | 00m00s [434187.021431] dnf5[30]: [173/278] Removing findutils-1:4.10.0-4 100% | 4.1 KiB/s | 59.0 B | 00m00s [434187.035746] dnf5[30]: [174/278] Removing libmodulemd-0:2.15.0 100% | 1.2 KiB/s | 16.0 B | 00m00s [434187.045749] dnf5[30]: [175/278] Removing glib2-0:2.82.5-1.fc4 100% | 17.3 KiB/s | 177.0 B | 00m00s [434187.047843] dnf5[30]: [176/278] Removing rpm-libs-0:4.20.1-1. 100% | 4.9 KiB/s | 10.0 B | 00m00s [434187.050055] dnf5[30]: [177/278] Removing gnutls-0:3.8.9-2.fc4 100% | 16.1 KiB/s | 33.0 B | 00m00s [434187.052242] dnf5[30]: [178/278] Removing libmount-0:2.40.4-1. 100% | 2.9 KiB/s | 6.0 B | 00m00s [434187.054426] dnf5[30]: [179/278] Removing rpm-sequoia-0:1.7.0- 100% | 4.4 KiB/s | 9.0 B | 00m00s [434187.056334] dnf5[30]: [180/278] Removing libblkid-0:2.40.4-1. 100% | 5.9 KiB/s | 6.0 B | 00m00s [434187.058213] dnf5[30]: [181/278] Removing sqlite-libs-0:3.46.1 100% | 6.8 KiB/s | 7.0 B | 00m00s [434187.067312] dnf5[30]: [182/278] Removing authselect-libs-0:1. 100% | 12.2 KiB/s | 112.0 B | 00m00s [434187.082469] dnf5[30]: [183/278] Removing libpwquality-0:1.4.5 100% | 5.2 KiB/s | 80.0 B | 00m00s [434187.090681] dnf5[30]: [184/278] Removing curl-0:8.9.1-3.fc41. 100% | 2.1 KiB/s | 17.0 B | 00m00s [434187.092783] dnf5[30]: [185/278] Removing libcurl-0:8.9.1-3.fc 100% | 3.4 KiB/s | 7.0 B | 00m00s [434187.095190] dnf5[30]: [186/278] Removing openldap-0:2.6.9-1.f 100% | 12.7 KiB/s | 26.0 B | 00m00s [434187.097259] dnf5[30]: [187/278] Removing libevent-0:2.1.12-14 100% | 12.2 KiB/s | 25.0 B | 00m00s [434187.108664] dnf5[30]: [188/278] Removing libssh-0:0.10.6-8.fc 100% | 1.2 KiB/s | 14.0 B | 00m00s [434187.119728] dnf5[30]: Symlinking /usr/sbin/sasldblistusers2->/usr/bin/sasldblistusers2 [434187.119853] dnf5[30]: Symlinking /usr/sbin/saslpasswd2->/usr/bin/saslpasswd2 [434187.126542] dnf5[30]: [189/278] Removing cyrus-sasl-lib-0:2.1 100% | 2.2 KiB/s | 40.0 B | 00m00s [434187.134225] dnf5[30]: [190/278] Removing lua-libs-0:5.4.7-3.f 100% | 1.1 KiB/s | 8.0 B | 00m00s [434187.142358] dnf5[30]: [191/278] Removing less-0:668-1.fc41.x8 100% | 2.8 KiB/s | 23.0 B | 00m00s [434187.150612] dnf5[30]: [192/278] Removing popt-0:1.19-7.fc41.x 100% | 5.4 KiB/s | 44.0 B | 00m00s [434187.162269] dnf5[30]: [193/278] Removing gdbm-1:1.23-7.fc41.x 100% | 2.9 KiB/s | 33.0 B | 00m00s [434187.168743] dnf5[30]: Symlinking /usr/sbin/cracklib-check->/usr/bin/cracklib-check [434187.168810] dnf5[30]: Symlinking /usr/sbin/cracklib-format->/usr/bin/cracklib-format [434187.168836] dnf5[30]: Symlinking /usr/sbin/cracklib-packer->/usr/bin/cracklib-packer [434187.168859] dnf5[30]: Symlinking /usr/sbin/cracklib-unpacker->/usr/bin/cracklib-unpacker [434187.168901] dnf5[30]: Symlinking /usr/sbin/cracklib-update->/usr/bin/cracklib-update [434187.168925] dnf5[30]: Symlinking /usr/sbin/create-cracklib-dict->/usr/bin/create-cracklib-dict [434187.180202] dnf5[30]: [194/278] Removing cracklib-0:2.9.11-6. 100% | 4.1 KiB/s | 75.0 B | 00m00s [434187.188457] dnf5[30]: [195/278] Removing gzip-0:1.13-2.fc41.x 100% | 4.9 KiB/s | 40.0 B | 00m00s [434187.190181] dnf5[30]: [196/278] Removing libuuid-0:2.40.4-1.f 100% | 6.8 KiB/s | 7.0 B | 00m00s [434187.198277] dnf5[30]: [197/278] Removing xz-libs-1:5.6.2-2.fc 100% | 875.0 B/s | 7.0 B | 00m00s [434187.206482] dnf5[30]: [198/278] Removing expat-0:2.7.1-1.fc41 100% | 1.7 KiB/s | 14.0 B | 00m00s [434187.208159] dnf5[30]: [199/278] Removing libsmartcols-0:2.40. 100% | 6.8 KiB/s | 7.0 B | 00m00s [434187.209860] dnf5[30]: [200/278] Removing libpsl-0:0.21.5-4.fc 100% | 6.8 KiB/s | 7.0 B | 00m00s [434187.221956] dnf5[30]: [201/278] Removing libutempter-0:1.2.1- 100% | 1.2 KiB/s | 13.0 B | 00m00s [434187.232612] dnf5[30]: Symlinking /usr/sbin/adduser->/usr/bin/adduser [434187.232703] dnf5[30]: Symlinking /usr/sbin/chgpasswd->/usr/bin/chgpasswd [434187.232733] dnf5[30]: Symlinking /usr/sbin/chpasswd->/usr/bin/chpasswd [434187.232755] dnf5[30]: Symlinking /usr/sbin/groupadd->/usr/bin/groupadd [434187.232777] dnf5[30]: Symlinking /usr/sbin/groupdel->/usr/bin/groupdel [434187.232800] dnf5[30]: Symlinking /usr/sbin/groupmems->/usr/bin/groupmems [434187.232818] dnf5[30]: Symlinking /usr/sbin/groupmod->/usr/bin/groupmod [434187.232840] dnf5[30]: Symlinking /usr/sbin/grpck->/usr/bin/grpck [434187.232859] dnf5[30]: Symlinking /usr/sbin/grpconv->/usr/bin/grpconv [434187.232881] dnf5[30]: Symlinking /usr/sbin/grpunconv->/usr/bin/grpunconv [434187.232906] dnf5[30]: Symlinking /usr/sbin/newusers->/usr/bin/newusers [434187.232928] dnf5[30]: Symlinking /usr/sbin/pwck->/usr/bin/pwck [434187.232948] dnf5[30]: Symlinking /usr/sbin/pwconv->/usr/bin/pwconv [434187.232969] dnf5[30]: Symlinking /usr/sbin/pwunconv->/usr/bin/pwunconv [434187.232989] dnf5[30]: Symlinking /usr/sbin/useradd->/usr/bin/useradd [434187.233013] dnf5[30]: Symlinking /usr/sbin/userdel->/usr/bin/userdel [434187.233030] dnf5[30]: Symlinking /usr/sbin/usermod->/usr/bin/usermod [434187.233052] dnf5[30]: Symlinking /usr/sbin/vigr->/usr/bin/vigr [434187.233073] dnf5[30]: Symlinking /usr/sbin/vipw->/usr/bin/vipw [434187.240440] dnf5[30]: [202/278] Removing shadow-utils-2:4.15. 100% | 22.6 KiB/s | 439.0 B | 00m00s [434187.242467] dnf5[30]: [203/278] Removing libsemanage-0:3.7-2. 100% | 5.9 KiB/s | 12.0 B | 00m00s [434187.244172] dnf5[30]: [204/278] Removing zchunk-libs-0:1.5.1- 100% | 8.8 KiB/s | 9.0 B | 00m00s [434187.245580] dnf5[30]: [205/278] Removing json-c-0:0.17-4.fc41 100% | 7.8 KiB/s | 8.0 B | 00m00s [434187.247178] dnf5[30]: [206/278] Removing libzstd-0:1.5.7-1.fc 100% | 6.8 KiB/s | 7.0 B | 00m00s [434187.249099] dnf5[30]: [207/278] Removing libxcrypt-0:4.4.38-7 100% | 17.6 KiB/s | 18.0 B | 00m00s [434187.257242] dnf5[30]: [208/278] Removing libtool-ltdl-0:2.4.7 100% | 875.0 B/s | 7.0 B | 00m00s [434187.264956] dnf5[30]: [209/278] Removing which-0:2.21-42.fc41 100% | 2.1 KiB/s | 15.0 B | 00m00s [434187.266690] dnf5[30]: [210/278] Removing gdbm-libs-1:1.23-7.f 100% | 10.7 KiB/s | 11.0 B | 00m00s [434187.268479] dnf5[30]: [211/278] Removing libbrotli-0:1.1.0-5. 100% | 14.6 KiB/s | 15.0 B | 00m00s [434187.276419] dnf5[30]: [212/278] Removing nettle-0:3.10-3.fc41 100% | 2.6 KiB/s | 21.0 B | 00m00s [434187.284165] dnf5[30]: [213/278] Removing xxd-2:9.1.1275-1.fc4 100% | 1.0 KiB/s | 7.0 B | 00m00s [434187.285863] dnf5[30]: [214/278] Removing libnsl2-0:2.0.1-2.fc 100% | 6.8 KiB/s | 7.0 B | 00m00s [434187.288245] dnf5[30]: [215/278] Removing libtirpc-0:1.3.6-1.r 100% | 5.9 KiB/s | 12.0 B | 00m00s [434187.297078] dnf5[30]: [216/278] Removing krb5-libs-0:1.21.3-4 100% | 6.9 KiB/s | 64.0 B | 00m00s [434187.316494] dnf5[30]: [217/278] Removing ca-certificates-0:20 100% | 2.8 KiB/s | 49.0 B | 00m00s [434187.327612] dnf5[30]: Symlinking /usr/sbin/chroot->/usr/bin/chroot [434187.344917] dnf5[30]: [218/278] Removing coreutils-0:9.5-11.f 100% | 9.8 KiB/s | 300.0 B | 00m00s [434187.354672] dnf5[30]: [219/278] Removing openssl-libs-1:3.2.4 100% | 4.2 KiB/s | 39.0 B | 00m00s [434187.373709] dnf5[30]: [220/278] Removing p11-kit-trust-0:0.25 100% | 666.0 B/s | 12.0 B | 00m00s [434187.389325] dnf5[30]: [221/278] Removing gawk-0:5.3.0-4.fc41. 100% | 7.4 KiB/s | 113.0 B | 00m00s [434187.401462] dnf5[30]: [222/278] Removing p11-kit-0:0.25.5-3.f 100% | 8.8 KiB/s | 99.0 B | 00m00s [434187.409371] dnf5[30]: Symlinking /usr/sbin/capsh->/usr/bin/capsh [434187.409442] dnf5[30]: Symlinking /usr/sbin/getcap->/usr/bin/getcap [434187.409468] dnf5[30]: Symlinking /usr/sbin/getpcaps->/usr/bin/getpcaps [434187.409501] dnf5[30]: Symlinking /usr/sbin/setcap->/usr/bin/setcap [434187.420165] dnf5[30]: [223/278] Removing libcap-0:2.70-4.fc41 100% | 1.8 KiB/s | 34.0 B | 00m00s [434187.428855] dnf5[30]: [224/278] Removing sed-0:4.9-3.fc41.x86 100% | 6.7 KiB/s | 55.0 B | 00m00s [434187.431223] dnf5[30]: [225/278] Removing libselinux-0:3.7-5.f 100% | 3.9 KiB/s | 8.0 B | 00m00s [434187.439400] dnf5[30]: [226/278] Removing pam-libs-0:1.6.1-7.f 100% | 1.8 KiB/s | 15.0 B | 00m00s [434187.447858] dnf5[30]: [227/278] Removing grep-0:3.11-9.fc41.x 100% | 7.9 KiB/s | 65.0 B | 00m00s [434187.449862] dnf5[30]: [228/278] Removing libsepol-0:3.7-2.fc4 100% | 2.9 KiB/s | 6.0 B | 00m00s [434187.451900] dnf5[30]: [229/278] Removing readline-0:8.2-10.fc 100% | 6.8 KiB/s | 14.0 B | 00m00s [434187.454432] dnf5[30]: [230/278] Removing audit-libs-0:4.0.3-1 100% | 6.8 KiB/s | 14.0 B | 00m00s [434187.460023] dnf5[30]: [231/278] Removing libcap-ng-0:0.8.5-3. 100% | 2.3 KiB/s | 12.0 B | 00m00s [434187.466250] dnf5[30]: Symlinking /usr/sbin/alternatives->/usr/bin/alternatives [434187.466310] dnf5[30]: Symlinking /usr/sbin/update-alternatives->/usr/bin/update-alternatives [434187.470416] dnf5[30]: [232/278] Removing alternatives-0:1.31- 100% | 1.2 KiB/s | 12.0 B | 00m00s [434187.472141] dnf5[30]: [233/278] Removing libeconf-0:0.6.2-3.f 100% | 10.7 KiB/s | 11.0 B | 00m00s [434187.474062] dnf5[30]: [234/278] Removing libacl-0:2.3.2-2.fc4 100% | 3.9 KiB/s | 4.0 B | 00m00s [434187.475860] dnf5[30]: [235/278] Removing pcre2-0:10.44-1.fc41 100% | 8.8 KiB/s | 9.0 B | 00m00s [434187.477543] dnf5[30]: [236/278] Removing libffi-0:3.4.6-3.fc4 100% | 8.8 KiB/s | 9.0 B | 00m00s [434187.479199] dnf5[30]: [237/278] Removing mpfr-0:4.2.1-5.fc41. 100% | 10.7 KiB/s | 11.0 B | 00m00s [434187.480799] dnf5[30]: [238/278] Removing gmp-1:6.3.0-2.fc41.x 100% | 14.6 KiB/s | 15.0 B | 00m00s [434187.482274] dnf5[30]: [239/278] Removing gpm-libs-0:1.20.7-48 100% | 4.9 KiB/s | 5.0 B | 00m00s [434187.483911] dnf5[30]: [240/278] Removing libsodium-0:1.0.20-2 100% | 6.8 KiB/s | 7.0 B | 00m00s [434187.485740] dnf5[30]: [241/278] Removing libpipeline-0:1.5.7- 100% | 9.8 KiB/s | 10.0 B | 00m00s [434187.487434] dnf5[30]: [242/278] Removing libtasn1-0:4.20.0-1. 100% | 11.7 KiB/s | 12.0 B | 00m00s [434187.489007] dnf5[30]: [243/278] Removing zlib-ng-compat-0:2.2 100% | 4.9 KiB/s | 5.0 B | 00m00s [434187.490558] dnf5[30]: [244/278] Removing libcom_err-0:1.47.1- 100% | 6.8 KiB/s | 7.0 B | 00m00s [434187.492590] dnf5[30]: [245/278] Removing libidn2-0:2.3.8-1.fc 100% | 19.5 KiB/s | 40.0 B | 00m00s [434187.494243] dnf5[30]: [246/278] Removing libunistring-0:1.1-8 100% | 11.7 KiB/s | 12.0 B | 00m00s [434187.495780] dnf5[30]: [247/278] Removing libnghttp2-0:1.62.1- 100% | 5.9 KiB/s | 6.0 B | 00m00s [434187.497534] dnf5[30]: [248/278] Removing keyutils-libs-0:1.6. 100% | 8.8 KiB/s | 9.0 B | 00m00s [434187.499271] dnf5[30]: [249/278] Removing libverto-0:0.3.2-9.f 100% | 11.7 KiB/s | 12.0 B | 00m00s [434187.500959] dnf5[30]: [250/278] Removing bzip2-libs-0:1.0.8-1 100% | 6.8 KiB/s | 7.0 B | 00m00s [434187.502672] dnf5[30]: [251/278] Removing libyaml-0:0.2.5-15.f 100% | 8.8 KiB/s | 9.0 B | 00m00s [434187.504583] dnf5[30]: [252/278] Removing libattr-0:2.5.2-4.fc 100% | 5.9 KiB/s | 6.0 B | 00m00s [434187.509839] dnf5[30]: [253/278] Removing lz4-libs-0:1.10.0-1. 100% | 1.4 KiB/s | 7.0 B | 00m00s [434187.514962] dnf5[30]: [254/278] Removing dbus-common-1:1.14.1 100% | 3.1 KiB/s | 16.0 B | 00m00s [434187.517861] dnf5[30]: [255/278] Removing crypto-policies-0:20 100% | 92.8 KiB/s | 190.0 B | 00m00s [434187.519396] dnf5[30]: [256/278] Removing pcre2-syntax-0:10.44 100% | 15.6 KiB/s | 16.0 B | 00m00s [434187.522369] dnf5[30]: [257/278] Removing coreutils-common-0:9 100% | 123.0 KiB/s | 252.0 B | 00m00s [434187.523515] dnf5[30]: [258/278] Removing publicsuffix-list-da 100% | 3.9 KiB/s | 4.0 B | 00m00s [434187.524737] dnf5[30]: [259/278] Removing libssh-config-0:0.10 100% | 2.9 KiB/s | 3.0 B | 00m00s [434187.525926] dnf5[30]: [260/278] Removing vim-data-2:9.1.1275- 100% | 6.8 KiB/s | 7.0 B | 00m00s [434187.528698] dnf5[30]: [261/278] Removing vim-filesystem-2:9.1 100% | 16.1 KiB/s | 33.0 B | 00m00s [434187.528792] dnf5[30]: warning: posix.fork(): .fork(), .exec(), .wait() and .redirect2null() are deprecated, use rpm.spawn() or rpm.execute() instead [434187.529788] dnf5[30]: warning: posix.wait(): .fork(), .exec(), .wait() and .redirect2null() are deprecated, use rpm.spawn() or rpm.execute() instead [434187.529968] dnf5[1060]: warning: posix.exec(): .fork(), .exec(), .wait() and .redirect2null() are deprecated, use rpm.spawn() or rpm.execute() instead [434187.540840] dnf5[30]: [262/278] Removing libgcc-0:14.2.1-7.fc 100% | 1.5 KiB/s | 11.0 B | 00m00s [434187.547552] dnf5[30]: [263/278] Removing glibc-gconv-extra-0: 100% | 62.0 KiB/s | 635.0 B | 00m00s [434187.556500] dnf5[30]: [264/278] Removing ncurses-libs-0:6.5-2 100% | 4.9 KiB/s | 45.0 B | 00m00s [434187.569946] dnf5[30]: [265/278] Removing bash-0:5.2.32-1.fc41 100% | 10.8 KiB/s | 133.0 B | 00m00s [434187.578143] dnf5[30]: [266/278] Removing glibc-0:2.40-24.fc41 100% | 12.5 KiB/s | 102.0 B | 00m00s [434187.588933] dnf5[30]: [267/278] Removing glibc-minimal-langpa 100% | 9.8 KiB/s | 100.0 B | 00m00s [434187.599973] dnf5[30]: Symlinking /usr/sbin/zic->/usr/bin/zic [434187.606719] dnf5[30]: [268/278] Removing glibc-common-0:2.40- 100% | 2.9 KiB/s | 53.0 B | 00m00s [434187.608279] dnf5[30]: [269/278] Removing basesystem-0:11-21.f 100% | 97.7 KiB/s | 100.0 B | 00m00s [434187.625759] dnf5[30]: [270/278] Removing fedora-gpg-keys-0:41 100% | 16.8 KiB/s | 292.0 B | 00m00s [434187.656210] dnf5[30]: [271/278] Removing filesystem-0:3.18-23 100% | 563.2 KiB/s | 16.9 KiB | 00m00s [434187.658460] dnf5[30]: [272/278] Removing setup-0:2.15.0-8.fc4 100% | 22.5 KiB/s | 46.0 B | 00m00s [434187.659769] dnf5[30]: [273/278] Removing fedora-repos-0:41-1. 100% | 4.9 KiB/s | 5.0 B | 00m00s [434187.660954] dnf5[30]: [274/278] Removing fedora-release-0:41- 100% | 97.7 KiB/s | 100.0 B | 00m00s [434187.663359] dnf5[30]: [275/278] Removing fedora-release-commo 100% | 14.2 KiB/s | 29.0 B | 00m00s [434187.664844] dnf5[30]: [276/278] Removing fedora-release-ident 100% | 1.0 KiB/s | 1.0 B | 00m00s [434187.667127] dnf5[30]: [277/278] Removing ncurses-base-0:6.5-2 100% | 87.4 KiB/s | 179.0 B | 00m00s [434187.667201] dnf5[30]: warning: posix.fork(): .fork(), .exec(), .wait() and .redirect2null() are deprecated, use rpm.spawn() or rpm.execute() instead [434187.668112] dnf5[30]: warning: posix.wait(): .fork(), .exec(), .wait() and .redirect2null() are deprecated, use rpm.spawn() or rpm.execute() instead [434187.669300] dnf5[1062]: warning: posix.exec(): .fork(), .exec(), .wait() and .redirect2null() are deprecated, use rpm.spawn() or rpm.execute() instead [434187.682613] dnf5[30]: /usr/sbin cannot be merged, found /usr/sbin/iconvconfig [434187.682696] dnf5[30]: All files under /usr/local/sbin are symlinks; linking to ./bin... [434187.682720] dnf5[30]: ...done Enough people have reported the problem with /usr/sbin/iconvconfig so we know that this is not a fluke. But I don't understand what is special about that file so it doesn't get replaced by a symlink in some upgrades.
> (But there's some strange behaviour, because rpm both knows and doesn't know > about the file: > > $ rpm -ql glibc-2.41-3.fc42.x86_64|grep bin/ldconfig > /usr/bin/ldconfig > $ rpm -qf /usr/sbin/ldconfig > glibc-2.41-3.fc42.x86_64 > > But rpm lists a few other symlinks in /usr/sbin as owned by a package, > so maybe that's OK.) OK, this is expected. 'rpm -qf' will show the package as owned if there's a provides with the given path. glibc has Provides:/usr/sbin/ldconfig and Provides:/sbin/ldconfig, and the actual file at /usr/bin/ldconfig, so 'rpm -qf' will report any of those paths as owned by glibc. So that is all fine. The question is why those two files don't get replaced by a symlink as expected.
This seems to be a bug in rpm. When the file is owned by two arch packages, after the upgrade, we're left with a stale file. A reproducer that works reliably for me: sudo dnf -y --use-host-config --releasever=41 --installroot=/var/lib/machines/f41 --repo=fedora --repo=updates --setopt=install_weak_deps=0 install passwd dnf fedora-release util-linux systemd authselect glibc.i686 sudo sed -i -r "s|root:\*:|root:$(openssl passwd xx):|" /var/lib/machines/f41/etc/shadow sudo systemd-nspawn -D /var/lib/machines/f41 -b (in the container) dnf system-upgrade download --releasever=42 --exclude=[^fg]* --exclude=g[^l]* --exclude=findutils --exclude=fmt --exclude=glib2 -y (in the container) dnf offline reboot -y (in the container) ls -Ul /usr/*/ldconfig /usr/*/iconvconfig -rwxr-xr-x 1 root root 1019072 Apr 14 02:00 /usr/bin/ldconfig -rwxr-xr-x 1 root root 1023376 Apr 8 02:00 /usr/sbin/ldconfig -rwxr-xr-x 1 root root 32552 Apr 14 02:00 /usr/bin/iconvconfig -rwxr-xr-x 1 root root 36816 Apr 8 02:00 /usr/sbin/iconvconfig The required condition is that glibc.x86_64 and glibc.i686 need to be both installed. Without the foreign-arch package, there is no problem. I'd like to have this diagnosed from the rpm side. If it turns out that the behaviour is expected and/or unfixable, we can add an ad-hoc workaround in glibc or one of the other packages… But let's figure out *why* this happens first.
Sounds like this https://github.com/rpm-software-management/rpm/issues/1800 - so it's been there for a while. Nobody's been too eager to touch it after a failed attempt that caused a worse regression...
Yeah, sounds like the same issue. Is there any chance of this being fixed in rpm? Adding a work-around scriptlet in individual packages seems rather ugly. I think we'll need a targeted work-around scriptlet to help with the upgrades to F42. Many people reported on fedora-devel and in other places that they're getting /usr/sbin/iconvconfig and /usr/sbin/ldconfig. glibc is one of those packages that is the most likely to have foreign-arch installs. But even if we fixed rpm, then people who upgraded previously would be affected. I'll try to extend the scriptlets in filesystem.rpm to cover this. But I hope that we can fix rpm long-term.
Of course this ultimately needs fixing in rpm, and the sbin merge giving it way more exposure does raise the priority somewhat. We're pretty booked up to 6.0 beta but we can at least *try* to fix this in time for 6.0 rc (and then backport).
I think we need a solution for this. And even if rpm is fixed, we'll still have the stale files on systems which were upgraded previously. Two pull requests for glibc: https://src.fedoraproject.org/rpms/glibc/pull-request/144 https://src.fedoraproject.org/rpms/glibc/pull-request/145 PTAL.
FEDORA-2025-8127a1e7d6 (glibc-2.41-6.fc42) has been submitted as an update to Fedora 42. https://bodhi.fedoraproject.org/updates/FEDORA-2025-8127a1e7d6
FEDORA-2025-8127a1e7d6 has been pushed to the Fedora 42 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-8127a1e7d6` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-8127a1e7d6 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2025-8127a1e7d6 (glibc-2.41-6.fc42 and rpm-ostree-2025.8-2.fc42) has been pushed to the Fedora 42 stable repository. If problem still persists, please make note of it in this bug report.