Description of problem: I have followed the fedora upgrade instructions at https://docs.fedoraproject.org/en-US/quick-docs/dnf-system-upgrade/ and in the end they always end up breaking the consistency of some rpm packages When executing the symlinks -r -d /usr a fair number of rpms break. State immediately before the symlinks step ``` [root@minerva ~]# rpm -qa --verify S.5....T. c /etc/krb5.conf S.5....T. c /etc/samba/smb.conf .M....... g /var/lib/selinux/targeted/active/modules/200/smartmon .......T. c /etc/pcp/pmcd/pmcd.conf S.5....T. c /etc/chrony.conf S.5....T. /var/log/salt/minion SM5....T. c /etc/salt/minion .M....... n /etc/salt/pki/minion .M....... /var/lib/samba/sysvol .....UG.. g /var/lib/nfs/statd/state S.5....T. c /etc/ssh/sshd_config .M....... c /boot/grub2/grub.cfg S.5....T. c /root/.bashrc ``` Finding dangling symlinks... ``` [root@minerva ~]# symlinks -r /usr | grep dangling dangling: /usr/lib/.build-id/c1/7bd973c1e316bfee4f4aa5ac16830a5dc3bcd3 -> ../../../../usr/sbin/quota_nld dangling: /usr/lib/.build-id/e3/dab686192ee682c642dc1a1e3e894e88a5af63 -> ../../../../usr/sbin/warnquota dangling: /usr/lib/.build-id/55/486e343b14b6f6badd7275b014dd82f44680d3 -> ../../../../usr/lib64/libncurses++.so.6.3 dangling: /usr/lib/.build-id/6a/d960b89a4448d76ead4892524133af065748b2 -> ../../../../usr/sbin/rpc.rquotad dangling: /usr/lib/.build-id/7e/36f7119010825fb025ca944a61801662c7aa11 -> ../../../../usr/bin/coreutils.single dangling: /usr/lib/.build-id/a1/d1843eed661509d750d69baef2a61e1fbf19e7 -> ../../../../usr/sbin/tc dangling: /usr/lib/.build-id/18/186f3e1b1b577a19b2f1f296dbc2e41b6970f2 -> ../../../../usr/lib64/libncurses++w.so.6.3 dangling: /usr/lib/.build-id/67/0f5d8033ec23ca4777aeca9ff236608e5e11af -> ../../../../usr/sbin/grub2-sparc64-setup dangling: /usr/lib/.build-id/67/66146da07119be1e86dfcf4aa867f5e30bea43 -> ../../../../usr/sbin/xfs_scrub dangling: /usr/lib/.build-id/2f/767bee207dca37dc5105a599722d6f2f4106f3 -> ../../../../usr/sbin/grub2-ofpathname dangling: /usr/lib/firmware/qcom/LENOVO/21BX.xz -> ../sc8280xp/LENOVO/21BX.xz dangling: /usr/lib/modules/6.2.7-200.fc37.x86_64/build -> /usr/src/kernels/6.2.7-200.fc37.x86_64 dangling: /usr/lib/modules/6.2.7-200.fc37.x86_64/source -> build dangling: /usr/share/doc/hunspell/README -> README.md dangling: /usr/share/cockpit/branding/arch/apple-touch-icon.png -> ../../../pixmaps/archlinux-logo.png dangling: /usr/share/cockpit/branding/arch/favicon.ico -> ../../../pixmaps/archlinux-logo.png dangling: /usr/share/cockpit/branding/arch/logo.png -> ../../../pixmaps/archlinux-logo.png dangling: /usr/share/cockpit/branding/debian/favicon.ico -> ../../../pixmaps/debian-logo.png dangling: /usr/share/cockpit/branding/debian/logo.png -> ../../../pixmaps/debian-logo.png dangling: /usr/share/cockpit/branding/opensuse/default-1920x1200.jpg -> ../../../wallpapers/default-1920x1200.jpg dangling: /usr/share/cockpit/branding/opensuse/favicon.ico -> ../../../pixmaps/distribution-logos/favicon.ico dangling: /usr/share/cockpit/branding/opensuse/square-hicolor.svg -> ../../../pixmaps/distribution-logos/square-hicolor.svg dangling: /usr/share/cockpit/branding/ubuntu/logo.png -> ../../../plymouth/ubuntu-logo.png [root@minerva ~]# ls -l /usr/lib/.build-id/c1/7bd973c1e316bfee4f4aa5ac16830a5dc3bcd3 lrwxrwxrwx. 1 root root 30 Jul 23 2022 /usr/lib/.build-id/c1/7bd973c1e316bfee4f4aa5ac16830a5dc3bcd3 -> ../../../../usr/sbin/quota_nld ``` Yes, they appear to be dangling when inspected Follow instructions... ``` [root@minerva ~]# symlinks -r -d /usr absolute: /usr/bin/ex -> /usr/libexec/vi absolute: /usr/bin/rvi -> /usr/libexec/vi absolute: /usr/bin/rview -> /usr/libexec/vi absolute: /usr/bin/soelim -> /etc/alternatives/soelim messy: /usr/bin/pydoc -> ./pydoc3 absolute: /usr/bin/abrt-cli -> /usr/bin/abrt absolute: /usr/bin/whois -> /etc/alternatives/whois absolute: /usr/bin/apropos -> /etc/alternatives/apropos absolute: /usr/bin/whatis -> /etc/alternatives/whatis absolute: /usr/bin/nc -> /etc/alternatives/nc absolute: /usr/bin/man -> /etc/alternatives/man absolute: /usr/bin/yum-debug-restore -> /usr/libexec/dnf-utils absolute: /usr/bin/find-repos-of-install -> /usr/libexec/dnf-utils absolute: /usr/bin/repomanage -> /usr/libexec/dnf-utils absolute: /usr/bin/clevis-decrypt-tpm2plus -> /usr/bin/clevis-pin-tpm2 absolute: /usr/bin/yum-groups-manager -> /usr/libexec/dnf-utils absolute: /usr/bin/yumdownloader -> /usr/libexec/dnf-utils absolute: /usr/bin/package-cleanup -> /usr/libexec/dnf-utils absolute: /usr/bin/repotrack -> /usr/libexec/dnf-utils absolute: /usr/bin/yum-builddep -> /usr/libexec/dnf-utils absolute: /usr/bin/yum-config-manager -> /usr/libexec/dnf-utils absolute: /usr/bin/repoclosure -> /usr/libexec/dnf-utils absolute: /usr/bin/yum-debug-dump -> /usr/libexec/dnf-utils absolute: /usr/bin/debuginfo-install -> /usr/libexec/dnf-utils absolute: /usr/bin/repo-graph -> /usr/libexec/dnf-utils absolute: /usr/bin/repoquery -> /usr/libexec/dnf-utils absolute: /usr/bin/repodiff -> /usr/libexec/dnf-utils absolute: /usr/bin/needs-restarting -> /usr/libexec/dnf-utils absolute: /usr/bin/reposync -> /usr/libexec/dnf-utils messy: /usr/bin/python -> ./python3 absolute: /usr/bin/clevis-encrypt-tpm2plus -> /usr/bin/clevis-pin-tpm2 absolute: /usr/sbin/ipset-translate -> /usr/sbin/ipset absolute: /usr/sbin/pidof -> /usr/bin/pidof absolute: /usr/sbin/iptables-save -> /etc/alternatives/iptables-save absolute: /usr/sbin/arptables -> /etc/alternatives/arptables absolute: /usr/sbin/ip6tables-save -> /etc/alternatives/ip6tables-save absolute: /usr/sbin/arptables-save -> /etc/alternatives/arptables-save absolute: /usr/sbin/iptables-restore -> /etc/alternatives/iptables-restore absolute: /usr/sbin/ebtables-restore -> /etc/alternatives/ebtables-restore absolute: /usr/sbin/ebtables -> /etc/alternatives/ebtables absolute: /usr/sbin/ifup -> /etc/alternatives/ifup absolute: /usr/sbin/arptables-restore -> /etc/alternatives/arptables-restore absolute: /usr/sbin/iptables -> /etc/alternatives/iptables absolute: /usr/sbin/ip6tables -> /etc/alternatives/ip6tables absolute: /usr/sbin/ip6tables-restore -> /etc/alternatives/ip6tables-restore absolute: /usr/sbin/ebtables-save -> /etc/alternatives/ebtables-save absolute: /usr/sbin/mkfs.ntfs -> /usr/sbin/mkntfs absolute: /usr/sbin/ifdown -> /etc/alternatives/ifdown absolute: /usr/sbin/mount.lowntfs-3g -> /usr/bin/lowntfs-3g absolute: /usr/sbin/mount.ntfs-3g -> /usr/bin/ntfs-3g dangling: /usr/lib/.build-id/c1/7bd973c1e316bfee4f4aa5ac16830a5dc3bcd3 -> ../../../../usr/sbin/quota_nld deleted: /usr/lib/.build-id/c1/7bd973c1e316bfee4f4aa5ac16830a5dc3bcd3 -> ../../../../usr/sbin/quota_nld dangling: /usr/lib/.build-id/e3/dab686192ee682c642dc1a1e3e894e88a5af63 -> ../../../../usr/sbin/warnquota deleted: /usr/lib/.build-id/e3/dab686192ee682c642dc1a1e3e894e88a5af63 -> ../../../../usr/sbin/warnquota dangling: /usr/lib/.build-id/55/486e343b14b6f6badd7275b014dd82f44680d3 -> ../../../../usr/lib64/libncurses++.so.6.3 deleted: /usr/lib/.build-id/55/486e343b14b6f6badd7275b014dd82f44680d3 -> ../../../../usr/lib64/libncurses++.so.6.3 dangling: /usr/lib/.build-id/6a/d960b89a4448d76ead4892524133af065748b2 -> ../../../../usr/sbin/rpc.rquotad deleted: /usr/lib/.build-id/6a/d960b89a4448d76ead4892524133af065748b2 -> ../../../../usr/sbin/rpc.rquotad dangling: /usr/lib/.build-id/7e/36f7119010825fb025ca944a61801662c7aa11 -> ../../../../usr/bin/coreutils.single deleted: /usr/lib/.build-id/7e/36f7119010825fb025ca944a61801662c7aa11 -> ../../../../usr/bin/coreutils.single dangling: /usr/lib/.build-id/a1/d1843eed661509d750d69baef2a61e1fbf19e7 -> ../../../../usr/sbin/tc deleted: /usr/lib/.build-id/a1/d1843eed661509d750d69baef2a61e1fbf19e7 -> ../../../../usr/sbin/tc dangling: /usr/lib/.build-id/18/186f3e1b1b577a19b2f1f296dbc2e41b6970f2 -> ../../../../usr/lib64/libncurses++w.so.6.3 deleted: /usr/lib/.build-id/18/186f3e1b1b577a19b2f1f296dbc2e41b6970f2 -> ../../../../usr/lib64/libncurses++w.so.6.3 dangling: /usr/lib/.build-id/67/0f5d8033ec23ca4777aeca9ff236608e5e11af -> ../../../../usr/sbin/grub2-sparc64-setup deleted: /usr/lib/.build-id/67/0f5d8033ec23ca4777aeca9ff236608e5e11af -> ../../../../usr/sbin/grub2-sparc64-setup dangling: /usr/lib/.build-id/67/66146da07119be1e86dfcf4aa867f5e30bea43 -> ../../../../usr/sbin/xfs_scrub deleted: /usr/lib/.build-id/67/66146da07119be1e86dfcf4aa867f5e30bea43 -> ../../../../usr/sbin/xfs_scrub dangling: /usr/lib/.build-id/2f/767bee207dca37dc5105a599722d6f2f4106f3 -> ../../../../usr/sbin/grub2-ofpathname deleted: /usr/lib/.build-id/2f/767bee207dca37dc5105a599722d6f2f4106f3 -> ../../../../usr/sbin/grub2-ofpathname dangling: /usr/lib/firmware/qcom/LENOVO/21BX.xz -> ../sc8280xp/LENOVO/21BX.xz deleted: /usr/lib/firmware/qcom/LENOVO/21BX.xz -> ../sc8280xp/LENOVO/21BX.xz dangling: /usr/lib/modules/6.2.7-200.fc37.x86_64/build -> /usr/src/kernels/6.2.7-200.fc37.x86_64 deleted: /usr/lib/modules/6.2.7-200.fc37.x86_64/build -> /usr/src/kernels/6.2.7-200.fc37.x86_64 dangling: /usr/lib/modules/6.2.7-200.fc37.x86_64/source -> build deleted: /usr/lib/modules/6.2.7-200.fc37.x86_64/source -> build absolute: /usr/lib/python3.11/site-packages/urllib3/packages/__pycache__/six.cpython-311.opt-1.pyc -> /usr/lib/python3.11/site-packages/__pycache__/six.cpython-311.opt-1.pyc absolute: /usr/lib/python3.11/site-packages/urllib3/packages/__pycache__/six.cpython-311.pyc -> /usr/lib/python3.11/site-packages/__pycache__/six.cpython-311.pyc absolute: /usr/lib/python3.11/site-packages/urllib3/packages/six.py -> /usr/lib/python3.11/site-packages/six.py absolute: /usr/lib64/libnssckbi.so -> /etc/alternatives/libnssckbi.so.x86_64 absolute: /usr/lib64/libwbclient.so.0.15 -> /etc/alternatives/libwbclient.so.0.15-64 dangling: /usr/share/doc/hunspell/README -> README.md deleted: /usr/share/doc/hunspell/README -> README.md absolute: /usr/share/doc/python-systemd/html/_static/jquery.js -> /usr/share/javascript/jquery/latest/jquery.min.js absolute: /usr/share/doc/atmel-firmware/COPYING -> /lib/firmware/COPYING.atmel-firmware absolute: /usr/share/doc/atmel-firmware/COPYRIGHT-usb -> /lib/firmware/COPYRIGHT-usb.atmel-firmware absolute: /usr/share/doc/atmel-firmware/README -> /lib/firmware/README.atmel-firmware absolute: /usr/share/doc/atmel-firmware/README-usb -> /lib/firmware/README-usb.atmel-firmware absolute: /usr/share/man/man7/man.7.gz -> /etc/alternatives/man.7.gz absolute: /usr/share/man/man1/soelim.1.gz -> /etc/alternatives/soelim.1.gz absolute: /usr/share/man/man1/whatis.1.gz -> /etc/alternatives/whatis.1.gz absolute: /usr/share/man/man1/abrt-cli.1.gz -> /usr/share/man/man1/abrt.1.gz absolute: /usr/share/man/man1/man.1.gz -> /etc/alternatives/man.1.gz absolute: /usr/share/man/man1/apropos.1.gz -> /etc/alternatives/apropos.1.gz absolute: /usr/share/man/man1/nc.1.gz -> /etc/alternatives/nc-man absolute: /usr/share/man/man1/repoquery.1.gz -> /usr/share/man/man1/dnf-utils.1.gz absolute: /usr/share/man/man1/repotrack.1.gz -> /usr/share/man/man1/dnf-utils.1.gz absolute: /usr/share/man/man1/find-repos-of-install.1.gz -> /usr/share/man/man1/dnf-utils.1.gz messy: /usr/share/man/man1/python.1.gz -> ./python3.1.gz absolute: /usr/share/man/man1/whois.1.gz -> /etc/alternatives/whois-man absolute: /usr/share/man/man8/ebtables.8.gz -> /etc/alternatives/ebtables-man absolute: /usr/share/man/man8/arptables-restore.8.gz -> /etc/alternatives/arptables-restore-man absolute: /usr/share/man/man8/arptables.8.gz -> /etc/alternatives/arptables-man absolute: /usr/share/man/man8/arptables-save.8.gz -> /etc/alternatives/arptables-save-man absolute: /usr/share/man/man8/dnf-offline-distrosync.8.gz -> /usr/share/man/man8/dnf-system-upgrade.8.gz absolute: /usr/share/man/man8/dnf-offline-upgrade.8.gz -> /usr/share/man/man8/dnf-system-upgrade.8.gz absolute: /usr/share/gawk -> /usr/share/awk dangling: /usr/share/cockpit/branding/arch/apple-touch-icon.png -> ../../../pixmaps/archlinux-logo.png deleted: /usr/share/cockpit/branding/arch/apple-touch-icon.png -> ../../../pixmaps/archlinux-logo.png dangling: /usr/share/cockpit/branding/arch/favicon.ico -> ../../../pixmaps/archlinux-logo.png deleted: /usr/share/cockpit/branding/arch/favicon.ico -> ../../../pixmaps/archlinux-logo.png dangling: /usr/share/cockpit/branding/arch/logo.png -> ../../../pixmaps/archlinux-logo.png deleted: /usr/share/cockpit/branding/arch/logo.png -> ../../../pixmaps/archlinux-logo.png dangling: /usr/share/cockpit/branding/debian/favicon.ico -> ../../../pixmaps/debian-logo.png deleted: /usr/share/cockpit/branding/debian/favicon.ico -> ../../../pixmaps/debian-logo.png dangling: /usr/share/cockpit/branding/debian/logo.png -> ../../../pixmaps/debian-logo.png deleted: /usr/share/cockpit/branding/debian/logo.png -> ../../../pixmaps/debian-logo.png dangling: /usr/share/cockpit/branding/opensuse/default-1920x1200.jpg -> ../../../wallpapers/default-1920x1200.jpg deleted: /usr/share/cockpit/branding/opensuse/default-1920x1200.jpg -> ../../../wallpapers/default-1920x1200.jpg dangling: /usr/share/cockpit/branding/opensuse/favicon.ico -> ../../../pixmaps/distribution-logos/favicon.ico deleted: /usr/share/cockpit/branding/opensuse/favicon.ico -> ../../../pixmaps/distribution-logos/favicon.ico dangling: /usr/share/cockpit/branding/opensuse/square-hicolor.svg -> ../../../pixmaps/distribution-logos/square-hicolor.svg deleted: /usr/share/cockpit/branding/opensuse/square-hicolor.svg -> ../../../pixmaps/distribution-logos/square-hicolor.svg dangling: /usr/share/cockpit/branding/ubuntu/logo.png -> ../../../plymouth/ubuntu-logo.png deleted: /usr/share/cockpit/branding/ubuntu/logo.png -> ../../../plymouth/ubuntu-logo.png absolute: /usr/libexec/gawk -> /usr/libexec/awk absolute: /usr/libexec/arptables-helper -> /etc/alternatives/arptables-helper absolute: /usr/libexec/samba/cups_backend_smb -> /etc/alternatives/cups_backend_smb ``` Some rpms are now inconsistent ``` [root@minerva ~]# rpm -qa --verify missing /usr/lib/firmware/qcom/LENOVO/21BX.xz missing a /usr/lib/.build-id/18/186f3e1b1b577a19b2f1f296dbc2e41b6970f2 missing a /usr/lib/.build-id/55/486e343b14b6f6badd7275b014dd82f44680d3 missing d /usr/share/doc/hunspell/README missing a /usr/lib/.build-id/67/66146da07119be1e86dfcf4aa867f5e30bea43 missing a /usr/lib/.build-id/a1/d1843eed661509d750d69baef2a61e1fbf19e7 missing a /usr/lib/.build-id/7e/36f7119010825fb025ca944a61801662c7aa11 S.5....T. c /etc/krb5.conf missing a /usr/lib/.build-id/2f/767bee207dca37dc5105a599722d6f2f4106f3 missing a /usr/lib/.build-id/67/0f5d8033ec23ca4777aeca9ff236608e5e11af S.5....T. c /etc/samba/smb.conf missing /lib/modules/6.2.7-200.fc37.x86_64/build missing /lib/modules/6.2.7-200.fc37.x86_64/source .M....... g /var/lib/selinux/targeted/active/modules/200/smartmon .......T. c /etc/pcp/pmcd/pmcd.conf S.5....T. c /etc/chrony.conf missing a /usr/lib/.build-id/6a/d960b89a4448d76ead4892524133af065748b2 missing a /usr/lib/.build-id/c1/7bd973c1e316bfee4f4aa5ac16830a5dc3bcd3 missing a /usr/lib/.build-id/e3/dab686192ee682c642dc1a1e3e894e88a5af63 missing /usr/share/cockpit/branding/arch/apple-touch-icon.png missing /usr/share/cockpit/branding/arch/favicon.ico missing /usr/share/cockpit/branding/arch/logo.png missing /usr/share/cockpit/branding/debian/favicon.ico missing /usr/share/cockpit/branding/debian/logo.png missing /usr/share/cockpit/branding/opensuse/default-1920x1200.jpg missing /usr/share/cockpit/branding/opensuse/favicon.ico missing /usr/share/cockpit/branding/opensuse/square-hicolor.svg missing /usr/share/cockpit/branding/ubuntu/logo.png S.5....T. /var/log/salt/minion SM5....T. c /etc/salt/minion .M....... n /etc/salt/pki/minion .M....... /var/lib/samba/sysvol missing a /usr/lib/.build-id/2f/767bee207dca37dc5105a599722d6f2f4106f3 missing a /usr/lib/.build-id/67/0f5d8033ec23ca4777aeca9ff236608e5e11af .....UG.. g /var/lib/nfs/statd/state S.5....T. c /etc/ssh/sshd_config .M....... c /boot/grub2/grub.cfg S.5....T. c /root/.bashrc ``` Version-Release number of selected component (if applicable): Fedora upgrade to 35, 36 and 37 (at least) How reproducible: Very, very, very often Steps to Reproduce: 1. Use fedora on a VM for a while 2. Follow instructions at https://docs.fedoraproject.org/en-US/quick-docs/dnf-system-upgrade/ 3. rpm -qa --verify now show broken contents Actual results: rpm -qa --verify shows broken packages Expected results: Successful upgrade with consistent rpms Additional info: