Bug 2181534

Summary: Running symlinks -r -d /usr after dnf-system-upgrade breaks packages
Product: [Fedora] Fedora Reporter: Micke <u23104331>
Component: symlinksAssignee: Tim Waugh <twaugh>
Status: NEW --- QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 37CC: twaugh
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: ---
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Micke 2023-03-24 12:59:10 UTC 
Description of problem:

I have followed the fedora upgrade instructions at https://docs.fedoraproject.org/en-US/quick-docs/dnf-system-upgrade/ and in the end they always end up breaking the consistency of some rpm packages

When executing the symlinks -r -d /usr  a fair number of rpms break.


State immediately before the symlinks step
```
[root@minerva ~]# rpm -qa --verify
S.5....T.  c /etc/krb5.conf
S.5....T.  c /etc/samba/smb.conf
.M.......  g /var/lib/selinux/targeted/active/modules/200/smartmon
.......T.  c /etc/pcp/pmcd/pmcd.conf
S.5....T.  c /etc/chrony.conf
S.5....T.    /var/log/salt/minion
SM5....T.  c /etc/salt/minion
.M.......  n /etc/salt/pki/minion
.M.......    /var/lib/samba/sysvol
.....UG..  g /var/lib/nfs/statd/state
S.5....T.  c /etc/ssh/sshd_config
.M.......  c /boot/grub2/grub.cfg
S.5....T.  c /root/.bashrc
```

Finding dangling symlinks...
```
[root@minerva ~]# symlinks -r /usr | grep dangling
dangling: /usr/lib/.build-id/c1/7bd973c1e316bfee4f4aa5ac16830a5dc3bcd3 -> ../../../../usr/sbin/quota_nld
dangling: /usr/lib/.build-id/e3/dab686192ee682c642dc1a1e3e894e88a5af63 -> ../../../../usr/sbin/warnquota
dangling: /usr/lib/.build-id/55/486e343b14b6f6badd7275b014dd82f44680d3 -> ../../../../usr/lib64/libncurses++.so.6.3
dangling: /usr/lib/.build-id/6a/d960b89a4448d76ead4892524133af065748b2 -> ../../../../usr/sbin/rpc.rquotad
dangling: /usr/lib/.build-id/7e/36f7119010825fb025ca944a61801662c7aa11 -> ../../../../usr/bin/coreutils.single
dangling: /usr/lib/.build-id/a1/d1843eed661509d750d69baef2a61e1fbf19e7 -> ../../../../usr/sbin/tc
dangling: /usr/lib/.build-id/18/186f3e1b1b577a19b2f1f296dbc2e41b6970f2 -> ../../../../usr/lib64/libncurses++w.so.6.3
dangling: /usr/lib/.build-id/67/0f5d8033ec23ca4777aeca9ff236608e5e11af -> ../../../../usr/sbin/grub2-sparc64-setup
dangling: /usr/lib/.build-id/67/66146da07119be1e86dfcf4aa867f5e30bea43 -> ../../../../usr/sbin/xfs_scrub
dangling: /usr/lib/.build-id/2f/767bee207dca37dc5105a599722d6f2f4106f3 -> ../../../../usr/sbin/grub2-ofpathname
dangling: /usr/lib/firmware/qcom/LENOVO/21BX.xz -> ../sc8280xp/LENOVO/21BX.xz
dangling: /usr/lib/modules/6.2.7-200.fc37.x86_64/build -> /usr/src/kernels/6.2.7-200.fc37.x86_64
dangling: /usr/lib/modules/6.2.7-200.fc37.x86_64/source -> build
dangling: /usr/share/doc/hunspell/README -> README.md
dangling: /usr/share/cockpit/branding/arch/apple-touch-icon.png -> ../../../pixmaps/archlinux-logo.png
dangling: /usr/share/cockpit/branding/arch/favicon.ico -> ../../../pixmaps/archlinux-logo.png
dangling: /usr/share/cockpit/branding/arch/logo.png -> ../../../pixmaps/archlinux-logo.png
dangling: /usr/share/cockpit/branding/debian/favicon.ico -> ../../../pixmaps/debian-logo.png
dangling: /usr/share/cockpit/branding/debian/logo.png -> ../../../pixmaps/debian-logo.png
dangling: /usr/share/cockpit/branding/opensuse/default-1920x1200.jpg -> ../../../wallpapers/default-1920x1200.jpg
dangling: /usr/share/cockpit/branding/opensuse/favicon.ico -> ../../../pixmaps/distribution-logos/favicon.ico
dangling: /usr/share/cockpit/branding/opensuse/square-hicolor.svg -> ../../../pixmaps/distribution-logos/square-hicolor.svg
dangling: /usr/share/cockpit/branding/ubuntu/logo.png -> ../../../plymouth/ubuntu-logo.png
[root@minerva ~]# ls -l /usr/lib/.build-id/c1/7bd973c1e316bfee4f4aa5ac16830a5dc3bcd3
lrwxrwxrwx. 1 root root 30 Jul 23  2022 /usr/lib/.build-id/c1/7bd973c1e316bfee4f4aa5ac16830a5dc3bcd3 -> ../../../../usr/sbin/quota_nld
```

Yes, they appear to be dangling when inspected

Follow instructions...
```
[root@minerva ~]# symlinks -r -d /usr
absolute: /usr/bin/ex -> /usr/libexec/vi
absolute: /usr/bin/rvi -> /usr/libexec/vi
absolute: /usr/bin/rview -> /usr/libexec/vi
absolute: /usr/bin/soelim -> /etc/alternatives/soelim
messy:    /usr/bin/pydoc -> ./pydoc3
absolute: /usr/bin/abrt-cli -> /usr/bin/abrt
absolute: /usr/bin/whois -> /etc/alternatives/whois
absolute: /usr/bin/apropos -> /etc/alternatives/apropos
absolute: /usr/bin/whatis -> /etc/alternatives/whatis
absolute: /usr/bin/nc -> /etc/alternatives/nc
absolute: /usr/bin/man -> /etc/alternatives/man
absolute: /usr/bin/yum-debug-restore -> /usr/libexec/dnf-utils
absolute: /usr/bin/find-repos-of-install -> /usr/libexec/dnf-utils
absolute: /usr/bin/repomanage -> /usr/libexec/dnf-utils
absolute: /usr/bin/clevis-decrypt-tpm2plus -> /usr/bin/clevis-pin-tpm2
absolute: /usr/bin/yum-groups-manager -> /usr/libexec/dnf-utils
absolute: /usr/bin/yumdownloader -> /usr/libexec/dnf-utils
absolute: /usr/bin/package-cleanup -> /usr/libexec/dnf-utils
absolute: /usr/bin/repotrack -> /usr/libexec/dnf-utils
absolute: /usr/bin/yum-builddep -> /usr/libexec/dnf-utils
absolute: /usr/bin/yum-config-manager -> /usr/libexec/dnf-utils
absolute: /usr/bin/repoclosure -> /usr/libexec/dnf-utils
absolute: /usr/bin/yum-debug-dump -> /usr/libexec/dnf-utils
absolute: /usr/bin/debuginfo-install -> /usr/libexec/dnf-utils
absolute: /usr/bin/repo-graph -> /usr/libexec/dnf-utils
absolute: /usr/bin/repoquery -> /usr/libexec/dnf-utils
absolute: /usr/bin/repodiff -> /usr/libexec/dnf-utils
absolute: /usr/bin/needs-restarting -> /usr/libexec/dnf-utils
absolute: /usr/bin/reposync -> /usr/libexec/dnf-utils
messy:    /usr/bin/python -> ./python3
absolute: /usr/bin/clevis-encrypt-tpm2plus -> /usr/bin/clevis-pin-tpm2
absolute: /usr/sbin/ipset-translate -> /usr/sbin/ipset
absolute: /usr/sbin/pidof -> /usr/bin/pidof
absolute: /usr/sbin/iptables-save -> /etc/alternatives/iptables-save
absolute: /usr/sbin/arptables -> /etc/alternatives/arptables
absolute: /usr/sbin/ip6tables-save -> /etc/alternatives/ip6tables-save
absolute: /usr/sbin/arptables-save -> /etc/alternatives/arptables-save
absolute: /usr/sbin/iptables-restore -> /etc/alternatives/iptables-restore
absolute: /usr/sbin/ebtables-restore -> /etc/alternatives/ebtables-restore
absolute: /usr/sbin/ebtables -> /etc/alternatives/ebtables
absolute: /usr/sbin/ifup -> /etc/alternatives/ifup
absolute: /usr/sbin/arptables-restore -> /etc/alternatives/arptables-restore
absolute: /usr/sbin/iptables -> /etc/alternatives/iptables
absolute: /usr/sbin/ip6tables -> /etc/alternatives/ip6tables
absolute: /usr/sbin/ip6tables-restore -> /etc/alternatives/ip6tables-restore
absolute: /usr/sbin/ebtables-save -> /etc/alternatives/ebtables-save
absolute: /usr/sbin/mkfs.ntfs -> /usr/sbin/mkntfs
absolute: /usr/sbin/ifdown -> /etc/alternatives/ifdown
absolute: /usr/sbin/mount.lowntfs-3g -> /usr/bin/lowntfs-3g
absolute: /usr/sbin/mount.ntfs-3g -> /usr/bin/ntfs-3g
dangling: /usr/lib/.build-id/c1/7bd973c1e316bfee4f4aa5ac16830a5dc3bcd3 -> ../../../../usr/sbin/quota_nld
deleted:  /usr/lib/.build-id/c1/7bd973c1e316bfee4f4aa5ac16830a5dc3bcd3 -> ../../../../usr/sbin/quota_nld
dangling: /usr/lib/.build-id/e3/dab686192ee682c642dc1a1e3e894e88a5af63 -> ../../../../usr/sbin/warnquota
deleted:  /usr/lib/.build-id/e3/dab686192ee682c642dc1a1e3e894e88a5af63 -> ../../../../usr/sbin/warnquota
dangling: /usr/lib/.build-id/55/486e343b14b6f6badd7275b014dd82f44680d3 -> ../../../../usr/lib64/libncurses++.so.6.3
deleted:  /usr/lib/.build-id/55/486e343b14b6f6badd7275b014dd82f44680d3 -> ../../../../usr/lib64/libncurses++.so.6.3
dangling: /usr/lib/.build-id/6a/d960b89a4448d76ead4892524133af065748b2 -> ../../../../usr/sbin/rpc.rquotad
deleted:  /usr/lib/.build-id/6a/d960b89a4448d76ead4892524133af065748b2 -> ../../../../usr/sbin/rpc.rquotad
dangling: /usr/lib/.build-id/7e/36f7119010825fb025ca944a61801662c7aa11 -> ../../../../usr/bin/coreutils.single
deleted:  /usr/lib/.build-id/7e/36f7119010825fb025ca944a61801662c7aa11 -> ../../../../usr/bin/coreutils.single
dangling: /usr/lib/.build-id/a1/d1843eed661509d750d69baef2a61e1fbf19e7 -> ../../../../usr/sbin/tc
deleted:  /usr/lib/.build-id/a1/d1843eed661509d750d69baef2a61e1fbf19e7 -> ../../../../usr/sbin/tc
dangling: /usr/lib/.build-id/18/186f3e1b1b577a19b2f1f296dbc2e41b6970f2 -> ../../../../usr/lib64/libncurses++w.so.6.3
deleted:  /usr/lib/.build-id/18/186f3e1b1b577a19b2f1f296dbc2e41b6970f2 -> ../../../../usr/lib64/libncurses++w.so.6.3
dangling: /usr/lib/.build-id/67/0f5d8033ec23ca4777aeca9ff236608e5e11af -> ../../../../usr/sbin/grub2-sparc64-setup
deleted:  /usr/lib/.build-id/67/0f5d8033ec23ca4777aeca9ff236608e5e11af -> ../../../../usr/sbin/grub2-sparc64-setup
dangling: /usr/lib/.build-id/67/66146da07119be1e86dfcf4aa867f5e30bea43 -> ../../../../usr/sbin/xfs_scrub
deleted:  /usr/lib/.build-id/67/66146da07119be1e86dfcf4aa867f5e30bea43 -> ../../../../usr/sbin/xfs_scrub
dangling: /usr/lib/.build-id/2f/767bee207dca37dc5105a599722d6f2f4106f3 -> ../../../../usr/sbin/grub2-ofpathname
deleted:  /usr/lib/.build-id/2f/767bee207dca37dc5105a599722d6f2f4106f3 -> ../../../../usr/sbin/grub2-ofpathname
dangling: /usr/lib/firmware/qcom/LENOVO/21BX.xz -> ../sc8280xp/LENOVO/21BX.xz
deleted:  /usr/lib/firmware/qcom/LENOVO/21BX.xz -> ../sc8280xp/LENOVO/21BX.xz
dangling: /usr/lib/modules/6.2.7-200.fc37.x86_64/build -> /usr/src/kernels/6.2.7-200.fc37.x86_64
deleted:  /usr/lib/modules/6.2.7-200.fc37.x86_64/build -> /usr/src/kernels/6.2.7-200.fc37.x86_64
dangling: /usr/lib/modules/6.2.7-200.fc37.x86_64/source -> build
deleted:  /usr/lib/modules/6.2.7-200.fc37.x86_64/source -> build
absolute: /usr/lib/python3.11/site-packages/urllib3/packages/__pycache__/six.cpython-311.opt-1.pyc -> /usr/lib/python3.11/site-packages/__pycache__/six.cpython-311.opt-1.pyc
absolute: /usr/lib/python3.11/site-packages/urllib3/packages/__pycache__/six.cpython-311.pyc -> /usr/lib/python3.11/site-packages/__pycache__/six.cpython-311.pyc
absolute: /usr/lib/python3.11/site-packages/urllib3/packages/six.py -> /usr/lib/python3.11/site-packages/six.py
absolute: /usr/lib64/libnssckbi.so -> /etc/alternatives/libnssckbi.so.x86_64
absolute: /usr/lib64/libwbclient.so.0.15 -> /etc/alternatives/libwbclient.so.0.15-64
dangling: /usr/share/doc/hunspell/README -> README.md
deleted:  /usr/share/doc/hunspell/README -> README.md
absolute: /usr/share/doc/python-systemd/html/_static/jquery.js -> /usr/share/javascript/jquery/latest/jquery.min.js
absolute: /usr/share/doc/atmel-firmware/COPYING -> /lib/firmware/COPYING.atmel-firmware
absolute: /usr/share/doc/atmel-firmware/COPYRIGHT-usb -> /lib/firmware/COPYRIGHT-usb.atmel-firmware
absolute: /usr/share/doc/atmel-firmware/README -> /lib/firmware/README.atmel-firmware
absolute: /usr/share/doc/atmel-firmware/README-usb -> /lib/firmware/README-usb.atmel-firmware
absolute: /usr/share/man/man7/man.7.gz -> /etc/alternatives/man.7.gz
absolute: /usr/share/man/man1/soelim.1.gz -> /etc/alternatives/soelim.1.gz
absolute: /usr/share/man/man1/whatis.1.gz -> /etc/alternatives/whatis.1.gz
absolute: /usr/share/man/man1/abrt-cli.1.gz -> /usr/share/man/man1/abrt.1.gz
absolute: /usr/share/man/man1/man.1.gz -> /etc/alternatives/man.1.gz
absolute: /usr/share/man/man1/apropos.1.gz -> /etc/alternatives/apropos.1.gz
absolute: /usr/share/man/man1/nc.1.gz -> /etc/alternatives/nc-man
absolute: /usr/share/man/man1/repoquery.1.gz -> /usr/share/man/man1/dnf-utils.1.gz
absolute: /usr/share/man/man1/repotrack.1.gz -> /usr/share/man/man1/dnf-utils.1.gz
absolute: /usr/share/man/man1/find-repos-of-install.1.gz -> /usr/share/man/man1/dnf-utils.1.gz
messy:    /usr/share/man/man1/python.1.gz -> ./python3.1.gz
absolute: /usr/share/man/man1/whois.1.gz -> /etc/alternatives/whois-man
absolute: /usr/share/man/man8/ebtables.8.gz -> /etc/alternatives/ebtables-man
absolute: /usr/share/man/man8/arptables-restore.8.gz -> /etc/alternatives/arptables-restore-man
absolute: /usr/share/man/man8/arptables.8.gz -> /etc/alternatives/arptables-man
absolute: /usr/share/man/man8/arptables-save.8.gz -> /etc/alternatives/arptables-save-man
absolute: /usr/share/man/man8/dnf-offline-distrosync.8.gz -> /usr/share/man/man8/dnf-system-upgrade.8.gz
absolute: /usr/share/man/man8/dnf-offline-upgrade.8.gz -> /usr/share/man/man8/dnf-system-upgrade.8.gz
absolute: /usr/share/gawk -> /usr/share/awk
dangling: /usr/share/cockpit/branding/arch/apple-touch-icon.png -> ../../../pixmaps/archlinux-logo.png
deleted:  /usr/share/cockpit/branding/arch/apple-touch-icon.png -> ../../../pixmaps/archlinux-logo.png
dangling: /usr/share/cockpit/branding/arch/favicon.ico -> ../../../pixmaps/archlinux-logo.png
deleted:  /usr/share/cockpit/branding/arch/favicon.ico -> ../../../pixmaps/archlinux-logo.png
dangling: /usr/share/cockpit/branding/arch/logo.png -> ../../../pixmaps/archlinux-logo.png
deleted:  /usr/share/cockpit/branding/arch/logo.png -> ../../../pixmaps/archlinux-logo.png
dangling: /usr/share/cockpit/branding/debian/favicon.ico -> ../../../pixmaps/debian-logo.png
deleted:  /usr/share/cockpit/branding/debian/favicon.ico -> ../../../pixmaps/debian-logo.png
dangling: /usr/share/cockpit/branding/debian/logo.png -> ../../../pixmaps/debian-logo.png
deleted:  /usr/share/cockpit/branding/debian/logo.png -> ../../../pixmaps/debian-logo.png
dangling: /usr/share/cockpit/branding/opensuse/default-1920x1200.jpg -> ../../../wallpapers/default-1920x1200.jpg
deleted:  /usr/share/cockpit/branding/opensuse/default-1920x1200.jpg -> ../../../wallpapers/default-1920x1200.jpg
dangling: /usr/share/cockpit/branding/opensuse/favicon.ico -> ../../../pixmaps/distribution-logos/favicon.ico
deleted:  /usr/share/cockpit/branding/opensuse/favicon.ico -> ../../../pixmaps/distribution-logos/favicon.ico
dangling: /usr/share/cockpit/branding/opensuse/square-hicolor.svg -> ../../../pixmaps/distribution-logos/square-hicolor.svg
deleted:  /usr/share/cockpit/branding/opensuse/square-hicolor.svg -> ../../../pixmaps/distribution-logos/square-hicolor.svg
dangling: /usr/share/cockpit/branding/ubuntu/logo.png -> ../../../plymouth/ubuntu-logo.png
deleted:  /usr/share/cockpit/branding/ubuntu/logo.png -> ../../../plymouth/ubuntu-logo.png
absolute: /usr/libexec/gawk -> /usr/libexec/awk
absolute: /usr/libexec/arptables-helper -> /etc/alternatives/arptables-helper
absolute: /usr/libexec/samba/cups_backend_smb -> /etc/alternatives/cups_backend_smb
```

Some rpms are now inconsistent
```
[root@minerva ~]# rpm -qa --verify
missing     /usr/lib/firmware/qcom/LENOVO/21BX.xz
missing   a /usr/lib/.build-id/18/186f3e1b1b577a19b2f1f296dbc2e41b6970f2
missing   a /usr/lib/.build-id/55/486e343b14b6f6badd7275b014dd82f44680d3
missing   d /usr/share/doc/hunspell/README
missing   a /usr/lib/.build-id/67/66146da07119be1e86dfcf4aa867f5e30bea43
missing   a /usr/lib/.build-id/a1/d1843eed661509d750d69baef2a61e1fbf19e7
missing   a /usr/lib/.build-id/7e/36f7119010825fb025ca944a61801662c7aa11
S.5....T.  c /etc/krb5.conf
missing   a /usr/lib/.build-id/2f/767bee207dca37dc5105a599722d6f2f4106f3
missing   a /usr/lib/.build-id/67/0f5d8033ec23ca4777aeca9ff236608e5e11af
S.5....T.  c /etc/samba/smb.conf
missing     /lib/modules/6.2.7-200.fc37.x86_64/build
missing     /lib/modules/6.2.7-200.fc37.x86_64/source
.M.......  g /var/lib/selinux/targeted/active/modules/200/smartmon
.......T.  c /etc/pcp/pmcd/pmcd.conf
S.5....T.  c /etc/chrony.conf
missing   a /usr/lib/.build-id/6a/d960b89a4448d76ead4892524133af065748b2
missing   a /usr/lib/.build-id/c1/7bd973c1e316bfee4f4aa5ac16830a5dc3bcd3
missing   a /usr/lib/.build-id/e3/dab686192ee682c642dc1a1e3e894e88a5af63
missing     /usr/share/cockpit/branding/arch/apple-touch-icon.png
missing     /usr/share/cockpit/branding/arch/favicon.ico
missing     /usr/share/cockpit/branding/arch/logo.png
missing     /usr/share/cockpit/branding/debian/favicon.ico
missing     /usr/share/cockpit/branding/debian/logo.png
missing     /usr/share/cockpit/branding/opensuse/default-1920x1200.jpg
missing     /usr/share/cockpit/branding/opensuse/favicon.ico
missing     /usr/share/cockpit/branding/opensuse/square-hicolor.svg
missing     /usr/share/cockpit/branding/ubuntu/logo.png
S.5....T.    /var/log/salt/minion
SM5....T.  c /etc/salt/minion
.M.......  n /etc/salt/pki/minion
.M.......    /var/lib/samba/sysvol
missing   a /usr/lib/.build-id/2f/767bee207dca37dc5105a599722d6f2f4106f3
missing   a /usr/lib/.build-id/67/0f5d8033ec23ca4777aeca9ff236608e5e11af
.....UG..  g /var/lib/nfs/statd/state
S.5....T.  c /etc/ssh/sshd_config
.M.......  c /boot/grub2/grub.cfg
S.5....T.  c /root/.bashrc
```

Version-Release number of selected component (if applicable):

Fedora upgrade to 35, 36 and 37 (at least)


How reproducible:

Very, very, very often

Steps to Reproduce:
1. Use fedora on a VM for a while
2. Follow instructions at https://docs.fedoraproject.org/en-US/quick-docs/dnf-system-upgrade/
3. rpm -qa --verify   now show broken contents

Actual results:

rpm -qa --verify   shows broken packages


Expected results:

Successful upgrade with consistent rpms


Additional info: