Bug 2181534 - Running symlinks -r -d /usr after dnf-system-upgrade breaks packages
Summary: Running symlinks -r -d /usr after dnf-system-upgrade breaks packages
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: symlinks
Version: 37
Hardware: Unspecified
OS: Linux
unspecified
unspecified
Target Milestone: ---
Assignee: Tim Waugh
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-03-24 12:59 UTC by Micke
Modified: 2024-01-12 23:19 UTC (History)
1 user (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2024-01-12 23:19:55 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Micke 2023-03-24 12:59:10 UTC
Description of problem:

I have followed the fedora upgrade instructions at https://docs.fedoraproject.org/en-US/quick-docs/dnf-system-upgrade/ and in the end they always end up breaking the consistency of some rpm packages

When executing the symlinks -r -d /usr  a fair number of rpms break.


State immediately before the symlinks step
```
[root@minerva ~]# rpm -qa --verify
S.5....T.  c /etc/krb5.conf
S.5....T.  c /etc/samba/smb.conf
.M.......  g /var/lib/selinux/targeted/active/modules/200/smartmon
.......T.  c /etc/pcp/pmcd/pmcd.conf
S.5....T.  c /etc/chrony.conf
S.5....T.    /var/log/salt/minion
SM5....T.  c /etc/salt/minion
.M.......  n /etc/salt/pki/minion
.M.......    /var/lib/samba/sysvol
.....UG..  g /var/lib/nfs/statd/state
S.5....T.  c /etc/ssh/sshd_config
.M.......  c /boot/grub2/grub.cfg
S.5....T.  c /root/.bashrc
```

Finding dangling symlinks...
```
[root@minerva ~]# symlinks -r /usr | grep dangling
dangling: /usr/lib/.build-id/c1/7bd973c1e316bfee4f4aa5ac16830a5dc3bcd3 -> ../../../../usr/sbin/quota_nld
dangling: /usr/lib/.build-id/e3/dab686192ee682c642dc1a1e3e894e88a5af63 -> ../../../../usr/sbin/warnquota
dangling: /usr/lib/.build-id/55/486e343b14b6f6badd7275b014dd82f44680d3 -> ../../../../usr/lib64/libncurses++.so.6.3
dangling: /usr/lib/.build-id/6a/d960b89a4448d76ead4892524133af065748b2 -> ../../../../usr/sbin/rpc.rquotad
dangling: /usr/lib/.build-id/7e/36f7119010825fb025ca944a61801662c7aa11 -> ../../../../usr/bin/coreutils.single
dangling: /usr/lib/.build-id/a1/d1843eed661509d750d69baef2a61e1fbf19e7 -> ../../../../usr/sbin/tc
dangling: /usr/lib/.build-id/18/186f3e1b1b577a19b2f1f296dbc2e41b6970f2 -> ../../../../usr/lib64/libncurses++w.so.6.3
dangling: /usr/lib/.build-id/67/0f5d8033ec23ca4777aeca9ff236608e5e11af -> ../../../../usr/sbin/grub2-sparc64-setup
dangling: /usr/lib/.build-id/67/66146da07119be1e86dfcf4aa867f5e30bea43 -> ../../../../usr/sbin/xfs_scrub
dangling: /usr/lib/.build-id/2f/767bee207dca37dc5105a599722d6f2f4106f3 -> ../../../../usr/sbin/grub2-ofpathname
dangling: /usr/lib/firmware/qcom/LENOVO/21BX.xz -> ../sc8280xp/LENOVO/21BX.xz
dangling: /usr/lib/modules/6.2.7-200.fc37.x86_64/build -> /usr/src/kernels/6.2.7-200.fc37.x86_64
dangling: /usr/lib/modules/6.2.7-200.fc37.x86_64/source -> build
dangling: /usr/share/doc/hunspell/README -> README.md
dangling: /usr/share/cockpit/branding/arch/apple-touch-icon.png -> ../../../pixmaps/archlinux-logo.png
dangling: /usr/share/cockpit/branding/arch/favicon.ico -> ../../../pixmaps/archlinux-logo.png
dangling: /usr/share/cockpit/branding/arch/logo.png -> ../../../pixmaps/archlinux-logo.png
dangling: /usr/share/cockpit/branding/debian/favicon.ico -> ../../../pixmaps/debian-logo.png
dangling: /usr/share/cockpit/branding/debian/logo.png -> ../../../pixmaps/debian-logo.png
dangling: /usr/share/cockpit/branding/opensuse/default-1920x1200.jpg -> ../../../wallpapers/default-1920x1200.jpg
dangling: /usr/share/cockpit/branding/opensuse/favicon.ico -> ../../../pixmaps/distribution-logos/favicon.ico
dangling: /usr/share/cockpit/branding/opensuse/square-hicolor.svg -> ../../../pixmaps/distribution-logos/square-hicolor.svg
dangling: /usr/share/cockpit/branding/ubuntu/logo.png -> ../../../plymouth/ubuntu-logo.png
[root@minerva ~]# ls -l /usr/lib/.build-id/c1/7bd973c1e316bfee4f4aa5ac16830a5dc3bcd3
lrwxrwxrwx. 1 root root 30 Jul 23  2022 /usr/lib/.build-id/c1/7bd973c1e316bfee4f4aa5ac16830a5dc3bcd3 -> ../../../../usr/sbin/quota_nld
```

Yes, they appear to be dangling when inspected

Follow instructions...
```
[root@minerva ~]# symlinks -r -d /usr
absolute: /usr/bin/ex -> /usr/libexec/vi
absolute: /usr/bin/rvi -> /usr/libexec/vi
absolute: /usr/bin/rview -> /usr/libexec/vi
absolute: /usr/bin/soelim -> /etc/alternatives/soelim
messy:    /usr/bin/pydoc -> ./pydoc3
absolute: /usr/bin/abrt-cli -> /usr/bin/abrt
absolute: /usr/bin/whois -> /etc/alternatives/whois
absolute: /usr/bin/apropos -> /etc/alternatives/apropos
absolute: /usr/bin/whatis -> /etc/alternatives/whatis
absolute: /usr/bin/nc -> /etc/alternatives/nc
absolute: /usr/bin/man -> /etc/alternatives/man
absolute: /usr/bin/yum-debug-restore -> /usr/libexec/dnf-utils
absolute: /usr/bin/find-repos-of-install -> /usr/libexec/dnf-utils
absolute: /usr/bin/repomanage -> /usr/libexec/dnf-utils
absolute: /usr/bin/clevis-decrypt-tpm2plus -> /usr/bin/clevis-pin-tpm2
absolute: /usr/bin/yum-groups-manager -> /usr/libexec/dnf-utils
absolute: /usr/bin/yumdownloader -> /usr/libexec/dnf-utils
absolute: /usr/bin/package-cleanup -> /usr/libexec/dnf-utils
absolute: /usr/bin/repotrack -> /usr/libexec/dnf-utils
absolute: /usr/bin/yum-builddep -> /usr/libexec/dnf-utils
absolute: /usr/bin/yum-config-manager -> /usr/libexec/dnf-utils
absolute: /usr/bin/repoclosure -> /usr/libexec/dnf-utils
absolute: /usr/bin/yum-debug-dump -> /usr/libexec/dnf-utils
absolute: /usr/bin/debuginfo-install -> /usr/libexec/dnf-utils
absolute: /usr/bin/repo-graph -> /usr/libexec/dnf-utils
absolute: /usr/bin/repoquery -> /usr/libexec/dnf-utils
absolute: /usr/bin/repodiff -> /usr/libexec/dnf-utils
absolute: /usr/bin/needs-restarting -> /usr/libexec/dnf-utils
absolute: /usr/bin/reposync -> /usr/libexec/dnf-utils
messy:    /usr/bin/python -> ./python3
absolute: /usr/bin/clevis-encrypt-tpm2plus -> /usr/bin/clevis-pin-tpm2
absolute: /usr/sbin/ipset-translate -> /usr/sbin/ipset
absolute: /usr/sbin/pidof -> /usr/bin/pidof
absolute: /usr/sbin/iptables-save -> /etc/alternatives/iptables-save
absolute: /usr/sbin/arptables -> /etc/alternatives/arptables
absolute: /usr/sbin/ip6tables-save -> /etc/alternatives/ip6tables-save
absolute: /usr/sbin/arptables-save -> /etc/alternatives/arptables-save
absolute: /usr/sbin/iptables-restore -> /etc/alternatives/iptables-restore
absolute: /usr/sbin/ebtables-restore -> /etc/alternatives/ebtables-restore
absolute: /usr/sbin/ebtables -> /etc/alternatives/ebtables
absolute: /usr/sbin/ifup -> /etc/alternatives/ifup
absolute: /usr/sbin/arptables-restore -> /etc/alternatives/arptables-restore
absolute: /usr/sbin/iptables -> /etc/alternatives/iptables
absolute: /usr/sbin/ip6tables -> /etc/alternatives/ip6tables
absolute: /usr/sbin/ip6tables-restore -> /etc/alternatives/ip6tables-restore
absolute: /usr/sbin/ebtables-save -> /etc/alternatives/ebtables-save
absolute: /usr/sbin/mkfs.ntfs -> /usr/sbin/mkntfs
absolute: /usr/sbin/ifdown -> /etc/alternatives/ifdown
absolute: /usr/sbin/mount.lowntfs-3g -> /usr/bin/lowntfs-3g
absolute: /usr/sbin/mount.ntfs-3g -> /usr/bin/ntfs-3g
dangling: /usr/lib/.build-id/c1/7bd973c1e316bfee4f4aa5ac16830a5dc3bcd3 -> ../../../../usr/sbin/quota_nld
deleted:  /usr/lib/.build-id/c1/7bd973c1e316bfee4f4aa5ac16830a5dc3bcd3 -> ../../../../usr/sbin/quota_nld
dangling: /usr/lib/.build-id/e3/dab686192ee682c642dc1a1e3e894e88a5af63 -> ../../../../usr/sbin/warnquota
deleted:  /usr/lib/.build-id/e3/dab686192ee682c642dc1a1e3e894e88a5af63 -> ../../../../usr/sbin/warnquota
dangling: /usr/lib/.build-id/55/486e343b14b6f6badd7275b014dd82f44680d3 -> ../../../../usr/lib64/libncurses++.so.6.3
deleted:  /usr/lib/.build-id/55/486e343b14b6f6badd7275b014dd82f44680d3 -> ../../../../usr/lib64/libncurses++.so.6.3
dangling: /usr/lib/.build-id/6a/d960b89a4448d76ead4892524133af065748b2 -> ../../../../usr/sbin/rpc.rquotad
deleted:  /usr/lib/.build-id/6a/d960b89a4448d76ead4892524133af065748b2 -> ../../../../usr/sbin/rpc.rquotad
dangling: /usr/lib/.build-id/7e/36f7119010825fb025ca944a61801662c7aa11 -> ../../../../usr/bin/coreutils.single
deleted:  /usr/lib/.build-id/7e/36f7119010825fb025ca944a61801662c7aa11 -> ../../../../usr/bin/coreutils.single
dangling: /usr/lib/.build-id/a1/d1843eed661509d750d69baef2a61e1fbf19e7 -> ../../../../usr/sbin/tc
deleted:  /usr/lib/.build-id/a1/d1843eed661509d750d69baef2a61e1fbf19e7 -> ../../../../usr/sbin/tc
dangling: /usr/lib/.build-id/18/186f3e1b1b577a19b2f1f296dbc2e41b6970f2 -> ../../../../usr/lib64/libncurses++w.so.6.3
deleted:  /usr/lib/.build-id/18/186f3e1b1b577a19b2f1f296dbc2e41b6970f2 -> ../../../../usr/lib64/libncurses++w.so.6.3
dangling: /usr/lib/.build-id/67/0f5d8033ec23ca4777aeca9ff236608e5e11af -> ../../../../usr/sbin/grub2-sparc64-setup
deleted:  /usr/lib/.build-id/67/0f5d8033ec23ca4777aeca9ff236608e5e11af -> ../../../../usr/sbin/grub2-sparc64-setup
dangling: /usr/lib/.build-id/67/66146da07119be1e86dfcf4aa867f5e30bea43 -> ../../../../usr/sbin/xfs_scrub
deleted:  /usr/lib/.build-id/67/66146da07119be1e86dfcf4aa867f5e30bea43 -> ../../../../usr/sbin/xfs_scrub
dangling: /usr/lib/.build-id/2f/767bee207dca37dc5105a599722d6f2f4106f3 -> ../../../../usr/sbin/grub2-ofpathname
deleted:  /usr/lib/.build-id/2f/767bee207dca37dc5105a599722d6f2f4106f3 -> ../../../../usr/sbin/grub2-ofpathname
dangling: /usr/lib/firmware/qcom/LENOVO/21BX.xz -> ../sc8280xp/LENOVO/21BX.xz
deleted:  /usr/lib/firmware/qcom/LENOVO/21BX.xz -> ../sc8280xp/LENOVO/21BX.xz
dangling: /usr/lib/modules/6.2.7-200.fc37.x86_64/build -> /usr/src/kernels/6.2.7-200.fc37.x86_64
deleted:  /usr/lib/modules/6.2.7-200.fc37.x86_64/build -> /usr/src/kernels/6.2.7-200.fc37.x86_64
dangling: /usr/lib/modules/6.2.7-200.fc37.x86_64/source -> build
deleted:  /usr/lib/modules/6.2.7-200.fc37.x86_64/source -> build
absolute: /usr/lib/python3.11/site-packages/urllib3/packages/__pycache__/six.cpython-311.opt-1.pyc -> /usr/lib/python3.11/site-packages/__pycache__/six.cpython-311.opt-1.pyc
absolute: /usr/lib/python3.11/site-packages/urllib3/packages/__pycache__/six.cpython-311.pyc -> /usr/lib/python3.11/site-packages/__pycache__/six.cpython-311.pyc
absolute: /usr/lib/python3.11/site-packages/urllib3/packages/six.py -> /usr/lib/python3.11/site-packages/six.py
absolute: /usr/lib64/libnssckbi.so -> /etc/alternatives/libnssckbi.so.x86_64
absolute: /usr/lib64/libwbclient.so.0.15 -> /etc/alternatives/libwbclient.so.0.15-64
dangling: /usr/share/doc/hunspell/README -> README.md
deleted:  /usr/share/doc/hunspell/README -> README.md
absolute: /usr/share/doc/python-systemd/html/_static/jquery.js -> /usr/share/javascript/jquery/latest/jquery.min.js
absolute: /usr/share/doc/atmel-firmware/COPYING -> /lib/firmware/COPYING.atmel-firmware
absolute: /usr/share/doc/atmel-firmware/COPYRIGHT-usb -> /lib/firmware/COPYRIGHT-usb.atmel-firmware
absolute: /usr/share/doc/atmel-firmware/README -> /lib/firmware/README.atmel-firmware
absolute: /usr/share/doc/atmel-firmware/README-usb -> /lib/firmware/README-usb.atmel-firmware
absolute: /usr/share/man/man7/man.7.gz -> /etc/alternatives/man.7.gz
absolute: /usr/share/man/man1/soelim.1.gz -> /etc/alternatives/soelim.1.gz
absolute: /usr/share/man/man1/whatis.1.gz -> /etc/alternatives/whatis.1.gz
absolute: /usr/share/man/man1/abrt-cli.1.gz -> /usr/share/man/man1/abrt.1.gz
absolute: /usr/share/man/man1/man.1.gz -> /etc/alternatives/man.1.gz
absolute: /usr/share/man/man1/apropos.1.gz -> /etc/alternatives/apropos.1.gz
absolute: /usr/share/man/man1/nc.1.gz -> /etc/alternatives/nc-man
absolute: /usr/share/man/man1/repoquery.1.gz -> /usr/share/man/man1/dnf-utils.1.gz
absolute: /usr/share/man/man1/repotrack.1.gz -> /usr/share/man/man1/dnf-utils.1.gz
absolute: /usr/share/man/man1/find-repos-of-install.1.gz -> /usr/share/man/man1/dnf-utils.1.gz
messy:    /usr/share/man/man1/python.1.gz -> ./python3.1.gz
absolute: /usr/share/man/man1/whois.1.gz -> /etc/alternatives/whois-man
absolute: /usr/share/man/man8/ebtables.8.gz -> /etc/alternatives/ebtables-man
absolute: /usr/share/man/man8/arptables-restore.8.gz -> /etc/alternatives/arptables-restore-man
absolute: /usr/share/man/man8/arptables.8.gz -> /etc/alternatives/arptables-man
absolute: /usr/share/man/man8/arptables-save.8.gz -> /etc/alternatives/arptables-save-man
absolute: /usr/share/man/man8/dnf-offline-distrosync.8.gz -> /usr/share/man/man8/dnf-system-upgrade.8.gz
absolute: /usr/share/man/man8/dnf-offline-upgrade.8.gz -> /usr/share/man/man8/dnf-system-upgrade.8.gz
absolute: /usr/share/gawk -> /usr/share/awk
dangling: /usr/share/cockpit/branding/arch/apple-touch-icon.png -> ../../../pixmaps/archlinux-logo.png
deleted:  /usr/share/cockpit/branding/arch/apple-touch-icon.png -> ../../../pixmaps/archlinux-logo.png
dangling: /usr/share/cockpit/branding/arch/favicon.ico -> ../../../pixmaps/archlinux-logo.png
deleted:  /usr/share/cockpit/branding/arch/favicon.ico -> ../../../pixmaps/archlinux-logo.png
dangling: /usr/share/cockpit/branding/arch/logo.png -> ../../../pixmaps/archlinux-logo.png
deleted:  /usr/share/cockpit/branding/arch/logo.png -> ../../../pixmaps/archlinux-logo.png
dangling: /usr/share/cockpit/branding/debian/favicon.ico -> ../../../pixmaps/debian-logo.png
deleted:  /usr/share/cockpit/branding/debian/favicon.ico -> ../../../pixmaps/debian-logo.png
dangling: /usr/share/cockpit/branding/debian/logo.png -> ../../../pixmaps/debian-logo.png
deleted:  /usr/share/cockpit/branding/debian/logo.png -> ../../../pixmaps/debian-logo.png
dangling: /usr/share/cockpit/branding/opensuse/default-1920x1200.jpg -> ../../../wallpapers/default-1920x1200.jpg
deleted:  /usr/share/cockpit/branding/opensuse/default-1920x1200.jpg -> ../../../wallpapers/default-1920x1200.jpg
dangling: /usr/share/cockpit/branding/opensuse/favicon.ico -> ../../../pixmaps/distribution-logos/favicon.ico
deleted:  /usr/share/cockpit/branding/opensuse/favicon.ico -> ../../../pixmaps/distribution-logos/favicon.ico
dangling: /usr/share/cockpit/branding/opensuse/square-hicolor.svg -> ../../../pixmaps/distribution-logos/square-hicolor.svg
deleted:  /usr/share/cockpit/branding/opensuse/square-hicolor.svg -> ../../../pixmaps/distribution-logos/square-hicolor.svg
dangling: /usr/share/cockpit/branding/ubuntu/logo.png -> ../../../plymouth/ubuntu-logo.png
deleted:  /usr/share/cockpit/branding/ubuntu/logo.png -> ../../../plymouth/ubuntu-logo.png
absolute: /usr/libexec/gawk -> /usr/libexec/awk
absolute: /usr/libexec/arptables-helper -> /etc/alternatives/arptables-helper
absolute: /usr/libexec/samba/cups_backend_smb -> /etc/alternatives/cups_backend_smb
```

Some rpms are now inconsistent
```
[root@minerva ~]# rpm -qa --verify
missing     /usr/lib/firmware/qcom/LENOVO/21BX.xz
missing   a /usr/lib/.build-id/18/186f3e1b1b577a19b2f1f296dbc2e41b6970f2
missing   a /usr/lib/.build-id/55/486e343b14b6f6badd7275b014dd82f44680d3
missing   d /usr/share/doc/hunspell/README
missing   a /usr/lib/.build-id/67/66146da07119be1e86dfcf4aa867f5e30bea43
missing   a /usr/lib/.build-id/a1/d1843eed661509d750d69baef2a61e1fbf19e7
missing   a /usr/lib/.build-id/7e/36f7119010825fb025ca944a61801662c7aa11
S.5....T.  c /etc/krb5.conf
missing   a /usr/lib/.build-id/2f/767bee207dca37dc5105a599722d6f2f4106f3
missing   a /usr/lib/.build-id/67/0f5d8033ec23ca4777aeca9ff236608e5e11af
S.5....T.  c /etc/samba/smb.conf
missing     /lib/modules/6.2.7-200.fc37.x86_64/build
missing     /lib/modules/6.2.7-200.fc37.x86_64/source
.M.......  g /var/lib/selinux/targeted/active/modules/200/smartmon
.......T.  c /etc/pcp/pmcd/pmcd.conf
S.5....T.  c /etc/chrony.conf
missing   a /usr/lib/.build-id/6a/d960b89a4448d76ead4892524133af065748b2
missing   a /usr/lib/.build-id/c1/7bd973c1e316bfee4f4aa5ac16830a5dc3bcd3
missing   a /usr/lib/.build-id/e3/dab686192ee682c642dc1a1e3e894e88a5af63
missing     /usr/share/cockpit/branding/arch/apple-touch-icon.png
missing     /usr/share/cockpit/branding/arch/favicon.ico
missing     /usr/share/cockpit/branding/arch/logo.png
missing     /usr/share/cockpit/branding/debian/favicon.ico
missing     /usr/share/cockpit/branding/debian/logo.png
missing     /usr/share/cockpit/branding/opensuse/default-1920x1200.jpg
missing     /usr/share/cockpit/branding/opensuse/favicon.ico
missing     /usr/share/cockpit/branding/opensuse/square-hicolor.svg
missing     /usr/share/cockpit/branding/ubuntu/logo.png
S.5....T.    /var/log/salt/minion
SM5....T.  c /etc/salt/minion
.M.......  n /etc/salt/pki/minion
.M.......    /var/lib/samba/sysvol
missing   a /usr/lib/.build-id/2f/767bee207dca37dc5105a599722d6f2f4106f3
missing   a /usr/lib/.build-id/67/0f5d8033ec23ca4777aeca9ff236608e5e11af
.....UG..  g /var/lib/nfs/statd/state
S.5....T.  c /etc/ssh/sshd_config
.M.......  c /boot/grub2/grub.cfg
S.5....T.  c /root/.bashrc
```

Version-Release number of selected component (if applicable):

Fedora upgrade to 35, 36 and 37 (at least)


How reproducible:

Very, very, very often

Steps to Reproduce:
1. Use fedora on a VM for a while
2. Follow instructions at https://docs.fedoraproject.org/en-US/quick-docs/dnf-system-upgrade/
3. rpm -qa --verify   now show broken contents

Actual results:

rpm -qa --verify   shows broken packages


Expected results:

Successful upgrade with consistent rpms


Additional info:

Comment 1 Aoife Moloney 2023-11-23 01:33:40 UTC
This message is a reminder that Fedora Linux 37 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 37 on 2023-12-05.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '37'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version. Note that the version field may be hidden.
Click the "Show advanced fields" button if you do not see it.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 37 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Comment 2 Aoife Moloney 2024-01-12 23:19:55 UTC
Fedora Linux 37 entered end-of-life (EOL) status on 2023-12-05.

Fedora Linux 37 is no longer maintained, which means that it
will not receive any further security or bug fix updates. As a result we
are closing this bug.

If you can reproduce this bug against a currently maintained version of Fedora Linux
please feel free to reopen this bug against that version. Note that the version
field may be hidden. Click the "Show advanced fields" button if you do not see
the version field.

If you are unable to reopen this bug, please file a new report against an
active release.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.