Hi Everyone, After removing old kernels, I noticed artifacts in /lib/modules. I believe this is due to the scripts used to install and remove kernels. I've experienced the problem in the past, but it got fixed with a systemd bug. Now I am seeing it again. I think it started happening around the time (recently) when both 'kernel-modules-core' and 'kernel-modules-extras' started showing up. I think in the past, they were a single packages and removed cleanly. In the output below, `dnf repoquery --installonly --latest-limit=-1 -q` was used to get a list of old kernels. Then `dnf remove` was used to uninstall them. The 6.2.7 and 6.2.8 kernels were removed, but artifcats remained in /lib/modules. ---------- $ sudo ~/rm-kernels.sh ... Removing old kernels Dependencies resolved. ================================================================================ Package Arch Version Repository Size ================================================================================ Removing: kernel x86_64 6.2.7-200.fc37 @updates 0 kernel x86_64 6.2.8-200.fc37 @updates 0 kernel-core x86_64 6.2.7-200.fc37 @updates 64 M kernel-core x86_64 6.2.8-200.fc37 @updates 64 M kernel-devel x86_64 6.2.7-200.fc37 @updates 69 M kernel-devel x86_64 6.2.8-200.fc37 @updates 69 M kernel-modules x86_64 6.2.7-200.fc37 @updates 61 M kernel-modules x86_64 6.2.8-200.fc37 @updates 61 M kernel-modules-core x86_64 6.2.7-200.fc37 @updates 36 M kernel-modules-core x86_64 6.2.8-200.fc37 @updates 36 M kernel-modules-extra x86_64 6.2.7-200.fc37 @updates 3.4 M kernel-modules-extra x86_64 6.2.8-200.fc37 @updates 3.4 M Transaction Summary ================================================================================ Remove 12 Packages Freed space: 465 M Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Erasing : kernel-modules-extra-6.2.8-200.fc37.x86_64 1/12 Running scriptlet: kernel-modules-extra-6.2.8-200.fc37.x86_64 1/12 Erasing : kernel-modules-extra-6.2.7-200.fc37.x86_64 2/12 Running scriptlet: kernel-modules-extra-6.2.7-200.fc37.x86_64 2/12 Erasing : kernel-6.2.8-200.fc37.x86_64 3/12 Running scriptlet: kernel-6.2.8-200.fc37.x86_64 3/12 Erasing : kernel-6.2.7-200.fc37.x86_64 4/12 Running scriptlet: kernel-6.2.7-200.fc37.x86_64 4/12 Erasing : kernel-modules-6.2.7-200.fc37.x86_64 5/12 Running scriptlet: kernel-modules-6.2.7-200.fc37.x86_64 5/12 Erasing : kernel-modules-6.2.8-200.fc37.x86_64 6/12 Running scriptlet: kernel-modules-6.2.8-200.fc37.x86_64 6/12 Erasing : kernel-modules-core-6.2.8-200.fc37.x86_64 7/12 Running scriptlet: kernel-modules-core-6.2.8-200.fc37.x86_64 7/12 Running scriptlet: kernel-core-6.2.8-200.fc37.x86_64 8/12 Erasing : kernel-core-6.2.8-200.fc37.x86_64 8/12 Erasing : kernel-modules-core-6.2.7-200.fc37.x86_64 9/12 Running scriptlet: kernel-modules-core-6.2.7-200.fc37.x86_64 9/12 Running scriptlet: kernel-core-6.2.7-200.fc37.x86_64 10/12 Erasing : kernel-core-6.2.7-200.fc37.x86_64 10/12 Erasing : kernel-devel-6.2.8-200.fc37.x86_64 11/12 Erasing : kernel-devel-6.2.7-200.fc37.x86_64 12/12 Running scriptlet: kernel-devel-6.2.7-200.fc37.x86_64 12/12 Verifying : kernel-6.2.7-200.fc37.x86_64 1/12 Verifying : kernel-6.2.8-200.fc37.x86_64 2/12 Verifying : kernel-core-6.2.7-200.fc37.x86_64 3/12 Verifying : kernel-core-6.2.8-200.fc37.x86_64 4/12 Verifying : kernel-devel-6.2.7-200.fc37.x86_64 5/12 Verifying : kernel-devel-6.2.8-200.fc37.x86_64 6/12 Verifying : kernel-modules-6.2.7-200.fc37.x86_64 7/12 Verifying : kernel-modules-6.2.8-200.fc37.x86_64 8/12 Verifying : kernel-modules-core-6.2.7-200.fc37.x86_64 9/12 Verifying : kernel-modules-core-6.2.8-200.fc37.x86_64 10/12 Verifying : kernel-modules-extra-6.2.7-200.fc37.x86_64 11/12 Verifying : kernel-modules-extra-6.2.8-200.fc37.x86_64 12/12 Removed: kernel-6.2.7-200.fc37.x86_64 kernel-6.2.8-200.fc37.x86_64 kernel-core-6.2.7-200.fc37.x86_64 kernel-core-6.2.8-200.fc37.x86_64 kernel-devel-6.2.7-200.fc37.x86_64 kernel-devel-6.2.8-200.fc37.x86_64 kernel-modules-6.2.7-200.fc37.x86_64 kernel-modules-6.2.8-200.fc37.x86_64 kernel-modules-core-6.2.7-200.fc37.x86_64 kernel-modules-core-6.2.8-200.fc37.x86_64 kernel-modules-extra-6.2.7-200.fc37.x86_64 kernel-modules-extra-6.2.8-200.fc37.x86_64 Complete! Removed old kernels $ ls /lib/modules 6.2.7-200.fc37.x86_64 6.2.8-200.fc37.x86_64 6.2.9-200.fc37.x86_64
Here's a similar bug reported by H.J. Lu from 2021: https://bugzilla.redhat.com/show_bug.cgi?id=2016630. The 2016630 bug was fixed in 2022.
Is it just a directory, or is there anything under /lib/modules/6.2.7-200.fc37.x86_64 ?
This problem is caused by directory ownership and package installation unordering. kernel-modules-core owns /lib/modules/<version>/. kernel-modules-core, kernel-modules-extra, kernel-modules, and kernel-core own files beneath. When dnf uninstalls packages, it removes kernel-modules-extra, kernel, kernel-modules, kernel-modules-core, and then kernel-core last. This means that the directory cannot be removed when kernel-modules-core is being removed (because it contains files owned by other packages), and then when kernel-core is being removed, rpm removes its files, but doesn't remove the directory because it's not owned by that package and leaves behind an empty directory. (Also, kernel-core has a %preun scriptlet that calls "kernel-install remove <version>". but when that script is called, kernel-core is still installed and /lib/modules/<version>/ is still populated. It's not kernel-install's job to remove /lib/modules/<version>/, but even if it wanted to, it cannot, since there's a bunch of files there at the time it is called.) The fix seems pretty simple: make kernel-modules-core, kernel-modules-extra, kernel-modules, and kernel-core all co-own /lib/modules/<version>/. Then rpm will remove the directory when last of those packages is removed (and there is nothing else underneath the directory). (Note that the problem applies to multiple releases. E.g. I tested this on Fedora 38.)
This topic came up on the Fedora Users mailing list.[1] It appears multiple folks are seeing the issue. It is unsurprising, but it is worth mentioning because folks are starting to craft their own scripts to remove them. It would probably be best if `rpm` handled the situation as expected. [1] https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org/thread/P2VKQBGQUXU67H4NDGTOPSBWAE4533VF/
This message is a reminder that Fedora Linux 37 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora Linux 37 on 2023-12-05. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a 'version' of '37'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, change the 'version' to a later Fedora Linux version. Note that the version field may be hidden. Click the "Show advanced fields" button if you do not see it. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora Linux 37 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora Linux, you are encouraged to change the 'version' to a later version prior to this bug being closed.
This is being discussed by users again in https://discussion.fedoraproject.org/t/service-old-lib-modules-xx-xx-x-cleaner/104014/10.
FWIW, old artifacts in /lib/modules are occasionally causing me problems with zfs.ko due to weak-updates symbolic links.