Bug 2276271
Summary: | When using sd-boot and when upgrading the kernel the oldest kernel is not completely removed. | ||||||
---|---|---|---|---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Villy Kruse <ppywlkiqletw> | ||||
Component: | dracut | Assignee: | Pavel Valena <pvalena> | ||||
Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||
Severity: | medium | Docs Contact: | |||||
Priority: | unspecified | ||||||
Version: | 40 | CC: | abetakehiko, bmj001, dracut-maint-list, jamacku, lnykryn, netwiz, pvalena, samesmclark, sannnagy, yaneti | ||||
Target Milestone: | --- | Keywords: | Reopened | ||||
Target Release: | --- | ||||||
Hardware: | Unspecified | ||||||
OS: | Linux | ||||||
Whiteboard: | |||||||
Fixed In Version: | dracut-102-2.fc41 dracut-102-2.fc40 | Doc Type: | If docs needed, set a value | ||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | Environment: | ||||||
Last Closed: | 2024-07-19 01:45:52 UTC | Type: | --- | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Attachments: |
|
Description
Villy Kruse
2024-04-21 11:10:23 UTC
I'll soon switch to dracut-ng; new upstream [1], starting with version 101 (or later). There's a different implementation (and no downstream patch) of the `50-dracut.install` file. Will you be able to test? Thanks! (In reply to Pavel Valena from comment #1) > I'll soon switch to dracut-ng; new upstream [1], starting with version 101 > (or later). There's a different implementation (and no downstream patch) of > the `50-dracut.install` file. Will you be able to test? > > Thanks! Not until a new rpm is released. Off course! I can send you COPR build & scratch build (when I have it). You can also test when I submit it to testing, but that's quite late (as the version will be in Rawhide at that point) in comparison with the scratch-build, which I'll be able to do today/tomorrow. (In reply to Pavel Valena from comment #3) > Off course! > > I can send you COPR build & scratch build (when I have it). You can also > test when I submit it to testing, but that's quite late (as the version will > be in Rawhide at that point) in comparison with the scratch-build, which > I'll be able to do today/tomorrow. We need a fix for Fedora 40. Fedora 39 and earlier works fine. Yes, that's what I meant with "testing" - F40 Bodhi. Scratch-builds and COPR can be built for F40 as well. I've just checked, and the updated version will be probably in the next release. Here're the scratch/COPR builds nonetheless: - https://copr.fedorainfracloud.org/coprs/build/7376294 - https://koji.fedoraproject.org/koji/taskinfo?taskID=116911913 I have not tested them yet. Please let me know if some additional fix is needed. Please also note that dracut-ng upstream removed the `remove` functionality altogether, as it's handled elsewhere: https://github.com/dracut-ng/dracut-ng/pull/111#pullrequestreview-1985952090 (And we do not intend to keep downstream patch.) Sorry, I forgot to include new module in the spec file. Correct builds are here: - https://copr.fedorainfracloud.org/coprs/build/7376400 - https://koji.fedoraproject.org/koji/taskinfo?taskID=116912318 Updated via: dnf upgrade https://download.copr.fedorainfracloud.org/results/pvalena/dracut/fedora-40-x86_64/07376400-dracut/dracut-config-generic-101-0.1.fc40.x86_64.rpm https://download.copr.fedorainfracloud.org/results/pvalena/dracut/fedora-40-x86_64/07376400-dracut/dracut-101-0.1.fc40.x86_64.rpm Ran updates on that system which installed kernel 6.8.8. Rebooted, then tried to remove kernel 6.8.7 and got: Running transaction Preparing : 1/1 Erasing : kernel-modules-6.8.7-300.fc40.x86_64 1/3 Running scriptlet: kernel-modules-6.8.7-300.fc40.x86_64 1/3 Erasing : kernel-modules-core-6.8.7-300.fc40.x86_64 2/3 Running scriptlet: kernel-core-6.8.7-300.fc40.x86_64 3/3 rm: cannot remove '/efi/6e49b7842a1942bf91a9dc51e68971b3/6.8.7-300.fc40.x86_64/': Is a directory /usr/lib/kernel/install.d/50-dracut.install failed with exit status 1. error: %preun(kernel-core-6.8.7-300.fc40.x86_64) scriptlet failed, exit status 1 Error in PREUN scriptlet in rpm package kernel-core error: kernel-core-6.8.7-300.fc40.x86_64: erase failed Removed: kernel-modules-6.8.7-300.fc40.x86_64 kernel-modules-core-6.8.7-300.fc40.x86_64 Failed: kernel-core-6.8.7-300.fc40.x86_64 Error: Transaction failed As a side note, if you run the erase again, then it works: dnf erase kernel*6.8.7* Dependencies resolved. ============================================================================================================================================================================================================================================ Package Architecture Version Repository Size ============================================================================================================================================================================================================================================ Removing: kernel-core x86_64 6.8.7-300.fc40 @@System 66 M Transaction Summary ============================================================================================================================================================================================================================================ Remove 1 Package Freed space: 66 M Is this ok [y/N]: y Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Running scriptlet: kernel-core-6.8.7-300.fc40.x86_64 1/1 Erasing : kernel-core-6.8.7-300.fc40.x86_64 1/1 Running scriptlet: kernel-core-6.8.7-300.fc40.x86_64 1/1 Removed: kernel-core-6.8.7-300.fc40.x86_64 Complete! Any progress on this as yet? Still an issue as of kernel 6.8.8 for removals: Running transaction Preparing : 1/1 Erasing : kernel-modules-6.8.8-300.fc40.x86_64 1/3 Running scriptlet: kernel-modules-6.8.8-300.fc40.x86_64 1/3 Erasing : kernel-modules-core-6.8.8-300.fc40.x86_64 2/3 Running scriptlet: kernel-core-6.8.8-300.fc40.x86_64 3/3 rm: cannot remove '/efi/1f3beac22b174372878b369646feb3f3/6.8.8-300.fc40.x86_64/': Is a directory /usr/lib/kernel/install.d/50-dracut.install failed with exit status 1. error: %preun(kernel-core-6.8.8-300.fc40.x86_64) scriptlet failed, exit status 1 Error in PREUN scriptlet in rpm package kernel-core error: kernel-core-6.8.8-300.fc40.x86_64: erase failed Removed: kernel-modules-6.8.8-300.fc40.x86_64 kernel-modules-core-6.8.8-300.fc40.x86_64 Failed: kernel-core-6.8.8-300.fc40.x86_64 Error: Transaction failed dracut-060-1.fc40.x86_64 is still the latest version available on the repos... Updates for dracut came through today.... Still the same issue: Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : Erasing : kernel-modules-extra-6.8.7-300.fc40.x86_64 Running scriptlet: kernel-modules-extra-6.8.7-300.fc40.x86_64 Erasing : kernel-6.8.7-300.fc40.x86_64 Running scriptlet: kernel-6.8.7-300.fc40.x86_64 Erasing : kernel-modules-6.8.7-300.fc40.x86_64 Running scriptlet: kernel-modules-6.8.7-300.fc40.x86_64 Erasing : kernel-modules-core-6.8.7-300.fc40.x86_64 Running scriptlet: kernel-core-6.8.7-300.fc40.x86_64 rm: cannot remove '/efi/af41881635e94f9a9a5e6f8267f4a78f/6.8.7-300.fc40.x86_64/': Is a directory /usr/lib/kernel/install.d/50-dracut.install failed with exit status 1. error: %preun(kernel-core-6.8.7-300.fc40.x86_64) scriptlet failed, exit status 1 Error in PREUN scriptlet in rpm package kernel-core error: kernel-core-6.8.7-300.fc40.x86_64: erase failed # rpm -qa | grep dracut | sort dracut-101-1.fc40.x86_64 dracut-config-rescue-101-1.fc40.x86_64 dracut-live-101-1.fc40.x86_64 dracut-network-101-1.fc40.x86_64 dracut-squash-101-1.fc40.x86_64 I am too facing the same issue. As a workaround you can simply replace the `rm -f` command with `rm -rf` in the install script so it can remove the directory. Put the new script it in `/etc/kernel/install.d/` so it takes precedence. This one-liner should do the job. sudo cp /usr/lib/kernel/install.d/50-dracut.install /etc/kernel/install.d/ && sudo sed -i 's/rm -f/rm -rf/' /etc/kernel/install.d/50-dracut.install (In reply to sannnagy from comment #14) > I am too facing the same issue. > As a workaround you can simply replace the `rm -f` command with `rm -rf` in > the install script so it can remove the directory. Put the new script it in > `/etc/kernel/install.d/` so it takes precedence. > > This one-liner should do the job. > > sudo cp /usr/lib/kernel/install.d/50-dracut.install /etc/kernel/install.d/ > && sudo sed -i 's/rm -f/rm -rf/' /etc/kernel/install.d/50-dracut.install Better, and more correct --- 50-dracut.install.orig 2024-05-16 02:00:00.000000000 +0200 +++ 50-dracut.install 2024-05-24 08:10:24.610779279 +0200 @@ -19,7 +19,7 @@ fi if [[ -d "$BOOT_DIR_ABS" ]]; then - INITRD="initrd" + IMAGE="initrd" else # No layout information, use users --uefi/--no-uefi preference UEFI_OPTS="" Can confirm that changing INITRD to IMAGE in the above patch seems to fix things for me... @pvalena - Do you have an update on this? It's still causing problems for new installs etc. Reinstalling a kernel package will break the handling of packages causing two of the same package to be installed, ie: # rpm -qa | grep kernel | sort kernel-core-6.8.10-300.fc40.x86_64 kernel-core-6.8.5-301.fc40.x86_64 kernel-core-6.8.5-301.fc40.x86_64 kernel-modules-core-6.8.10-300.fc40.x86_64 kernel-modules-core-6.8.5-301.fc40.x86_64 You can't uninstall then via dnf: # dnf erase kernel*6.8.5* Dependencies resolved. ============================================================================================================== Package Architecture Version Repository Size ============================================================================================================== Removing: kernel-core x86_64 6.8.5-301.fc40 @fedora 66 M kernel-modules-core x86_64 6.8.5-301.fc40 @19bf5097ad194846bee3ed5cc98012c9 32 M Transaction Summary ============================================================================================================== Remove 2 Packages Freed space: 99 M Is this ok [y/N]: y Running transaction check Error: transaction check vs depsolve: kernel-modules-core-uname-r = 6.8.5-301.fc40.x86_64 is needed by (installed) kernel-core-6.8.5-301.fc40.x86_64 To diagnose the problem, try running: 'rpm -Va --nofiles --nodigest'. You probably have corrupted RPMDB, running 'rpm --rebuilddb' might fix the issue. You can't remove them via RPM directly: # rpm -e kernel-core-6.8.5-301.fc40.x86_64 --nodeps error: "kernel-core-6.8.5-301.fc40.x86_64" specifies multiple packages: kernel-core-6.8.5-301.fc40.x86_64 kernel-core-6.8.5-301.fc40.x86_64 I have multiple systems in this state now that I'm not quite sure how to fix. Currently, DNF will not be able to remove these packages when kernels update as time passes. Annoyingly, with it being a simple fix, its frustrating that this is over a month and still ongoing. > Annoyingly, with it being a simple fix, its frustrating that this is over a > month and still ongoing. Firstly, sorry, but there's no `simple fix` upstream. Like I wrote in [1], `install.d/*` scripts were reworked upstream, and I don't want to backport such complicated & multiple patches without a version release. Good news is upstream just released v102[3]. Hopefully it works for you. If not, please contribute some test cases with your setup. I'm sorry I didn't get around to incorporate your suggestsions to dracut-101. Feel free to file a PR with it anytime in Fedora `source-git` [2] (the build is done immediately after PR merge). I will upgrade to v102 ASAP though, so it would dropped right away at this point. I will post a scratch build here with dracut-102 for testing. [1] https://bugzilla.redhat.com/show_bug.cgi?id=2276271#c7 [2] https://github.com/redhat-plumbers/dracut-fedora/ [3] https://github.com/dracut-ng/dracut-ng/releases/tag/102 Just wondering what the current progress is on this? It's about a month later, but it looks like we're still on dracut 101, and I can't seem to find any scratch build of 102 that may have this issue fixed. I'm so sorry, but there was issue [1] with the upstream changes, which needs to be resolved first, so I have reverted it for now. I will send you a build for testing when there's support for it, but as of now, it just breaks Fedora (doesn't create initramfs at all). [1] https://github.com/dracut-ng/dracut-ng/issues/334 (In reply to Pavel Valena from comment #20) > I'm so sorry, but there was issue [1] with the upstream changes, which needs > to be resolved first, so I have reverted it for now. I will send you a build > for testing when there's support for it, but as of now, it just breaks > Fedora (doesn't create initramfs at all). > > [1] https://github.com/dracut-ng/dracut-ng/issues/334 While trying to figure that out, could you perhaps apply this diff, which will fix the immediate problem for now in Fedora 40? --- 50-dracut.install.orig 2024-05-16 02:00:00.000000000 +0200 +++ 50-dracut.install 2024-05-24 08:10:24.610779279 +0200 @@ -19,7 +19,7 @@ fi if [[ -d "$BOOT_DIR_ABS" ]]; then - INITRD="initrd" + IMAGE="initrd" else # No layout information, use users --uefi/--no-uefi preference UEFI_OPTS="" The explanation for this change was already given in message 1 and also in some other bugzilla report for the same issue. Created attachment 2038706 [details]
grub2 script to fish out initrd from the staging area.
This morning I threw together this shell scriptlet, should can be installed in /etc/kernel/install.d.
When this script is installed you can instal the current version of 50-dracut.install from dracut-ng and it will work. The old version of 50-dracut.install from Fedora 39 would work as well.
The script 60-fetch-initrd.install (or something similar) should be owned and maintained by the grub2 maintainers, as it would be the responsibility of grub2 scripts to fish out the initrd from the staging area.
this looks like a misapplied patch, I think the install script should look like this: [root@vm-10-0-185-120 install.d]# diff 50-dracut.install 50-dracut.install.orig 21,24c21,22 < # No layout information, use users --uefi/--no-uefi preference < UEFI_OPTS="" < if [[ -d $BOOT_DIR_ABS ]]; then < IMAGE="initrd" --- > if [[ -d "$BOOT_DIR_ABS" ]]; then > INITRD="initrd" 26,27c24,31 < BOOT_DIR_ABS="/boot" < IMAGE="initramfs-${KERNEL_VERSION}.img" --- > # No layout information, use users --uefi/--no-uefi preference > UEFI_OPTS="" > if [[ -d $BOOT_DIR_ABS ]]; then > IMAGE="initrd" > else > BOOT_DIR_ABS="/boot" > IMAGE="initramfs-${KERNEL_VERSION}.img" > fi (In reply to Lukáš Nykrýn from comment #23) > this looks like a misapplied patch, I think the install script should look > like this: > > [root@vm-10-0-185-120 install.d]# diff 50-dracut.install > 50-dracut.install.orig > 21,24c21,22 > < # No layout information, use users --uefi/--no-uefi preference > < UEFI_OPTS="" > < if [[ -d $BOOT_DIR_ABS ]]; then > < IMAGE="initrd" > --- > > if [[ -d "$BOOT_DIR_ABS" ]]; then > > INITRD="initrd" > 26,27c24,31 > < BOOT_DIR_ABS="/boot" > < IMAGE="initramfs-${KERNEL_VERSION}.img" > --- > > # No layout information, use users --uefi/--no-uefi preference > > UEFI_OPTS="" > > if [[ -d $BOOT_DIR_ABS ]]; then > > IMAGE="initrd" > > else > > BOOT_DIR_ABS="/boot" > > IMAGE="initramfs-${KERNEL_VERSION}.img" > > fi It you would, could you create a unitied diff and swap the original and new file like this? diff -u 50-dracut.install.orig 50-dracut.install Your version would probably work as well, as would the original Fedora 39 version. In the patch file "0001-feat-kernel-install-do-nothing-when-KERNEL_INSTALL_I.patch" it was mentioned (Cherry-picked commit f47bcdd7342ca0d46b889e712a1c7446e18434bc from PR#1825) which doesn't say in which git repository it came from. It looks like the picked cherry doesn't match the context where it was applied. Lukas, thanks for looking into it! Villy, like it says, it's PR 1825 (in the original upstream repo). Checking the original commit, I have indeed made a **big** mistake while applying the patch (unsure how it happened, sorry). I've applied the patch fix here: https://github.com/redhat-plumbers/dracut-fedora/pull/32 Included are also test builds and COPR repo. Please test if you can and let me know if it works for you. Thanks for the update. I installed it via dnf: dnf install https://kojipkgs.fedoraproject.org//work/tasks/2129/120312129/dracut-config-rescue-102-1.1.fc40.x86_64.rpm https://kojipkgs.fedoraproject.org//work/tasks/2129/120312129/dracut-102-1.1.fc40.x86_64.rpm I have a system that ended up with duplicate kernel packages in the RPMDB because of the breakage: # rpm -qa | grep kernel | sort kernel-6.9.7-200.fc40.x86_64 kernel-core-6.9.7-200.fc40.x86_64 kernel-core-6.9.7-200.fc40.x86_64 kernel-modules-6.9.7-200.fc40.x86_64 kernel-modules-core-6.9.7-200.fc40.x86_64 I tested this again via a: `dnf -y reinstall kernel-core` dnf reported no errors at this point, and the RPMDB seems to have been cleaned up from the duplicate: # rpm -qa | grep kernel | sort kernel-6.9.7-200.fc40.x86_64 kernel-core-6.9.7-200.fc40.x86_64 kernel-modules-6.9.7-200.fc40.x86_64 kernel-modules-core-6.9.7-200.fc40.x86_64 I'll have to wait until a new kernel comes out to ensure the upgrade path works - but this looks more promising than previous results. Works for me too Thanks for checking! (In reply to Pavel Valena from comment #25) > Lukas, thanks for looking into it! > > Villy, like it says, it's PR 1825 (in the original upstream repo). > Checking the original commit, I have indeed made a **big** mistake while > applying the patch (unsure how it happened, sorry). > > I've applied the patch fix here: > > https://github.com/redhat-plumbers/dracut-fedora/pull/32 > > Included are also test builds and COPR repo. Please test if you can and let > me know if it works for you. Testing kernel-install add and kernel-install remove for sd-boot and for grub2 and all works. Not test is 51-dracut-rescue.install which I don't install. It would have been simpler to just have separate plugin script to copy out the initrd from the staging area to the final destination and keep the upstream version unchanged. Such a plugin script may at some point be needed if someone would like to combine mkosi-initrd with grub2. That is, however, someone elses headache. FEDORA-2024-0495a4da6c (dracut-102-2.fc41) has been submitted as an update to Fedora 41. https://bodhi.fedoraproject.org/updates/FEDORA-2024-0495a4da6c FEDORA-2024-0495a4da6c (dracut-102-2.fc41) has been pushed to the Fedora 41 stable repository. If problem still persists, please make note of it in this bug report. F40 Has not yet been merged into stable. I've referenced this issue in F41 as well (hece it was auto-closed). FEDORA-2024-18215bc41f (dracut-102-2.fc40) has been submitted as an update to Fedora 40. https://bodhi.fedoraproject.org/updates/FEDORA-2024-18215bc41f FEDORA-2024-18215bc41f has been pushed to the Fedora 40 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-18215bc41f` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-18215bc41f See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates. (In reply to Fedora Update System from comment #33) > FEDORA-2024-18215bc41f (dracut-102-2.fc40) has been submitted as an update > to Fedora 40. > https://bodhi.fedoraproject.org/updates/FEDORA-2024-18215bc41f https://houseofhazards.com Thank you FEDORA-2024-18215bc41f (dracut-102-2.fc40) has been pushed to the Fedora 40 stable repository. If problem still persists, please make note of it in this bug report. This report is good example of why it's foolhardy to not always use 'set -u' (or '#!/usr/bin/sh -u') to detect uninitialized variables. Worst case scenario of what, for example, a typo could do (DO NOT TRY THIS!): tmpdir=temp.$$ ... rm -rf $HOME/$tpmdir |