Bug 1768132 - if new kernel version needs new rescue kernel, say so or make it
Summary: if new kernel version needs new rescue kernel, say so or make it
Keywords:
Status: POST
Alias: None
Product: Fedora
Classification: Fedora
Component: dracut
Version: 38
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-11-02 19:48 UTC by Nick Levinson
Modified: 2024-01-09 22:16 UTC (History)
37 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-12-13 15:13:43 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Nick Levinson 2019-11-02 19:48:50 UTC
Description of problem:
A new Fedora version probably should not use an old rescue kernel.

Version-Release number of selected component (if applicable):
Unknown. I guessed the component.

How reproducible:
Always.

Steps to Reproduce:
1. Boot.
2. When list of kernels available for booting shows up, see name of current rescue kernel.
3. If rescue kernel is for old Fedora, presume use will cause a problem which conflicts with the mission of being a rescue kernel.

Actual results:
I upgraded F30 to F31 but the rescue kernel, per its name, is for F30.

Expected results:
A reminder to make a new rescue kernel (not everyone who upgrades is geeky enough to recognize the need before using the rescue kernel) or that the upgrade process automatically creates a new rescue kernel.

Additional info:
I didn't notice the problem for a couple of days after upgrading, and I'm geeky. Given the potential importance, we should not wait until we need it to try to make one.
I Googled but found no instruction for creating a new rescue kernel; however, I assume some kind of method exists somewhere. If you add the reminder, please point us to instructions on how.

Comment 1 Jiri Konecny 2020-03-04 12:56:47 UTC
I guess this is about kernel packaging. Switching component.

Comment 2 Justin M. Forbes 2020-03-04 16:29:01 UTC
Why does a known working rescue kernel cause a problem?  You can technically run an F30 kernel, F31 kernel, or even a rawhide kernel, and it will work for rescue purposes. The only problems you would have there would be with 3rd party modules, which should not be included in a rescue image to begin with.  I am closing this as won't fix as there would be more risk in replacing the rescue kernel than not doing so.

Comment 3 Nick Levinson 2020-03-14 18:45:35 UTC
Okay, but it looks like Fedora is offering to rescue by simply letting us choose the rescue kernel at bootup. It turns out it's not, so something needs to be changed, for the rescue kernel to be useful.

Per https://docs.fedoraproject.org/en-US/Fedora/13/html/Installation_Guide/s1-rescuemode-boot.html (as accessed 3-7-20), we would normally boot from another medium, thus could use any of a variety of OSes, distros, and spins, thus probably an external medium, but therefore not a rescue kernel on the newly-worrisome hard drive. Therefore, a user should have or make a rescue medium, but, to make one, a rescue kernel in the GRUB list is probably practically inaccessible even with a healthy HDD.

The rescue kernel in the GRUB list implies rescuing by booting the same HDD we're trying to rescue from. If we do that, we'd be using whatever apps are on the HDD. That may require, for app compatibility, frequent updates to the rescue kernel. If that's a bad idea, then users should not find the rescue kernel in the GRUB list and also should be advised (at installation and occasionally thereafter) to make a rescue medium and where to find what should be included in that medium, e.g., a rescue kernel and apps compatible with it. Perhaps that could be in a folder arranged so that a rescue image can easily be created; perhaps it could be an .iso file accompanied by a readme advising writing to a writable optical disc without locking against subsequent writes in case more apps should be added.

A rescue from the HDD might be needed when CDs, thumb flash, etc. can't be used. And it's not impossible. Per the above URL, we'll be "prompt[ed] . . . to select where a valid rescue image is located. Select from Local CD-ROM, Hard Drive, NFS image, FTP, or HTTP. The location selected must contain a valid installation tree, and the installation tree must be for the same version of Fedora as the Fedora disk from which you booted." I don't entirely understand this and don't disagree, but, in short, I don't know how to create rescue media using this instruction.

I don't know whether what's needed is clearer documentation, moving of the rescue kernel into a directory with other apps to support making an external rescue medium, or setting up the rescue kernel in the GRUB list so that only the relevant apps are used unless the user confirms allowing execution of some or all other apps.

The title of this bug should be changed, but how depends on which problem and therefore which solution would prevail.

I'm reopening just to consider this issue. If you still prefer wontfix, go ahead. You'd know the programming issues better than I would.

Thank you.

Comment 4 Steve 2020-03-14 20:42:49 UTC
(In reply to Nick Levinson from comment #0)
...
> Actual results:
> I upgraded F30 to F31 but the rescue kernel, per its name, is for F30.
...

I've been wondering about exactly when that kernel gets installed too.

However, the term "rescue" is very ambiguous. If you look at the size of the rescue initramfs, it is much bigger than the initramfs for standard installs. That's because it contains more kernel modules. That's really the only significant difference.

Thus:

$ ls -l /boot/initramfs-0-rescue-*.img /boot/initramfs-5.5.9-100.fc30.x86_64.img
-rw-------. 1 root root 70886763 Nov 27  2018 /boot/initramfs-0-rescue-7d0c239ba497409ea6db764545f97c10.img <<<< Note size.
-rw-------. 1 root root 26046408 Mar 14 06:26 /boot/initramfs-5.5.9-100.fc30.x86_64.img

If you want to see what files are in the rescue initramfs, run "lsinitrd":

# lsinitrd /boot/initramfs-0-rescue-*.img | fgrep modules | wc -l
1197

# lsinitrd /boot/initramfs-5.5.9-100.fc30.x86_64.img | fgrep modules | wc -l
238

There may be another important difference: "dnf" won't automatically remove the rescue kernel when a new kernel is installed.

Comment 5 Steve 2020-03-14 20:45:13 UTC
(In reply to Nick Levinson from comment #3)
...
> I'm reopening just to consider this issue.
...

Try changing the component to "dracut":

Bug 1581478 - dnf system upgrade does not upgrade rescue kernel

Comment 6 Steve 2020-03-14 21:07:52 UTC
(In reply to Nick Levinson from comment #0)
...
> Actual results:
> I upgraded F30 to F31 but the rescue kernel, per its name, is for F30.
...

Well, it's even older on my system: :-)

$ file /boot/vmlinuz-0-rescue-7d0c239ba497409ea6db764545f97c10
/boot/vmlinuz-0-rescue-7d0c239ba497409ea6db764545f97c10: Linux kernel x86 boot executable bzImage, version 4.16.3-301.fc28.x86_64 (mockbuild.fedoraproject.org) #1 SMP Mon Apr 23 21:59:58 , RO-rootFS, swap_dev 0x7, Normal VGA

"fc28" is two releases older than my current system:

$ uname -r
5.5.9-100.fc30.x86_64

When you say "per its name", could you be more explicit? What do you get for this:

$ ls -l /boot/*rescue*

Comment 7 Nick Levinson 2020-03-23 00:47:17 UTC
Per comment 4:

My guess is that a rescue kernel gets installed during a new installation of Fedora but not during an update or upgrade. If someone deleted the rescue kernel and then updates or upgrades, I don't know what happens.

lsinitrd listed hundreds of files but it was too much for me to figure out what apps I'd have when running a rescue. Those of us who rely on GUI because we so rarely use CLI that we don't maintain an intimate recall of commands will have the problem that apps, including utilities, seem often to have different names for GUI than they do for CLI, so reviewing the lsinitrd output is less informative for us.

Per comment 5: I'll see what happens with a component change.

Per comment 6:

I mentioned the rescue kernel's name, which was perhaps unfortunate on my part. I was referring to how the GRUB list identifies it, which is not exactly how ls -l /boot/*rescue* identifies it, but may be close enough.

ls -l /boot/*rescue* outputs this:

-rw-------. 1 root root 71061086 Dec  2  2018 /boot/initramfs-0-rescue-627807d457c340cf96d966eb875b9328.img -rwxr-xr-x. 1 root root  8618168 Dec  2  2018 /boot/vmlinuz-0-rescue-627807d457c340cf96d966eb875b9328

The GRUB list lately (before the update to 5.5.10-200.fc31) said this:

Fedora (5.5.9-200.fc31.x86_64) 31 (Thirty One)
. . . . .
Fedora (0-rescue-627807d457c340cf96d966eb875b9328) 30 Workstation Editi

To the right of the rescue line is a right-pointing arrow character and then the edge of the screen. Keyboarding either right-arrow or Enter got an old-style HDD login screen, not the rest of the GRUB list line.

Your current system kernel was 5.5.9-100.fc30.x86_64 and mine was 5.5.9-200.fc31.x86_64 (at least before Fedora updated to 5.5.10), so we were both running 5.5.9 but you were on F30 and I was on F31. I forgot or never knew what "200" or "100" means. I'm a little surprised that you had 5.5.9 for F30, because I think that means you passed up on upgrading to F31, but that likely doesn't affect either of our relationships to this issue.

Comment 8 Steve 2020-03-23 04:09:32 UTC
(In reply to Nick Levinson from comment #7)

Thanks for your reply and clarifications.

> Per comment 4:
> 
> My guess is that a rescue kernel gets installed during a new installation of
> Fedora but not during an update or upgrade. If someone deleted the rescue
> kernel and then updates or upgrades, I don't know what happens.
...

The kernel installer shell script simply checks that the rescue vmlinuz, initramfs, and .conf files are all present. If any one is missing (or renamed) a new rescue kernel is installed.

I wrote up a longer comment for Bug 1815102, Comment 78, but will simply post the un-annotated technical details below. My conclusion is that a preinstall script could optionally rename the three files before the kernel installation. That would avoid any need to modify the existing scripts. The catch in that proposal is the word "optional" -- how would a user who cares specify that a new rescue kernel is to be installed?

Technical details:

$ rpm -ql dracut-config-rescue.x86_64
/etc/kernel/postinst.d/51-dracut-rescue-postinst.sh
/usr/lib/dracut/dracut.conf.d/02-rescue.conf
/usr/lib/kernel/install.d/51-dracut-rescue.install

$ less -N /usr/lib/kernel/install.d/51-dracut-rescue.install
...
     76 case "$COMMAND" in
     77     add)
     78         [[ -f "$LOADER_ENTRY" ]] && [[ -f "$BOOT_DIR_ABS/$KERNEL" ]] \
     79             && [[ -f "$BOOT_DIR_ABS/$INITRD" ]] && exit 0
...

BTW, the long number in the name is:
$ cat /etc/machine-id
$ man machine-id

Comment 9 Steve 2020-03-23 15:26:03 UTC
(In reply to Nick Levinson from comment #3)
...
> If we do that, we'd be using whatever apps are on the HDD.
...

Those apps would actually be Linux command-line utilities.

If you want to see how rescue mode looks, add "rd.break" to the kernel command-line from grub2 and boot. That will drop you into the dracut shell, where you have access to a limited selection of Linux commands.

If you want more apps, "chroot /sysroot".

That works with a non-rescue kernel too, although there are some differences, based on a quick test in a VM.

Obviously, there are failure cases where none of that is possible. But in other cases, such as with a mangled /etc/fstab, all that is needed is to edit the file and reboot.

Documentation:

$ man dracut.cmdline

Comment 10 Steve 2020-03-23 15:58:28 UTC
(In reply to Steve from comment #8)
...
> My conclusion is that a preinstall script could optionally rename the three files before the kernel installation.
> That would avoid any need to modify the existing scripts.
> The catch in that proposal is the word "optional" -- how would a user who cares specify that a new rescue kernel is to be installed?
...

Here is a slightly more refined proposal:

1. Retain the current behavior -- for new installs, a rescue kernel is created and installed, and it is never updated automatically.

2. For users who want to update their rescue kernel, add a command that lets users manage their rescue kernel[s]: dracut-rescue-manager.

The dracut-rescue-manager command would let users add or update rescue kernels. Removal would also be possible with a suitable warning about not having a rescue kernel on your system.

The default behavior would be to add a new rescue kernel based on the currently running kernel and to not remove any existing rescue kernels.

The kernel names would follow the current naming scheme:

"vmlinuz-0-rescue- ..."
"vmlinuz-1-rescue- ..."
"vmlinuz-2-rescue- ..."

Although, realistically, how many people would really want more than one rescue kernel on their system? :-)

Comment 11 Steve 2020-03-23 16:15:01 UTC
(In reply to Steve from comment #10)
...
> The kernel names would follow the current naming scheme:
> 
> "vmlinuz-0-rescue- ..."
> "vmlinuz-1-rescue- ..."
> "vmlinuz-2-rescue- ..."
...

There is an implementation issue with that -- 51-dracut-rescue.install currently looks for an exact name match, so it would need to look for a pattern instead. Simplistically:

"vmlinuz-.*-rescue- ..."
         ^^

For the record, the dracut git repo is here:
https://git.kernel.org/pub/scm/boot/dracut/dracut.git/

51-dracut-rescue.install is here:
https://git.kernel.org/pub/scm/boot/dracut/dracut.git/tree/51-dracut-rescue.install

Comment 12 Steve 2020-03-24 04:39:17 UTC
(In reply to Nick Levinson from comment #7)
...
> To the right of the rescue line is a right-pointing arrow character and then
> the edge of the screen. Keyboarding either right-arrow or Enter got an
> old-style HDD login screen, not the rest of the GRUB list line.
...

I'm not sure why that would be happening, but rather than got off-topic in this bug, I suggest opening a new bug against grub2.

Comment 13 Nick Levinson 2020-03-27 22:58:53 UTC
On comment 9:

> Those apps would actually be Linux command-line utilities.

Usually, I guess. The CLI utilities tend to be stable and critical, so they'd be more likely to be used for a rescue, but not always.

The dracut manual is interesting (and recently updated). It's been years since I've had to try a major rescue and hopefully it'll be years before the next time and hopefully I'll remember this tool then.

On comment 10 (comment 11 is over my head today):

I like these ideas. Most people like simplicity, even Linux/BSD users, but as long as these are easily-invoked options I think these are helpful.

Probably, creating new rescue kernels should default to deleting old ones on a principle like that for main kernels: a maximum of three coexisting unless the user opts to override the limit in either direction. Otherwise, some day, some symptom of an overfull HDD will lead to chuckles from the repair guru about finding 3,294 rescue kernels.

Three might make sense especially without timing them to coincide with the three main kernels, like if a user updates a rescue kernel only once every 6 months to ensure compatibility. Even so, the latest rescue kernel will likely suffice (unless none would ever work in a situation).

I think of relatively inexperienced users, so I would save people in a growing panic about their data from trying to parse that 0=1, 1=2, and so on, so I'd number from 1 up and I'd put the highest number on the top. The saving in bytes from counting starting at 0 is infinitesmal.

On comment 12:

Failure to wrap: I'm not sure it's a bug. If it is, I doubt it's important, since enough is shown. But I think I can't narrow the screen during boot the way I can in the Firefox browser, so I don't know if the wrap-lack hinders a user on a smartphone booting Linux.

If you were referring to the login screen, I don't think that's a bug. I think that's how the login screen looked when F30 was new.

If I misunderstand, please let me know and maybe I'll post a bug report, or you may post one.

Comment 14 Steve 2020-03-29 01:42:30 UTC
(In reply to Nick Levinson from comment #13)
...
> ... creating new rescue kernels should default to deleting old ones on
> a principle like that for main kernels: a maximum of three coexisting unless
> the user opts to override the limit in either direction.
...

That's a good suggestion. dnf retains three kernels by default, unless the number is changed in dnf.conf.

Since I sometimes test with more than three kernels at once, I have changed the value to "0", which means to never remove kernels during a kernel update:

$ grep installonly_limit /etc/dnf/dnf.conf
#installonly_limit=3
installonly_limit=0

There is a problem, however: If dracut-rescue-manager reads dnf.conf, there would be a dependency between dracut and dnf that could cause future maintenance or compatibility problems.

...
> I think of relatively inexperienced users, so I would save people in a
> growing panic about their data from trying to parse that 0=1, 1=2, and so
> on, so I'd number from 1 up and I'd put the highest number on the top. The
> saving in bytes from counting starting at 0 is infinitesmal.
...

OK, I suppose the numbers in the file name could always be compacted so that the oldest is always 1, the next oldest is 2, etc.

However, the numbering must be compatible with standard kernel version numbers, so that they can be sorted correctly:

vmlinuz-5.5.10-100.fc30.x86_64
vmlinuz-5.5.9-100.fc30.x86_64
vmlinuz-5.4.21-100.fc30.x86_64
vmlinuz-5.4.19-100.fc30.x86_64
vmlinuz-3-rescue-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
vmlinuz-2-rescue-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
vmlinuz-1-rescue-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

So there would be a problem if there were a "vmlinuz-6-rescue-...", since it would sort above all the production kernel names. A possible solution would be to add a leading "0":

vmlinuz-03-rescue-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
vmlinuz-02-rescue-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
vmlinuz-01-rescue-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

Anyone who retains more than nine rescue kernels deserves what they get: :-)
vmlinuz-010-rescue-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

Comment 15 Steve 2020-03-29 01:48:13 UTC
(In reply to Steve from comment #14)
...
> There is a problem, however: If dracut-rescue-manager reads dnf.conf, there
> would be a dependency between dracut and dnf that could cause future
> maintenance or compatibility problems.
...

Dracut already supports configuration files, so that isn't really a problem:

/etc/dracut.conf
/etc/dracut.conf.d/

Comment 16 Steve 2020-03-31 12:32:43 UTC
(In reply to Nick Levinson from comment #13)
...
> I think of relatively inexperienced users, so I would save people in a
> growing panic about their data from trying to parse that 0=1, 1=2, and so
> on, so I'd number from 1 up and I'd put the highest number on the top. The
> saving in bytes from counting starting at 0 is infinitesmal.
...

The file names would also be more informative if they contained the kernel version instead of /etc/machine-id. For example:

vmlinuz-1-rescue-5.6.0-0.rc5.git0.2.fc32.x86_64

See:

Bug 1819184 - rescue kernel name should use hashed machine-id

Comment 17 Steve 2020-03-31 18:56:51 UTC
(In reply to Steve from comment #10)
...
> 2. For users who want to update their rescue kernel, add a command that lets users manage their rescue kernel[s]: dracut-rescue-manager.
...

Another approach would be to integrate the rescue kernels with the RPM package management system. Basically, the idea would be that dracut-rescue-manager would create an RPM package for each rescue kernel. From then on, the dnf command could be used to remove them or to update them.

As it is, the rescue kernels are completely outside the RPM package management system.

Comment 18 Steve 2020-04-01 14:52:20 UTC
(In reply to Steve from comment #17)
...
> Another approach would be to integrate the rescue kernels with the RPM
> package management system. Basically, the idea would be that
> dracut-rescue-manager would create an RPM package for each rescue kernel.
> From then on, the dnf command could be used to remove them or to update them.
...

The rescue kernel package wouldn't need to be much different from existing kernel packages, because the main difference would be with how the initramfs is built.

Indeed, it might even be possible to integrate the rescue kernel install logic WITH standard kernel packages.

Comment 19 Steve 2020-04-01 15:22:28 UTC
(In reply to Steve from comment #18)
...
> Indeed, it might even be possible to integrate the rescue kernel install logic WITH standard kernel packages.

That could be done with an optional subpackage called:

kernel-rescue-0:5.6.0-300.fc32.x86_64

That name follows the existing naming convention:

$ dnf repoquery --installed kernel\*-5.6.0-300.fc32.x86_64
kernel-0:5.6.0-300.fc32.x86_64
kernel-core-0:5.6.0-300.fc32.x86_64
kernel-headers-0:5.6.0-300.fc32.x86_64
kernel-modules-0:5.6.0-300.fc32.x86_64
kernel-modules-extra-0:5.6.0-300.fc32.x86_64

NB: The kernel-rescue subpackage would not have any actual kernel code -- it would have a dependency on other kernel packages.

Comment 20 Nick Levinson 2020-04-03 00:31:40 UTC
On comment 14: Perhaps insert the production kernel number into the rescue kernel's name and still maintain the numbering system (1-9 or 01-09) being proposed? Knowing it's 5.5.11 could be very helpful and likely difficult to dig out from the object code before rescuing. A nice place to put it could be after "rescue-", perhaps as "vmlinuz-03-rescue-5.5.11-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" or "vmlinuz-03-rescue-modelkernel-5.5.11-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa".

On comment 16: Dropping the machine-id looks like a good idea.

Comment 21 Ben Cotton 2020-11-03 17:05:00 UTC
This message is a reminder that Fedora 31 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 31 on 2020-11-24.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '31'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 31 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 22 Nick Levinson 2020-11-18 01:06:16 UTC
I'm keeping open by updating the version. There's been no change in the problem and the rescue kernel numbering has not changed.

Here are my boot choices (hopefully, I didn't make 1-2 errors in the rescue hex string) as of 11-25-20 on my system kept evergreen:

Fedora (5.8.18-300.fc33.x86_64) 33 (Workstation Edition)
Fedora (5.8.17-300.fc33.x86_64) 33 (Workstation Edition)
Fedora (5.8.16-300.fc33.x86_64) 33 (Workstation Edition)
Fedora (0-rescue-0088e3e799354b9095ec4a4cb749b601) 32 (Thirty Two)

On comment 20: Embedding a number (e.g., "01") would require renumbering the kernel name whenever the GRUB list is updated.

Comment 23 Nicolas Sapa 2021-01-16 07:11:52 UTC
The rescue kernel can be very old.
On my fc33 system:

# head -n3 /boot/loader/entries/d909406eaaac4b67b61435aa9a2dab37-0-rescue.conf 
title Fedora (0-rescue-d909406eaaac4b67b61435aa9a2dab37) 29 (Twenty Nine)
version 0-rescue-d909406eaaac4b67b61435aa9a2dab37
linux /vmlinuz-0-rescue-d909406eaaac4b67b61435aa9a2dab37
# file /boot/vmlinuz-0-rescue-d909406eaaac4b67b61435aa9a2dab37
/boot/vmlinuz-0-rescue-d909406eaaac4b67b61435aa9a2dab37: Linux kernel x86 boot executable bzImage, version 4.5.5-300.fc24.x86_64 (mockbuild.fedoraproject.org) #1 SMP Thu May 19 13:05:32 UTC 2016, RO-rootFS, swap_dev 0x5, Normal VGA

Having a kernel from nearly 10 release ago is probably not useful.
Maybe dnf system-upgrade should cleanup this.

Comment 24 Endre Bjørsvik 2021-01-17 17:58:24 UTC
I also use a fc33 system that has been upgraded several times over many years using system-upgrade without seeing a new rescue image:
# ll /boot
-rw-rw-r--. 1 root root  50M 2015-12-21 21:52 initramfs-0-rescue-0d52ad18abb84a01b73ad6776eca7e8e.img
-rwxr-xr-x. 1 root root 5,8M 2015-12-21 21:52 vmlinuz-0-rescue-0d52ad18abb84a01b73ad6776eca7e8e
# file /boot/bak-old/vmlinuz-0-rescue-0d52ad18abb84a01b73ad6776eca7e8e
vmlinuz-0-rescue-0d52ad18abb84a01b73ad6776eca7e8e: Linux kernel x86 boot executable bzImage, version 4.2.3-300.fc23.x86_64 (mockbuild.fedoraproject.org) #1 SMP Mon Oct 5 15:42:54 UTC 2015, RO-rootFS, swap_dev 0x5, Normal VGA

This image seems to be from the inital fc23 installation. When attempted to boot the computer using this old image, I was thrown into an emergency terminal. Keeping such an old rescue image seems to not have any value.
In order to get a newer rescue image I performed the following steps from a working session (using vanilla kernel):
- rm /boot/vmlinuz-0-rescue-0d52ad18abb84a01b73ad6776eca7e8e
- rm /boot/initramfs-0-rescue-0d52ad18abb84a01b73ad6776eca7e8e.img
- dnf reinstall kernel-core

The newly generated rescue image boots perfectly fine. I like Nicolas' suggestion of letting system-upgrade generate this.

Comment 25 Ben Cotton 2021-11-04 13:57:50 UTC
This message is a reminder that Fedora 33 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 33 on 2021-11-30.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '33'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 33 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 26 Ben Cotton 2021-11-04 14:27:11 UTC
This message is a reminder that Fedora 33 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 33 on 2021-11-30.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '33'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 33 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 27 Ben Cotton 2021-11-04 15:24:50 UTC
This message is a reminder that Fedora 33 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 33 on 2021-11-30.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '33'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 33 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 28 Nick Levinson 2021-11-20 21:06:01 UTC
I'm keeping this open because I haven't seen a change and it appears that there was some good discussion about needs and solutions above.

My system was just upgraded from 34 to 35 but the rescue kernel stayed the same. It seems dubious that an old kernel should be used even with the limited subset of apps meant for a rescue, since any app can have been upgraded and might need a newer kernel.

By the way -- I don't know if this is relevant -- I installed F34 from the same live disc on 3 nearly identical laptops, each a refurbished Dell Latitude E4310, without updating Fedora, and the rescue kernel filename on each has a different hex string.

Comment 29 Billy DeVincentis 2022-02-15 03:10:45 UTC
Just wanted to point out, if you want new rescue kernel which is pretty simple to do
1- in /etc/dracut.conf.d/ create a file called 02-rescue.conf
2- file text should read    dracut_rescue_image="yes"
3- delete old recue kernel files from /boot   initramfs-0-rescue-****** and vmlinuz-0-rescue-******
4- run dnf reinstall kernel kernel-core
5- new rescue kernel will be reinstalled matching latest kernel version. This is useful if from time to time you change hardware as you will be able to boot into a new motherboard, new hardware easily. Then simply rerun dnf reinstall kernel kernel-core to create a kernel customized for your new hardware

Comment 30 Nicolas Sapa 2022-04-13 21:46:34 UTC
I confirm the issue was still present when I upgraded from 33 to 35.
I also confirm that removing /boot/*-0-rescue-* and reinstalling/upgrading kernel-core is enough to regenerate the rescue kernel.

Since these rescue kernel are very outdated, dnf system-upgrade should remove them before the first reboot.
When the upgrade process reach kernel-core, a rescue kernel will be generated.

@Nick:
The different hex string come from /etc/machine-id - an id that is supposed to be different for every system.

Comment 31 Justin M. Forbes 2022-04-13 22:37:38 UTC
(In reply to Nicolas Sapa from comment #30)
> I confirm the issue was still present when I upgraded from 33 to 35.
> I also confirm that removing /boot/*-0-rescue-* and reinstalling/upgrading
> kernel-core is enough to regenerate the rescue kernel.
> 
> Since these rescue kernel are very outdated, dnf system-upgrade should
> remove them before the first reboot.
> When the upgrade process reach kernel-core, a rescue kernel will be
> generated.


I would strongly argue that dnf system-upgrade do no such thing. The installer has booted the kernel to at least a gui enough to run anaconda, or a tui if you are doing such an install, but that is enough. It has gotten the system to the point of being able to install, partition disks, likely use network, and such.  It is also a fact that if the install kernel doesn't work on your system, there is nothing to recover, because you are doing a fresh install.  The dracut-system-update doesn't verify thing about the kernel it is installing, and this could leave you with not only a broken system update, but also a broken recovery.
Realistically, a new rescue kernel needs to be generated when you have changed the hardware, because the old kernel might not have a working driver for a new piece of hardware.  I suppose if you were to convert an xfs filesystem to use bigtime, you would need to make sure you rescue kernel was new enough to mount it. Anyway, point is, there are very few times that a new rescue kernel is actually needed, and many more opportunities to end up with a rescue kernel that doesn't work.  If you installed your system on f23, and upgraded it via dnf system-upgrade every release up to F36, As long as you didn't get new hardware, or convert filesystems, that f23 rescue kernel should still work just fine. 
There is discussion in the Workstation WG for a new graphical rescue environment, that would likely need more frequent updates, and perhaps a workable solution can be found there, but for the basic cli rescue kernel, I would rather rescue my f36 system in the above scenario with an old known good instead of failing to rescue with a new kernel that has never been tested on the system.

Comment 32 Endre Bjørsvik 2022-04-18 16:10:28 UTC
(In reply to Justin M. Forbes from comment #31)
> Realistically, a new rescue kernel needs to be generated when you have
> changed the hardware, because the old kernel might not have a working driver
> for a new piece of hardware.  I suppose if you were to convert an xfs
> filesystem to use bigtime, you would need to make sure you rescue kernel was
> new enough to mount it. Anyway, point is, there are very few times that a
> new rescue kernel is actually needed, and many more opportunities to end up
> with a rescue kernel that doesn't work.  If you installed your system on
> f23, and upgraded it via dnf system-upgrade every release up to F36, As long
> as you didn't get new hardware, or convert filesystems, that f23 rescue
> kernel should still work just fine. 

I agree that generating a new rescue kernel is something that should be done as sparingly and safely as possible. However, you claim that older rescue kernels always work on newer installations as long as the hardware has not changed. In my case, the rescue kernel from F23 was not functional after the system had been upgraded to F33 over many years. This was on a laptop with exactly the same hardware as when F23 was installed, so there must be something else in the mix that makes the rescue kernel break over time. Perhaps the newer installation with newer init depends on some new syscalls that does not exist in the rescue image?

Whatever it is, it seems like I can not universally trust the rescue kernel after some system upgrades, even though the hardware remains the same. So I still think there should be some mechanism in place that offer regeneration of rescue kernel at a safe point. Perhaps after first boot of a system upgrade?

Comment 33 Ben Cotton 2022-11-29 16:47:05 UTC
This message is a reminder that Fedora Linux 35 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 35 on 2022-12-13.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '35'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 35 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Comment 34 Ben Cotton 2022-12-13 15:13:43 UTC
Fedora Linux 35 entered end-of-life (EOL) status on 2022-12-13.

Fedora Linux 35 is no longer maintained, which means that it
will not receive any further security or bug fix updates. As a result we
are closing this bug.

If you can reproduce this bug against a currently maintained version of Fedora Linux
please feel free to reopen this bug against that version. Note that the version
field may be hidden. Click the "Show advanced fields" button if you do not see
the version field.

If you are unable to reopen this bug, please file a new report against an
active release.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 35 Sergio Basto 2022-12-31 04:51:42 UTC
(In reply to Billy DeVincentis from comment #29)
> Just wanted to point out, if you want new rescue kernel which is pretty
> simple to do
> 1- in /etc/dracut.conf.d/ create a file called 02-rescue.conf
> 2- file text should read    dracut_rescue_image="yes"
> 3- delete old recue kernel files from /boot   initramfs-0-rescue-****** and
> vmlinuz-0-rescue-******
> 4- run dnf reinstall kernel kernel-core
> 5- new rescue kernel will be reinstalled matching latest kernel version.
> This is useful if from time to time you change hardware as you will be able
> to boot into a new motherboard, new hardware easily. Then simply rerun dnf
> reinstall kernel kernel-core to create a kernel customized for your new
> hardware

if we look to scripts of kernel-core rpm package, with `rpm -q kernel-core-$(uname -r) --scripts | grep add` we got [1] 
After delete old rescue entries (`rm /boot/*-0-rescue-*`), run the `kernel-install add ...` ([1]) is enough to install rescue entry. 

moreover if we run kernel-install with --verbose, `/bin/kernel-install --verbose add 6.0.15-300.fc37.x86_64 /lib/modules/6.0.15-300.fc37.x86_64/vmlinuz` we will see the command that do the job is [2] 

moreover in /usr/lib/dracut/dracut.conf.d/02-rescue.conf we need be sure that dracut_rescue_image="yes" 


[1]
/bin/kernel-install add 6.0.15-300.fc37.x86_64 /lib/modules/6.0.15-300.fc37.x86_64/vmlinuz || exit $?

[2]
/usr/lib/kernel/install.d/51-dracut-rescue.install add 6.0.15-300.fc37.x86_64 /boot/9b64058eb4c74f9d9a4bd435329809eb/6.0.15-300.fc37.x86_64 /lib/modules/6.0.15-300.fc37.x86_64/vmlinuz

Comment 37 Ben Cotton 2023-02-07 15:12:56 UTC
This bug appears to have been reported against 'rawhide' during the Fedora Linux 38 development cycle.
Changing version to 38.

Comment 38 John Griffiths 2023-06-12 23:30:34 UTC
This has been going on for many new versions of Fedora. 

I think the solution could be as simple as an update to the optional tasks in the DNF upgrade documentation.

These commands have worked for me.

rm -f /boot/vmlinuz-0-rescue-* /boot/initramfs-0-rescue-*.img
/usr/lib/kernel/install.d/51-dracut-rescue.install add $(uname -r) "" /lib/modules/$(uname -r)/vmlinuz

Just put it in the upgrade documentation as an optional task possibly with a warning to backup the current rescue system before doing the update and to make sure the new rescue system works before discarding the backup.

Comment 39 Endre Bjørsvik 2023-06-13 06:55:45 UTC
I like John's suggestion. That would make me remember this step when I go through the upgrade. I usually follow the DNF upgrade documentation when upgrading to a new version of Fedora.

Comment 40 Tomasz Gąsior 2023-12-26 21:51:06 UTC
For best user experience rescue kernel and initramfs should be updated automatically with system upgrade (like from f38 to f39). Currently this does not happen automatically and rescue kernel generated from older releases does not work most of the time in current releases — which makes rescue option in GRUB boot menu just not useful at all.

I've just tried rescue kernel generated  on my laptop in fedora 38 release where I recently upgraded to fedora 39. The rescue kernel just was not able to boot so it's completely unuseful.

Rescue kernel regeneration should happen automatically without need for any user intervention. I've upgraded from f38 to f39 using built in GUI, gnome software, and as end user I don't want to care about need for rescue kernel regeneration. For me, it should just work everytime I will have to rescue my system. Currently, the "rescue" option provides just frustration and nothing more.

Comment 41 Tomasz Gąsior 2023-12-26 21:54:38 UTC
> I've just tried rescue kernel generated  on my laptop in fedora 38 release where I recently upgraded to fedora 39. The rescue kernel just was not able to boot so it's completely unuseful.

Adding more information here: I didn't change anything in my hardware in the meantime. Also, I use Fedora since 2019 and this situation happened many times with many releases, not just this specific. Rescue kernel is just not reliable because of that.


Note You need to log in before you can comment on or make changes to this bug.