Bug 1866611

After seeing the `kexec_file_load failed: Required key not available` error that Kelvin saw in comment#11, I went looking for related problems/BZs and found https://access.redhat.com/solutions/5292541 + https://bugzilla.redhat.com/show_bug.cgi?id=1862072

I built a custom version of RHCOS 4.6 using the kernel linked in BZ#1862072 (and a newer version of `linux-firmware` for dependency purposes) and the linked version of `kexec-tools` on this BZ.

After appending `crashckernel=auto` to the kernel command line, I observed that the `kdump.service` successfully started without any intervention.

```
[core@cosa-devsh ~]$ rpm-ostree status
State: idle
Deployments:
* ostree://8221dfb9c9f5c204f7c7e7dcf39705a1f0790e0c995e09d678f7b74c2dab22cd
                   Version: 46.82.202011122225-0 (2020-11-12T22:28:33Z)
[core@cosa-devsh ~]$ rpm -q kernel kexec-tools linux-firmware
kernel-4.18.0-236.el8.x86_64
kexec-tools-2.0.20-35.el8.x86_64
linux-firmware-20200619-99.git3890db36.el8.noarch
[core@cosa-devsh ~]$ cat /proc/cmdline 
BOOT_IMAGE=(hd0,gpt1)/ostree/rhcos-864a012f295903d53f7dee3839542fbdfc00497dfb41e26bef30a6dd73b625ac/vmlinuz-4.18.0-236.el8.x86_64 rhcos.root=crypt_rootfs random.trust_cpu=on console=tty0 console=ttyS0,115200n8 ignition.platform.id=qemu rd.luks.options=discard ignition.firstboot ostree=/ostree/boot.1/rhcos/864a012f295903d53f7dee3839542fbdfc00497dfb41e26bef30a6dd73b625ac/0 crashkernel=auto
[core@cosa-devsh ~]$ systemctl status kdump
● kdump.service - Crash recovery kernel arming
   Loaded: loaded (/usr/lib/systemd/system/kdump.service; enabled; vendor preset: enabled)
   Active: active (exited) since Thu 2020-11-12 22:30:22 UTC; 22h ago
  Process: 1566 ExecStart=/usr/bin/kdumpctl start (code=exited, status=0/SUCCESS)
 Main PID: 1566 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4620)
   Memory: 0B
   CGroup: /system.slice/kdump.service

Nov 12 22:30:15 cosa-devsh dracut[1891]: *** Generating early-microcode cpio image ***
Nov 12 22:30:15 cosa-devsh dracut[1891]: *** Constructing GenuineIntel.bin ****
Nov 12 22:30:15 cosa-devsh dracut[1891]: *** Store current command line parameters ***
Nov 12 22:30:15 cosa-devsh dracut[1891]: Stored kernel commandline:
Nov 12 22:30:15 cosa-devsh dracut[1891]:  rd.luks.uuid=luks-00000000-0000-4000-a000-000000000002
Nov 12 22:30:15 cosa-devsh dracut[1891]: *** Creating image file '/boot/ostree/rhcos-864a012f295903d53f7dee3839542fbdfc00497dfb41e26bef30a6dd73b625ac/initramfs-4.18.0-236.el8.x86_64kdump.img' ***
Nov 12 22:30:22 cosa-devsh dracut[1891]: *** Creating initramfs image file '/boot/ostree/rhcos-864a012f295903d53f7dee3839542fbdfc00497dfb41e26bef30a6dd73b625ac/initramfs-4.18.0-236.el8.x86_64kdump.img' done ***
Nov 12 22:30:22 cosa-devsh kdumpctl[1566]: kexec: loaded kdump kernel
Nov 12 22:30:22 cosa-devsh kdumpctl[1566]: Starting kdump: [OK]
Nov 12 22:30:22 cosa-devsh systemd[1]: Started Crash recovery kernel arming.
[core@cosa-devsh ~]$ ls -latr /boot/ostree/rhcos-864a012f295903d53f7dee3839542fbdfc00497dfb41e26bef30a6dd73b625ac/
total 132453
-rwxr-xr-x. 1 root root  9571696 Nov 12 22:29 vmlinuz-4.18.0-236.el8.x86_64
-rw-r--r--. 1 root root 81682027 Nov 12 22:29 initramfs-4.18.0-236.el8.x86_64.img
drwxrwxr-x. 3 root root     1024 Nov 12 22:29 ..
-rw-r--r--. 1 root root      160 Nov 12 22:29 .vmlinuz-4.18.0-236.el8.x86_64.hmac
-rw-------. 1 root root 44366292 Nov 12 22:30 initramfs-4.18.0-236.el8.x86_64kdump.img
drwxrwxr-x. 2 root root     1024 Nov 12 22:30 .
```

From the RHCOS perspective, it appears that the changes to `kexec-tools` are working as intended.

OK now we need to decide whether to backport this to 8.3.z or just cherry-pick into RHCOS 4.7.

Do you guys have a preference?  I lean a bit to the latter because this fix only helps CoreOS systems as far as I know, and it's a low-impact package.
Assuming the 8.4 binary build works on 8.3 (no new glibc versions or shared library deps) then that seems viable.

(In reply to Colin Walters from comment #15)
> OK now we need to decide whether to backport this to 8.3.z or just
> cherry-pick into RHCOS 4.7.
> 
> Do you guys have a preference?  I lean a bit to the latter because this fix
> only helps CoreOS systems as far as I know, and it's a low-impact package.
> Assuming the 8.4 binary build works on 8.3 (no new glibc versions or shared
> library deps) then that seems viable.

I also prefer the cherry-pick way, 8.4 package should just work on 8.3

(In reply to Kairui Song from comment #16)
> (In reply to Colin Walters from comment #15)
> > OK now we need to decide whether to backport this to 8.3.z or just
> > cherry-pick into RHCOS 4.7.
> > 
> > Do you guys have a preference?  I lean a bit to the latter because this fix
> > only helps CoreOS systems as far as I know, and it's a low-impact package.
> > Assuming the 8.4 binary build works on 8.3 (no new glibc versions or shared
> > library deps) then that seems viable.
> 
> I also prefer the cherry-pick way, 8.4 package should just work on 8.3

Hmm, directly use 8.4 rpm in 8.3 might not work for some cases, such as we have some Hyper-V workarounds in 8.3 but reverted in 8.4 because of the kernel backport.  But I think cherry-pick the patches for this bz should just work fine.

(In reply to Colin Walters from comment #15)
> OK now we need to decide whether to backport this to 8.3.z or just
> cherry-pick into RHCOS 4.7.
> 
> Do you guys have a preference?  I lean a bit to the latter because this fix
> only helps CoreOS systems as far as I know, and it's a low-impact package.
> Assuming the 8.4 binary build works on 8.3 (no new glibc versions or shared
> library deps) then that seems viable.

Dave Young just mentioned some issue with cherry-picking, there are some kernel workaround for HyperV issue in 8.3 that are reverted in 8.4, if simply cherry-pick the package, it might break kdump on HyperV machines. So maybe it's better to create a zstream bug and backport the patches for this bug.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (kexec-tools bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1607