Hi Christian,

Thanks for reporting this issue!

92-crashkernel.install is actually an kernel installation/uninstallation hook [1] which is executed by kernel-install when a kernel is installed/removed. It's not a RPM scriptlet as mentioned in this github issue comment[2]. Currently, I'm working on Bug 1895258 and would like to take care of this bug as well during the implementation.  Setting `container=...` in the global env to check if it's running inside a container looks good to me. But I'm not sure how can I make use of this container variable. Am supposed to read this container variable as a bash environment variable? Btw, is there a recommended way to set up kernel cmdline  for osbuild? For anaconda, we have implemented an anaconda addon [3].

[1] https://www.freedesktop.org/software/systemd/man/kernel-install.html
[2] https://github.com/osbuild/osbuild-composer/issues/1819#issuecomment-966199997
[3] https://github.com/daveyoung/kdump-anaconda-addon

Hi Colby!


> 92-crashkernel.install is actually an kernel installation/uninstallation
> hook [1] which is executed by kernel-install when a kernel is
> installed/removed. It's not a RPM scriptlet as mentioned in this github
> issue comment[2]. 

True! The hooks are triggered via RPM scriptlets though (https://src.fedoraproject.org/rpms/kernel/blob/rawhide/f/kernel.spec#_2680) so in my mind they are always fused together, but you are of course technically correct ;)

> Currently, I'm working on Bug 1895258 and would like to
> take care of this bug as well during the implementation.  Setting
> `container=...` in the global env to check if it's running inside a
> container looks good to me. But I'm not sure how can I make use of this
> container variable. Am supposed to read this container variable as a bash
> environment variable?

According to https://systemd.io/CONTAINER_INTERFACE/:
"If you write software that wants to detect whether it is run in a container, please check /proc/1/environ and look for the container= environment variable. Do not assume the environment variable is inherited down the process tree. It generally is not."

I have made a PR to osbuild to have that env variable exposed in the container: https://github.com/osbuild/osbuild/pull/926

> Btw, is there a recommended way to set up kernel cmdline  for osbuild? For anaconda, we have implemented an anaconda addon [3].

We could implement a custom osbuild "stage" (https://github.com/osbuild/osbuild/tree/main/stages). That would be an alternative: do nothing in the hook if a container env is detected and then we would use a custom stage in osbuild to set the crashkernel variables. Maybe this is the cleaner way?

After discussing this within the team, I think relying on the kernel install hook is cleaner, since otherwise we have to repeat the numbers in the stage, or read the numbers for the kernel and then modify the BLS snippets. Which is basically what the install hook does, so that would just duplicate the code.

(In reply to Christian Kellner from comment #2)
> Hi Colby!
> 
> 
> > 92-crashkernel.install is actually an kernel installation/uninstallation
> > hook [1] which is executed by kernel-install when a kernel is
> > installed/removed. It's not a RPM scriptlet as mentioned in this github
> > issue comment[2]. 
> 
> True! The hooks are triggered via RPM scriptlets though
> (https://src.fedoraproject.org/rpms/kernel/blob/rawhide/f/kernel.spec#_2680)
> so in my mind they are always fused together, but you are of course
> technically correct ;)

Thanks for sharing the knowledge!

> 
> > Currently, I'm working on Bug 1895258 and would like to
> > take care of this bug as well during the implementation.  Setting
> > `container=...` in the global env to check if it's running inside a
> > container looks good to me. But I'm not sure how can I make use of this
> > container variable. Am supposed to read this container variable as a bash
> > environment variable?
> 
> According to https://systemd.io/CONTAINER_INTERFACE/:
> "If you write software that wants to detect whether it is run in a
> container, please check /proc/1/environ and look for the container=
> environment variable. Do not assume the environment variable is inherited
> down the process tree. It generally is not."
> 
> I have made a PR to osbuild to have that env variable exposed in the
> container: https://github.com/osbuild/osbuild/pull/926

Great, thanks for your work!

> 
> > Btw, is there a recommended way to set up kernel cmdline  for osbuild? For anaconda, we have implemented an anaconda addon [3].
> 
> We could implement a custom osbuild "stage"
> (https://github.com/osbuild/osbuild/tree/main/stages). That would be an
> alternative: do nothing in the hook if a container env is detected and then
> we would use a custom stage in osbuild to set the crashkernel variables.
> Maybe this is the cleaner way?

> After discussing this within the team, I think relying on the kernel install hook is cleaner, since otherwise we have to repeat the numbers in the stage, or read the numbers for the kernel and then modify the BLS snippets. Which is basically what the install hook does, so that would just duplicate the code.

I'd love to do it in the kernel hook provided by kexec-tools. There are a few more questions regarding implementation,
  1.  When osbuild builds RHEL, is kexec-tools installed before kernel or vice versa? If kexec-tools is installed later, the kernel hook won't have the chance to be executed. So we need to use RPM scriptlets to call grubby instead.
  2. Do osbuild needs to specify custom crashkernel value? In anaconda, we allow the users to specify their own custom value.  
  3.  Is osbuild responsible for building CoreOS image? CoreOS is very different. For example, it uses "rpm-ostree kargs" to manage kernel cmdline parameters and this command can't be run in RPM scriplet because system bus isn't available during RPM installation. 
  4.  I want to test my implementation for osbuild. Can you point me to some quick-start docs if they are available? 

Thanks!

> I'd love to do it in the kernel hook provided by kexec-tools. There are a
> few more questions regarding implementation,
>   1.  When osbuild builds RHEL, is kexec-tools installed before kernel or
> vice versa? If kexec-tools is installed later, the kernel hook won't have
> the chance to be executed. So we need to use RPM scriptlets to call grubby
> instead.

osbuild (composer) uses dnf for depsolving the package set and we use the order we get from there. If we want to ensure the order is correct we should probably fix this at the package dependency level, i.e. kernel depend (or recommend) kexec-tools and kexec-tools require grubby (I checked and kexec-tools currently only recommends it): https://src.osci.redhat.com/rpms/kexec-tools/blob/rhel-9.0.0/f/kexec-tools.spec#_74

Having said this, I checked what is the actual current installation order that we get and it is correct: grubby gets installed first, then kexec-tools and then the kernel (checked via https://github.com/osbuild/osbuild-composer/blob/main/test/data/manifests/rhel_90-x86_64-qcow2-boot.json). 

>   2. Do osbuild needs to specify custom crashkernel value? In anaconda, we
> allow the users to specify their own custom value.

Composer has a generic way of setting kernel command line arguments via blueprints. In RHEL 9 they would get set via `/etc/kernel/cmdline` which is written _before_ the rpms are installed. So the kexec-tools hook could check there if the user specified something else for crashkernel.

>   3.  Is osbuild responsible for building CoreOS image? CoreOS is very
> different. For example, it uses "rpm-ostree kargs" to manage kernel cmdline
> parameters and this command can't be run in RPM scriplet because system bus
> isn't available during RPM installation. 

No. CoreOS uses CoreOS assembler.

>   4.  I want to test my implementation for osbuild. Can you point me to some
> quick-start docs if they are available? 

A good start, including the overview of how osbuild and osbuild-composer fit together is probably https://www.osbuild.org/documentation. Feel free to reach out over Slack (or GChat) as well.

(In reply to Christian Kellner from comment #5)
> > I'd love to do it in the kernel hook provided by kexec-tools. There are a
> > few more questions regarding implementation,
> >   1.  When osbuild builds RHEL, is kexec-tools installed before kernel or
> > vice versa? If kexec-tools is installed later, the kernel hook won't have
> > the chance to be executed. So we need to use RPM scriptlets to call grubby
> > instead.
> 
> osbuild (composer) uses dnf for depsolving the package set and we use the
> order we get from there. If we want to ensure the order is correct we should
> probably fix this at the package dependency level, i.e. kernel depend (or
> recommend) kexec-tools and kexec-tools require grubby (I checked and
> kexec-tools currently only recommends it):
> https://src.osci.redhat.com/rpms/kexec-tools/blob/rhel-9.0.0/f/kexec-tools.
> spec#_74
> 
> Having said this, I checked what is the actual current installation order
> that we get and it is correct: grubby gets installed first, then kexec-tools
> and then the kernel (checked via
> https://github.com/osbuild/osbuild-composer/blob/main/test/data/manifests/
> rhel_90-x86_64-qcow2-boot.json). 
> 
> >   2. Do osbuild needs to specify custom crashkernel value? In anaconda, we
> > allow the users to specify their own custom value.
> 
> Composer has a generic way of setting kernel command line arguments via
> blueprints. In RHEL 9 they would get set via `/etc/kernel/cmdline` which is
> written _before_ the rpms are installed. So the kexec-tools hook could check
> there if the user specified something else for crashkernel.
> 
> >   3.  Is osbuild responsible for building CoreOS image? CoreOS is very
> > different. For example, it uses "rpm-ostree kargs" to manage kernel cmdline
> > parameters and this command can't be run in RPM scriplet because system bus
> > isn't available during RPM installation. 
> 
> No. CoreOS uses CoreOS assembler.
> 
> >   4.  I want to test my implementation for osbuild. Can you point me to some
> > quick-start docs if they are available? 
> 
> A good start, including the overview of how osbuild and osbuild-composer fit
> together is probably https://www.osbuild.org/documentation. Feel free to
> reach out over Slack (or GChat) as well.

Thanks for the helpful info! I've implemented the support for osbuild [1]. But when testing the implementation, running "grubby -info=ALL" in the kernel hook outputs something suspicious, 
 
index=0
kernel="/var/cache/osbuild-worker/osbuild-store/tmp/tmp2prywdy5object/tree/boot/vmlinuz-5.15.10-100.fc34.x86_64"
args="ro no_timer_check net.ifnames=0 console=tty1 console=ttyS0,115200n8"
root="UUID=76a22bf4-f153-4541-b6c7-0332c0dfaeac"
initrd="/var/cache/osbuild-worker/osbuild-store/tmp/tmp2prywdy5object/tree/boot/initramfs-5.15.10-100.fc34.x86_64.img"

Since the kernel hook is running in the chroot environment, shouldn't the kernel path be /boot/vmlinuz-5.15.10-100.fc34.x86_64? /var/cache/osbuild-worker/osbuild-store/tmp/tmp2prywdy5object/tree/boot/vmlinuz-5.15.10-100.fc34.x86_64 is the path on the host system.
Btw, I run "grubby --info=ALL" to get the kernel path given the kernel release because kernel-install only passes the kernel release e.g. 5.15.10-100.fc34.x86_64 to the kernel hook and updating the kernel cmdline using "grubby --update-kernel=KERNEL_PATH" requires specifying the kernel path.
 
[1] https://lists.fedoraproject.org/archives/list/kexec@lists.fedoraproject.org/message/CIXS4ZRMK4BFN4UBTZLE3XKYJOEMDEDQ/

Hey Coiby,

this is a known limitation of the grub2 probing code which also does not properly work with containers and bind-mounts. We should fix it at some point. We have a stage to fix up these paths: https://github.com/osbuild/osbuild/blob/main/stages/org.osbuild.fix-bls if you look at it, it contains some more information. Either way the path, although looking stupid, should be fine.

brew build:

https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1846493

(In reply to Christian Kellner from comment #5)
> > I'd love to do it in the kernel hook provided by kexec-tools. There are a
> > few more questions regarding implementation,
> >   1.  When osbuild builds RHEL, is kexec-tools installed before kernel or
> > vice versa? If kexec-tools is installed later, the kernel hook won't have
> > the chance to be executed. So we need to use RPM scriptlets to call grubby
> > instead.
> 
> osbuild (composer) uses dnf for depsolving the package set and we use the
> order we get from there. If we want to ensure the order is correct we should
> probably fix this at the package dependency level, i.e. kernel depend (or
> recommend) kexec-tools and kexec-tools require grubby (I checked and
> kexec-tools currently only recommends it):
> https://src.osci.redhat.com/rpms/kexec-tools/blob/rhel-9.0.0/f/kexec-tools.
> spec#_74
> 
> Having said this, I checked what is the actual current installation order
> that we get and it is correct: grubby gets installed first, then kexec-tools
> and then the kernel (checked via
> https://github.com/osbuild/osbuild-composer/blob/main/test/data/manifests/
> rhel_90-x86_64-qcow2-boot.json). 

Hi Christian,
FYI. I just notice kernel doesn't recommend kexec-tools but the installation order actually  doesn't matter at all.  Since dnf invokes kernel-install in the posttrans scriptlet (of kernel-core) which is always ran after all packages including kexec-tools and kernel have been installed, so hook 92-crashkernel.install installed by kexec-tools is guaranteed to be ran by kernel-install.

The upstream patch has been backported to c9s kexec-tools [1].

Tested against osbuild/osbuild-composer which has already removed deprecated crashkernel=auto option by following comment 22,

```
$ ./tools/image-info /var/tmp/osbuild-output/image/disk.img  > image-report.json
$ cat image-report.json
{
  "/etc/resolv.conf": [],
  "boot-environment": {
    "kernelopts": "root=UUID=6e4ff95f-f662-45ee-a82a-bdf44a2d0b75 console=tty0 console=ttyS0,115200n8 no_timer_check net.ifnames=0",
    "saved_entry": "ffffffffffffffffffffffffffffffff-5.14.0-4.el9.x86_64"
  },
  "bootloader": "grub",
  "bootmenu": [
    {
      "grub_arg": "--unrestricted",
      "grub_class": "kernel",
      "grub_users": "$grub_users",
      "initrd": "/initramfs-5.14.0-4.el9.x86_64.img $tuned_initrd",
      "linux": "/vmlinuz-5.14.0-4.el9.x86_64",
      "options": "root=UUID=6e4ff95f-f662-45ee-a82a-bdf44a2d0b75 console=tty0 console=ttyS0,115200n8 no_timer_check net.ifnames=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M $tuned_params",
      "title": "Red Hat Enterprise Linux (5.14.0-4.el9.x86_64) 9.0 (Plow)",
      "version": "5.14.0-4.el9.x86_64"
    }
  ],

```

[1] https://gitlab.com/redhat/centos-stream/rpms/kexec-tools/-/merge_requests/67

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: kexec-tools), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:3974