RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1663864 - generated entries don't respect GRUB_CMDLINE_LINUX from /etc/default/grub
Summary: generated entries don't respect GRUB_CMDLINE_LINUX from /etc/default/grub
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: boom-boot
Version: 8.0
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: rc
: 8.0
Assignee: Bryn M. Reeves
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On:
Blocks: 1660923 1663940
TreeView+ depends on / blocked
 
Reported: 2019-01-07 08:51 UTC by Martin Pitt
Modified: 2023-07-20 12:39 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1663940 (view as bug list)
Environment:
Last Closed: 2023-07-19 17:33:25 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github snapshotmanager boom issues 14 0 None closed Try to parse existing kernel command line for "boom profile create --from-host" 2023-04-24 18:22:10 UTC

Internal Links: 1894407

Description Martin Pitt 2019-01-07 08:51:47 UTC
Description of problem: As the subject says. This means that serial console, kdump, and other settings don't work when booting into a rollback.


Version-Release number of selected component (if applicable):

boom-boot-0.9-6.el8.noarch
grub2-common-2.02-66.el8.noarch

How reproducible: Always

1. Create a snapshot:
   lvcreate --snapshot -n rollback1 --size 1G rhel/root
2. Create a boom profile (work around https://bugzilla.redhat.com/show_bug.cgi?id=1649423):
   boom profile create --from-host --uname-pattern el8
3. Create a boom entry for the rollback:
   boom create --title "update rollback test" --rootlv rhel/rollback1
4. Reboot into the snapshot

Actual results: Serial console not working, /proc/cmdline is missing all the options:

# cat /proc/cmdline
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-57.el8.x86_64 root=/dev/rhel/rollback1 ro rd.lvm.lv=rhel/rollback1


Expected results: GRUB_CMDLINE_LINUX settings are respected, like in

# cat /proc/cmdline
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-57.el8.x86_64 root=/dev/mapper/rhel-root ro rd_NO_PLYMOUTH crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root console=ttyS0,115200 net.ifnames=0 biosdevname=0


Additional info:

Comment 1 Martin Pitt 2019-01-07 09:56:24 UTC
I cannot test this on Fedora 29 due to bug 1663881. It also affects RHEL 7.6.

Comment 2 Bryn M. Reeves 2019-01-07 14:32:41 UTC
Boom uses a system of templates to generate boot entries. Currently you can add these additional options to the profile that you are using to have them automatically added to new boom entries - see the 'boom profile' edit subcommand.

In the next release we will add a feature to allow this template to be automatically initialised from /etc/default/grub for distributions that use it, and there are additional command line options (--add-options, --del-options) that further allow customisation on a per-entry basis. Since the values in /etc/default hard code aspects of the boot configuration this requires us to strip out and reformat the options and this was one of the features that did not make the release cutoff for GA.

Comment 3 Martin Pitt 2019-01-07 16:13:47 UTC
Maybe instead of directly reading /etc/default/grub, the standard profile could use "options $kernelopts $tuned_params" to mirror the default grub entries? That would be consistent and should structurally be relatively easy?

Comment 4 Bryn M. Reeves 2019-01-07 16:46:43 UTC
> the standard profile could use "options $kernelopts $tuned_params"

I'm not sure I understand you? Afaik the value of "$kernelopts" isn't available from anywhere else on a RHEL8 system. We can't simply drag in a fixed string and apply it everywhere since boom needs to modify values for each boot entry it creates (e.g. root= and the various rd.* parameters controlling boot devices). For reference, the current RHEL-style OsProfile templates look like:

  BOOM_OS_KERNEL_PATTERN="/vmlinuz-%{version}"
  BOOM_OS_INITRAMFS_PATTERN="/initramfs-%{version}.img"
  BOOM_OS_ROOT_OPTS_LVM2="rd.lvm.lv=%{lvm_root_lv}"
  BOOM_OS_ROOT_OPTS_BTRFS="rootflags=%{btrfs_subvolume}"
  BOOM_OS_OPTIONS="root=%{root_device} ro %{root_opts} rhgb quiet"

We can reverse-parse BOOM_OS_OPTIONS from GRUB_CMDLINE_LINUX which should address the main part of this bug, but it will not be available until boom-1.0.

For the tuned aspect I think there is already an ongoing discussion on how this integration should work in RHEL8 which we need to accommodate, but if tuned provides a mechanism to query for its desired command line options then that's fairly easy to hook up: a new template format key, %{tuned_opts} that expands to the value retrieved.

Comment 7 Bryn M. Reeves 2020-01-30 10:46:47 UTC
> # boom profile create --from-host --uname-pattern el8 --os-options "crashkernel=auto console=ttyS0 console=ttyS0,115200n8"
> Created profile with os_id f44fb52:
>   OS ID: "f44fb528ff8360ad67e2fe0274750b838da0bd6a",
>   Name: "Red Hat Enterprise Linux", Short name: "rhel",
>   Version: "8.1 (Ootpa)", Version ID: "8.1",
>   UTS release pattern: "el8",
>   Kernel pattern: "/vmlinuz-%{version}", Initramfs pattern: "/initramfs-%{version}.img",
>   Root options (LVM2): "rd.lvm.lv=%{lvm_root_lv}",
>   Root options (BTRFS): "rootflags=%{btrfs_subvolume}",
>   Options: "crashkernel=auto console=ttyS0 console=ttyS0,115200n8",
>   Title: "%{os_name} %{os_version_id} (%{version})"

The problem here is that you've removed all the root device options from the OsProfile options template - these are required to boot the system and this is why the resulting entries are un-bootable. At a minimum you *must* have a "root=%{root_device}" in order to tell the kernel what device to use for the root file system. Without that there's no way for us to identify the device. This is also the reason for the "boom list" error:

# boom list
ERROR - Command failed: Value for report_str() must be a string type.

Because the generated BootEntry does not contain a valie "root=/dev/..." string, we cannot parse the root device and this causes an error in the reporting code (since the root_device of the corresponding BootParams object is set to None). You can see the full error by passing --debug=all to the boom command:

# boom list --debug=all
Traceback (most recent call last):
  File "/usr/bin/boom", line 7, in <module>
    r = main(sys.argv)
  File "/usr/lib/python3.6/site-packages/boom/command.py", line 2951, in main
    status = command[1](cmd_args, select, opts, identifier)
  File "/usr/lib/python3.6/site-packages/boom/command.py", line 1965, in _list_cmd
    print_entries)
  File "/usr/lib/python3.6/site-packages/boom/command.py", line 1944, in _generic_list_cmd
    expand=cmd_args.expand_variables)
  File "/usr/lib/python3.6/site-packages/boom/command.py", line 821, in print_entries
    opts=opts, sort_keys=sort_keys)
  File "/usr/lib/python3.6/site-packages/boom/command.py", line 440, in _do_print_type
    br.report_object(obj)
  File "/usr/lib/python3.6/site-packages/boom/report.py", line 959, in report_object
    fields[fp.field_num].report_fn(field, data)
  File "/usr/lib/python3.6/site-packages/boom/command.py", line 304, in <lambda>
    REP_STR, lambda f, d: f.report_str(d.root_device)),
  File "/usr/lib/python3.6/site-packages/boom/report.py", line 279, in report_str
    raise TypeError("Value for report_str() must be a string type.")
TypeError: Value for report_str() must be a string type.


If you want to clean this up without having to manually delete files from /boot/loader/entries you can list the "bad" entry like this:

# boom list -obootid
BootID 
1366e94

By using "-obootid" (to display only the  boot_id for each entry) we avoid accessing the bad root_device value in "boom list".

Then once you have the boot_id you can remove is as usual:

# boom delete 1366e94
Deleted 1 entry

We should definitely make this more robust in a future update: it should not be possible (using the boom command line tool) to create an OsProfile with an options template that does not include at least root=.

Comment 8 Bryn M. Reeves 2020-01-30 11:01:58 UTC
Here are the steps to modify the built-in RHEL7 OsProfile using lvm2-python-boom-0.9-18.el7_7.2.noarch:

1. Find the correct RHEL7 OsProfile

# boom profile list -I 7
OsID    Name                            OsVersion 
c0b921e Red Hat Enterprise Linux Server 7 (Server)

# boom profile show c0b921e
OS Profile (os_id=c0b921e)
  OS ID: "c0b921ea84dbc5259477cbff7a15b000dd222671",
  Name: "Red Hat Enterprise Linux Server", Short name: "rhel",
  Version: "7 (Server)", Version ID: "7",
  UTS release pattern: "el7",
  Kernel pattern: "/vmlinuz-%{version}", Initramfs pattern: "/initramfs-%{version}.img",
  Root options (LVM2): "rd.lvm.lv=%{lvm_root_lv}",
  Root options (BTRFS): "rootflags=%{btrfs_subvolume}",
  Options: "root=%{root_device} ro %{root_opts} rhgb quiet"

In lvm2-python-boom in RHEL7 we also include an OsProfile for RHEL7.2; this can cause confusion when using custom profiles (unless you include the os_id when creating entries), so I would recommend the user deletes the extra RHEL7 copy:

# boom profile delete 3fc389b
Deleted 1 profile

We will remove this duplicate OsProfile in the next RHEL7 update to the package.

2. Edit profile options by *appending* additional kernel command line parameters (you can also drop "rhgb quiet" if not required):

# boom profile edit c0b921e --os-options "root=%{root_device} ro %{root_opts} crashkernel=auto console=ttyS0 console=ttyS0,115200n8" 
Edited profile:
  OS ID: "c0b921ea84dbc5259477cbff7a15b000dd222671",
  Name: "Red Hat Enterprise Linux Server", Short name: "rhel",
  Version: "7 (Server)", Version ID: "7",
  UTS release pattern: "el7",
  Kernel pattern: "/vmlinuz-%{version}", Initramfs pattern: "/initramfs-%{version}.img",
  Root options (LVM2): "rd.lvm.lv=%{lvm_root_lv}",
  Root options (BTRFS): "rootflags=%{btrfs_subvolume}",
  Options: "root=%{root_device} ro %{root_opts} crashkernel=auto console=ttyS0 console=ttyS0,115200n8"

3. Create boot entries using the newly edited profile

# boom create --title "RHEL7 Snapshot" --rootlv rhel/root-snap
Created entry with boot_id 53852d1:
  title RHEL7 Snapshot
  machine-id ee40168508f54f53be681091bfbd2609
  version 3.10.0-1062.el7.x86_64
  linux /vmlinuz-3.10.0-1062.el7.x86_64
  initrd /initramfs-3.10.0-1062.el7.x86_64.img
  options root=/dev/rhel/root-snap ro rd.lvm.lv=rhel/root-snap crashkernel=auto console=ttyS0 console=ttyS0,115200n8

# boom list
BootID  Version                  Name                            RootDevice    
53852d1 3.10.0-1062.el7.x86_64   Red Hat Enterprise Linux Server /dev/rhel/root-snap

Comment 9 Bryn M. Reeves 2020-01-30 15:03:12 UTC
I've added a check upstream to enforce the inclusion of the "root=" device option in both new and edited OsProfile options templates - this should prevent the situation in comment #5 and comment #7 from happening again:

$ boom profile create --from-host --uname-pattern fc30 --os-options "ro %{root_opts} rhgb quiet"
OsProfile.options must include root= device option

$ boom profile edit c0b921e --os-options "ro %{root_opts} rhgb quiet"
OsProfile.options must include root= device option

Comment 11 Rupesh Girase 2021-01-01 05:20:41 UTC
(In reply to Bryn M. Reeves from comment #9)
> I've added a check upstream to enforce the inclusion of the "root=" device
> option in both new and edited OsProfile options templates - this should
> prevent the situation in comment #5 and comment #7 from happening again:
> 
> $ boom profile create --from-host --uname-pattern fc30 --os-options "ro
> %{root_opts} rhgb quiet"
> OsProfile.options must include root= device option
> 
> $ boom profile edit c0b921e --os-options "ro %{root_opts} rhgb quiet"
> OsProfile.options must include root= device option

Hello Bryn,

Happy new year!

Is there any response from upstream? Is it integrated with rhel tree?


Thanks,
Rupesh

Comment 12 Bryn M. Reeves 2021-01-29 16:09:49 UTC
It's not really clear what we should do with this - although the configuration variable is still present in /etc/default/grub on all RHEL8 systems it does not appear to be used any more unless the user manually runs grub2-mkconfig.

Both inspecting the kernel installation hook scripts, and testing (by inserting a string into GRUB_CMDLINE_LINUX, performing a kernel update, and then checking for the presence of the extra string in /boot) seem to suggest that this variable is no longer honoured. This appears to be the case because grub2-mkconfig is no longer run on each kernel update - it's only this script that applies the setting from GRUB_CMDLINE_LINUX. The BLS boot configuration files generated by the kernel install hooks just use the saved kernel command line in the Grub2 $kernelopts environment variable.

It's very easy for users to copy the GRUB_CMDLINE_LINUX value into a Boom OsProfile if they want to use it, but I'm not certain that it is useful to automate this any more today since the variable seems to be going out of use.

Comment 16 Rupesh Girase 2021-12-09 05:49:14 UTC
Hello Bryn,

I understand your concern as GRUB_CMDLINE_LINUX may not remain relevant in near future.
Is there any option/parameter in Boom OsProfile to make aware about the device changes?


Thanks,
Rupesh

Comment 18 Bryn M. Reeves 2022-03-14 16:39:01 UTC
There are really two distinct problems here, and it has been unclear what the best approach is for boom:

  1. Administrators want a simple way to make boom boot entries reflect the same configuration as system boot entries (for e.g. as suggested in this bug, by initialising the OsProfile template from the GRUB_CMDLINE_LINUX variable in /etc/default/grub)

  2. The management of the kernel command line defaults has changed substantially in RHEL8 with the move to BLS. The values defined in /etc/default/grub are no longer used for new system boot loader entries either: rather, the $kernelopts grub2 environment variable is initialised with the values that were previously stored in /etc/default/grub, and all new system boot entries reference this boot loader variable instead. In RHEL8 GRUB_CMDLINE_LINUX is only honoured for boot entries created on kernel installation in the case that the administrator disables BLS and reverts to running the grub2-mkconfig script on updates to create new boot entries.

Clearly, even with the change to kernel command line configuration brought about by BLS (1) is a valid request - it's completely reasonable to expect a simple (and automatic) method to set up boom to boot the system with an equivalent set of command line options as the existing system entries. This is roughly equivalent to the functionality provided for setting up an OsProfile using values from the running system ("boom profile create --from-host").

I think we can address this by using either $kernelopts (if it is defined), or GRUB_CMDLINE_LINUX to initialise the options template in new OsProfiles when --from-host is used. That way we will automatically pick up the system configuration for systems that are using the default RHEL8 boot mechanisms, while still keeping compatibility to automatically set the options template on systems where BLS has been disabled. This will need some review and testing to ensure there are no unexpected side effects but this seems like the most straightforward approach given the changes in how this is configured between RHEL7/8.

At the same time we should document that the Grub2 environment is the default source of the configuration on RHEL8 - GRUB_CMDLINE_LINUX is no longer the canonical place to make customisations (even though this was documented for previous RHEL versions). We should not be continuing to direct users to modify this variable on RHEL8 or later systems unless they have also disabled BLS and are aware of the implications and potential for confusion.

Comment 23 Bryn M. Reeves 2023-04-11 16:17:33 UTC
I've filed an issue for this problem:

  https://github.com/snapshotmanager/boom/issues/14

I'm not certain what the best approach is right now (although we only need to consider Fedora/RHEL8/9 at this stage) but we will try to include this functionality in the next upstream boom release.

Comment 24 Bryn M. Reeves 2023-04-24 15:57:59 UTC
This has now been implemented and will be in the next boom release (1.5):

commit 66c2cc0dbc250fc8abb8d0a999a98da3d9e9b076
Author: Bryn M. Reeves <bmr>
Date:   Mon Apr 17 16:32:40 2023 +0100

    Implement automatic --os-options templates from /proc/cmdline
    
    If no --os-options is given when creating a profile with --from-host
    attempt to automatically generate an options template from the running
    system's /proc/cmdline.
    
    Resolves: #14
    
    Signed-off-by: Bryn M. Reeves <bmr>

Comment 25 Milan P. Gandhi 2023-04-25 10:10:23 UTC
Thanks for the updates, Bryn!

Comment 26 Bryn M. Reeves 2023-04-25 13:31:43 UTC
As an example this is the profile that would be created for the command line in comment #0:

# boom profile create --from-host
Created profile with os_id 7da8886:
  OS ID: "7da8886a4cb0a64a3429faaecc3cca6273095fbd",
  Name: "Fedora Linux", Short name: "fedora",
  Version: "37 (Server Edition)", Version ID: "37",
  Kernel pattern: "/vmlinuz-%{version}", Initramfs pattern: "/initramfs-%{version}.img",
  Root options (LVM2): "rd.lvm.lv=%{lvm_root_lv}",
  Root options (BTRFS): "rootflags=%{btrfs_subvolume}",
  Options: "root=%{root_device} ro %{root_opts} rd_NO_PLYMOUTH crashkernel=auto resume=/dev/mapper/rhel-swap console=ttyS0,115200 net.ifnames=0 biosdevname=0",
  Title: "%{os_name} %{os_version_id} (%{version})",
  Optional keys: "grub_users grub_arg grub_class id", UTS release pattern: "fc37"


Note You need to log in before you can comment on or make changes to this bug.