2107621 – [RHEL-9] Cannot select TBoot kernel entry as default boot entry

This bug has been migrated to another issue tracking site. It has been closed here and may no longer be being monitored.

If you would like to get updates for this issue, or to participate in it, you may do so at Red Hat Issue Tracker .

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2107621 - [RHEL-9] Cannot select TBoot kernel entry as default boot entry

Summary: [RHEL-9] Cannot select TBoot kernel entry as default boot entry

Keywords:
Status:	CLOSED MIGRATED
Alias:	None
Product:	Red Hat Enterprise Linux 9
Classification:	Red Hat
Component:	tboot
Sub Component:
Version:	9.0
Hardware:	x86_64
OS:	Linux
Priority:	medium
Severity:	medium
Target Milestone:	rc
Target Release:	9.2
Assignee:	Tony Camuso
QA Contact:	Release Test Team
Docs Contact:
URL:
Whiteboard:
Depends On:	2107609 2107618
Blocks:
TreeView+	depends on / blocked

Reported:	2022-07-15 14:28 UTC by Renaud Métrich
Modified:	2023-09-08 17:30 UTC (History)
CC List:	5 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2023-09-03 01:43:57 UTC
Type:	Bug
Target Upstream Version:
Embargoed:
Dependent Products:
Flags:	pm-rhel: mirror+

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	RHEL-2080	0	None	Migrated	None	2023-09-03 01:43:54 UTC
Red Hat Issue Tracker	RHELPLAN-127866	0	None	None	None	2022-07-15 14:33:44 UTC

Description Renaud Métrich 2022-07-15 14:28:43 UTC

Description of problem:

Currently it's not possible to select a TBOOT entry by default, this has to be done manually.
The reason for this is TBOOT's /etc/grub.d/20_linux_tboot script creates a "submenu", which prevents considering "saved_entry" Grub variable.

I think TBOOT should rely on the setting of GRUB_DISABLE_SUBMENU to decide whether to create a sub-menu or not.
On RHEL, by default we have GRUB_DISABLE_SUBMENU=true in /etc/default/grub so we don't expect sub-menus.
Once this is implemented, the admin can then select the TBOOT kernel through using grub2-editenv, e.g.:
~~~
LATEST_TBOOT_ENTRY="$(grep -w ^menuentry /boot/grub2/grub.cfg | grep -m 1 "with tboot" | sed -e "s/menuentry '\(.*\)' --class .*/\1/")"
/usr/bin/grub2-editenv - set saved_entry="$LATEST_TBOOT_ENTRY"
~~~

which will always select the highest kernel TBOOT entry.

Version-Release number of selected component (if applicable):

tboot-1.10.2-6.el9.x86_64

How reproducible:

Always

Steps to Reproduce:
1. Select the default entry through editing grubenv file:

LATEST_TBOOT_ENTRY="$(grep -w ^menuentry /boot/grub2/grub.cfg | grep -m 1 "with tboot" | sed -e "s/menuentry '\(.*\)' --class .*/\1/")"
/usr/bin/grub2-editenv - set saved_entry="$LATEST_TBOOT_ENTRY"

2. Reboot

Actual results:

First kernel (non-TBOOT) selected because of the sub-menu.

Expected results:

TBOOT kernel selected

Additional info:

More generally, it would be nice if TBOOT could be implemented as a BLS snippet, which would ease all this handling. See also #2107618.

Comment 1 Tony Camuso 2022-08-26 17:09:25 UTC

The correct way to set the default boot is with grub2-set-default.

Please try the following:

# tbootentry=$(grep submenu /boot/grub2/grub.cfg | cut -d'"' -f2)

# echo $tbootentry  # <- make sure you got a valid 'tboot-<version>-<revision>'

# grub2-set-default "$tbootentry"

Comment 2 Jerone Young 2022-08-26 18:01:00 UTC

@Tony
   How is this non documented way the correct way?  How will this work on kernel upgrades?

   What you suggest is more of a workaround then it is a solution in the product.

Comment 3 Tony Camuso 2022-08-26 18:55:13 UTC

(In reply to Jerone Young from comment #2)
> @Tony
>    How is this non documented way the correct way?  How will this work on
> kernel upgrades?
> 
>    What you suggest is more of a workaround then it is a solution in the
> product.

Then do this.

~~~
# LATEST_TBOOT_ENTRY="$(grep -w ^submenu /boot/grub2/grub.cfg | cut -d'"' -f2)

# /usr/bin/grub2-editenv - set saved_entry="$LATEST_TBOOT_ENTRY"
~~~

The documentation should be updated accordingly.

Comment 4 Tony Camuso 2022-08-26 18:58:43 UTC

(In reply to Jerone Young from comment #2)
> @Tony
>    How is this non documented way the correct way?  How will this work on
> kernel upgrades?
> 
>    What you suggest is more of a workaround then it is a solution in the
> product.

Need another double quote at the end of that first line ...

# LATEST_TBOOT_ENTRY="$(grep -w ^submenu /boot/grub2/grub.cfg | cut -d'"' -f2)"

# /usr/bin/grub2-editenv - set saved_entry="$LATEST_TBOOT_ENTRY"

Please let me know if that works for you.

Comment 5 Jerone Young 2022-08-26 19:26:05 UTC

@Tony
  This would get reset if a new kernel update is installed, would it not?
  
  This is the big problem.

  With that said I did test the commands:

yum install tboot
grub2-mkconfig --output=/boot/grub2/grub.cfg

LATEST_TBOOT_ENTRY="$(grep -w ^submenu /boot/grub2/grub.cfg | cut -d'"' -f2)"
#echo $LATEST_TBOOT_ENTRY
# tboot 1.10.2
# GOOD!!

/usr/bin/grub2-editenv - set saved_entry="$LATEST_TBOOT_ENTRY"

# This did work! It defaults to the latest tboot entry.


So the big question is what happens on a kernel update. If this stays it can work. But if a kernel update resets this there is a problem.

Comment 6 Jerone Young 2022-08-26 19:43:55 UTC

This fix does not address the key problem. It's a work around but not a permanent fix.

I just tried the following and here is what happened:

#install additional older kernel
yum downgrade kernel-5.14.0-70.17.1.el9_0 kernel-core-5.14.0-70.17.1.el9_0 kernel-modules-5.14.0-70.17.1.el9_0
#remove newest kernel
rpm -e kernel-5.14.0-70.22.1.el9_0 kernel-core-5.14.0-70.22.1.el9_0 kernel-modules-5.14.0-70.22.1.el9_0

Reboot
#tboot is NOT default entry in grub this is BAD!
#It is the older kernel and not the tboot entry for it

#install reinstall newest kernel
yum update -y
reboot

#tboot is NOT default entry in grub this is BAD!
#It is the newest kerneland not the tboot entry for it

Comment 7 Tony Camuso 2022-08-26 23:29:25 UTC

(In reply to Jerone Young from comment #6)
> This fix does not address the key problem. It's a work around but not a
> permanent fix.
> 
> I just tried the following and here is what happened:
> 
> #install additional older kernel
> yum downgrade kernel-5.14.0-70.17.1.el9_0 kernel-core-5.14.0-70.17.1.el9_0
> kernel-modules-5.14.0-70.17.1.el9_0
> #remove newest kernel
> rpm -e kernel-5.14.0-70.22.1.el9_0 kernel-core-5.14.0-70.22.1.el9_0
> kernel-modules-5.14.0-70.22.1.el9_0
> 
> Reboot
> #tboot is NOT default entry in grub this is BAD!
> #It is the older kernel and not the tboot entry for it
> 
> #install reinstall newest kernel
> yum update -y
> reboot
> 
> #tboot is NOT default entry in grub this is BAD!
> #It is the newest kerneland not the tboot entry for it

There is nothing we can do about that, because grubby-bls does not support multiboot, which is needed for tboot. grubby-bls does not invoke grub2-mkconfig, nor does it preserve the grubenv with tboot as the default, so the newly installed kernel will be the default boot image.

Until this is fixed in the bootloader and grubby-bls, you must run grub2-mkconfig and grub2-editenv again to get tboot as the default boot image.

See the following bugzillas against bootloader.
https://bugzilla.redhat.com/show_bug.cgi?id=2107618
https://bugzilla.redhat.com/show_bug.cgi?id=2107609

There is a z-stream tboot that is being introduced shortly that will help minimize the pain. 
When you install a new kernel, just run the following command with the new tboot z-stream.

You can get the z-stream tboot rpm for testing here ...
https://people.redhat.com/tcamuso/tboot/el9/tboot-1.10.5-2.el9.x86_64.rpm

# dnf localinstall -y tboot-1.10.5-2.el9.x86_64.rpm

### install your new kernel ###

# dnf reinstall -y tboot-1.10.5-2.el9.x86_64.rpm

Let me know how that works.

Comment 8 Jerone Young 2022-08-27 15:53:29 UTC

@Tony
   This does not work. Remember these are systems that are being administered in a production environment. These solutions will break immediately when yum update gets run and a new kernel comes in.

   I have had a better work around that I have a customer using now that is much better then what you have proposed. Basically removing /etc/grub.d/10_linux . This basically makes sure that tboot is the only entry in grub and works.

  Though we are looking for a better permanent solution then what I have. Though what you have offered so far will not work.

Comment 9 Tony Camuso 2022-08-29 11:31:14 UTC

(In reply to Jerone Young from comment #8)
> @Tony
>    This does not work. Remember these are systems that are being
> administered in a production environment. These solutions will break
> immediately when yum update gets run and a new kernel comes in.
> 
>    I have had a better work around that I have a customer using now that is
> much better then what you have proposed. Basically removing
> /etc/grub.d/10_linux . This basically makes sure that tboot is the only
> entry in grub and works.
> 
>   Though we are looking for a better permanent solution then what I have.
> Though what you have offered so far will not work.

There may be something I can do with the kernel rpm specfile.
I'll have a look at that. 

But the correct solution must come from the bootloader and grubby developers.
https://bugzilla.redhat.com/show_bug.cgi?id=2107618
https://bugzilla.redhat.com/show_bug.cgi?id=2107609

Comment 10 Tony Camuso 2022-08-29 14:26:56 UTC

(In reply to Jerone Young from comment #8)
> @Tony
>    This does not work. Remember these are systems that are being
> administered in a production environment. These solutions will break
> immediately when yum update gets run and a new kernel comes in.
> 
>    I have had a better work around that I have a customer using now that is
> much better then what you have proposed. Basically removing
> /etc/grub.d/10_linux . This basically makes sure that tboot is the only
> entry in grub and works.
> 
>   Though we are looking for a better permanent solution then what I have.
> Though what you have offered so far will not work.

I'm looking at the kernel install scripts, but until these two bugzillas can be 
addressed, you will have to continue using your workaround.

Comment 11 Tony Camuso 2022-08-31 19:10:06 UTC

(In reply to Jerone Young from comment #8)
> @Tony
>    This does not work. Remember these are systems that are being
> administered in a production environment. These solutions will break
> immediately when yum update gets run and a new kernel comes in.
> 
>    I have had a better work around that I have a customer using now that is
> much better then what you have proposed. Basically removing
> /etc/grub.d/10_linux . This basically makes sure that tboot is the only
> entry in grub and works.
> 
>   Though we are looking for a better permanent solution then what I have.
> Though what you have offered so far will not work.

Concerning your workaround, consider doing this instead, whenever you update your kernel.

Then, whenever you update your kernel, run the following commands ...

# grub2-set-default 'tboot 1.10.5'
# grub2-mkconfig -o /boot/grub2/grub.cfg

No need for deleting any of the /etc/grub.d files.

Comment 12 Tony Camuso 2022-08-31 19:12:59 UTC

(In reply to Tony Camuso from comment #11)
> 
> Concerning your workaround, consider doing this instead, whenever you update
> your kernel.
> 
> Then, whenever you update your kernel, run the following commands ...
> 
> # grub2-set-default 'tboot 1.10.5'
> # grub2-mkconfig -o /boot/grub2/grub.cfg
> 
> No need for deleting any of the /etc/grub.d files.

To make that more abstract, so it doesn't matter which tboot ...

# grub2-set-default "$(grep -w ^submenu /boot/grub2/grub.cfg | cut -d'"' -f2)"
... then ...
# grub2-mkconfig -o /boot/grub2/grub.cfg

Comment 17 Doug Ledford 2023-08-31 15:50:59 UTC

This bug is scheduled for migration to Jira.  When that happens, this bugzilla issue will be permanently closed with the status MIGRATED and all future interaction on this issue will need to happen in the Jira issue.  The new issue will be part of the RHEL project (a project for Jira only issues, which will sync once, then close the bugzilla issue and all future updates will happen in Jira), not part of the RHELPLAN project (which is part of the automated bugzilla->Jira mirroring and which allows ongoing updates to the bugzilla bug and syncs those updates over to Jira).

For making sure you have access to Jira in order to continue accessing this issue, follow one of the appropriate knowledge base articles:

KB0016394 - https://redhat.service-now.com/help?id=kb_article_view&sysparm_article=KB0016394
KB0016694 - https://redhat.service-now.com/help?id=kb_article_view&sysparm_article=KB0016694
KB0016774 - https://redhat.service-now.com/help?id=kb_article_view&sysparm_article=KB0016774

For general issues with Jira, open a ticket with rh-issues

Comment 18 RHEL Program Management 2023-09-03 01:43:31 UTC

Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

Comment 19 RHEL Program Management 2023-09-03 01:43:57 UTC

This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues.

Comment 20 Doug Ledford 2023-09-08 17:30:47 UTC

Additional information on creating a Jira account to access the migrated issue can be found here:

https://access.redhat.com/articles/7032570

Note You need to log in before you can comment on or make changes to this bug.