Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1777874

Summary: RHEL-8.2 Tuned causes duplication of kernel comand line flags.
Product: Red Hat Enterprise Linux 8 Reporter: Joe Mario <jmario>
Component: tunedAssignee: Jaroslav Škarvada <jskarvad>
Status: CLOSED ERRATA QA Contact: Robin Hack <rhack>
Severity: medium Docs Contact: Prerana Sharma <presharm>
Priority: high    
Version: 8.2CC: alink, aratnapa, atheurer, ggrimaux, jeder, jskarvad, lcapitulino, mivollme, presharm, psklenar, pveiga, rcheerla, rhack, ttracy
Target Milestone: rcKeywords: Patch, TestCaseProvided, Triaged, Upstream, ZStream
Target Release: 8.0Flags: pm-rhel: mirror+
Hardware: All   
OS: Linux   
URL: https://github.com/redhat-performance/tuned/pull/311
Whiteboard:
Fixed In Version: tuned-2.15.0-0.1.rc1.el8 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 1867736 1918995 1918998 (view as bug list) Environment:
Last Closed: 2021-05-18 15:05:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1867736, 1883636, 1918995, 1918998    

Description Joe Mario 2019-11-28 14:44:44 UTC
Description of problem:

Under certain situations, tuned will cause duplicate flags to be added to the kernel command line.  This has been masked till now, but a grub2 bugfix due to go into RHEL-8.2 [BZ 1669252], will cause it to be visible.

Version-Release number of selected component (if applicable):
The RHEL-8.* versions of tuned.

Steps to Reproduce:

1) Set your tuned profile to a profile which modifies the kernel boot line.
   These include network-latency, cpu-partitioning, and realtime.

2) Look at the /boot/grub2/grubenv file.  It will have a row specifying "tuned_params=xyz" and it will not have a "$tuned_params" flag added to the kernelopts variable.  This is good.  No problem so far.

3) Now run grub2-mkconfig -o /boot/grub2/grub.cfg

3) Look at the /boot/grub2/grubenv file. It will now contain both a $tuned_params variable at the end of the kernelopts setting, and a separate entry for tuned_params=xyz.

See the example below:

  # cat /boot/grub2/grubenv
  # GRUB Environment Block
  saved_entry=5ed1a02487f742ce9e5abd9c8d3918a1-4.18.0-153.el8.x86_64 \
   kernelopts=root=/dev/mapper/rhel8.2_buzz-root ro \
   crashkernel=autoresume=/dev/mapper/rhel8.2_buzz-swap rd.lvm.lv=\
   rhel8.2_buzz/rootrd.lvm.lv=rhel8.2_buzz/swap ipv6_disable=1 $tuned_params
  boot_success=0
  tuned_params=skew_tick=1
  tuned_initrd=

The flags that tuned adds to the kernel boot line will be duplicated.
Here's an example from another system after I rebooted it:

# cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-147.el8.x86_64 root=/dev/mapper/rhel_perf130-root ro crashkernel=auto resume=/dev/mapper/rhel_perf130-swap rd.lvm.lv=rhel_perf130/root rd.lvm.lv=rhel_perf130/swap rhgb quiet crashkernel=auto skew_tick=1 skew_tick=1

Comment 1 Jaroslav Škarvada 2019-11-28 15:00:42 UTC
I think the following is happening. Tuned sets GRUB_CMDLINE_LINUX_DEFAULT variable in the /etc/default/grub. It seems its content can be then written to the kernelopts by /etc/grub.d/10_linux script which is called by grub2-mkconfig.

Comment 2 Jaroslav Škarvada 2019-11-28 15:29:07 UTC
I think this bug was introduced the following way:

- Tuned started to use /etc/default/grub config. In this time nothing like BLS existed and grub2-mkconfig was usually used to transform /etc/default/grub (with the help of /etc/grub.d/* scripts) to /boot/*/grub.cfg
- BLS was introduced
- Tuned started shipping /usr/lib/kernel/install.d/92-tuned.install script for BLS and BLS support was added to grub2-mkconfig and it started saving GRUB_CMDLINE_LINUX_DEFAULT from the /etc/default/grub as $kernelopts grubenv

At the moment I think about two possible solutions of this problem in Tuned:
a) If there is BLS enabled stop adding $tuned_params to GRUB_CMDLINE_LINUX_DEFAULT, this could have some side-effects which are currently unknown to me.
b) Stop adding $tuned_params to 'options' in /boot/loader/entries and start adding it to $kernelopts grubenv - for me this is not so clear solution as the current state (and it also relies on recurive expansion of the grubenv variables, which seems to be supported at the moment by grub2;, but doesn't seem to be standardized anywhere), but it should work.

Comment 3 Jaroslav Škarvada 2019-11-28 15:36:52 UTC
Joe, I am just curious, what's your usecase for 'grub2-mkconfig -o /boot/grub2/grub.cfg' with BLS? With BLS there is probably no need to regenerate the grub2.cfg. Every change you save to the grubenv (e.g. kernelopts) will be taken automatically upon next reboot by grub.

Comment 4 Joe Mario 2019-11-28 16:37:51 UTC
Hi Jaroslav:
The use case is when we're adding kernel command line flags, we add them to the /etc/default/grub file and then run "grub2-mkconfig > /boot/grub2/grub.cfg"

Joe

Comment 5 Jaroslav Škarvada 2019-11-29 10:26:17 UTC
(In reply to Joe Mario from comment #4)
> Hi Jaroslav:
> The use case is when we're adding kernel command line flags, we add them to
> the /etc/default/grub file and then run "grub2-mkconfig >
> /boot/grub2/grub.cfg"
> 
> Joe

Thanks for info. I think with BLS (RHEL-8) if anything you need is just adding kernel options, you could add them to kernelopts in grubenv and no configuration file needs to be regenerated with grub2-mkconfig. I just wanted to point this out. Regarding the duplication when grub2-mkconfig is used it's bug and we will fix it (somehow).

Comment 6 Joe Mario 2019-11-29 14:46:05 UTC
Thanks Jaroslav:

There was a time in the RHEL-8 development when editing the grubenv file directly would cause grub errors forcing a rebuild of grubenv.  I never tracked down if that was a bug or feature.  Thgough shouldn't using grub2-editenv always work?

But back to editing /etc/default/grub directly, it is a common way to modify the kernel command line parameters.  It's even documented as the way to do it in the RHEL-8 documentation.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/managing_monitoring_and_updating_the_kernel/configuring-kernel-command-line-parameters_managing-monitoring-and-updating-the-kernel#changing-kernel-command-line-parameters-for-all-boot-entries_setting-kernel-command-line-parameters

I can see from comment #2 that it will be a challenge to address this.  Fortunately it's not a correctness problem.
Thanks for doing whatever you can.

Joe

Comment 9 Priscila 2020-08-12 18:33:13 UTC
Seems there is nothing from kernelargs to do.
Once this is fixed the OSP side will be too.

Closing the dup, keeping this.

Comment 10 alink 2020-08-13 15:08:43 UTC
*** Bug 1867831 has been marked as a duplicate of this bug. ***

Comment 26 Jaroslav Škarvada 2020-11-29 20:41:26 UTC
Upstream PR:
https://github.com/redhat-performance/tuned/pull/311

In order to recover from the kernel cmdline dupes previously created by grub2-mkconfig, grub2-mkconfig has to be run after installation of the patched Tuned (we do not run it automatically, because it could break setup for users not using grub2-mkconfig).

Comment 34 Luiz Capitulino 2021-01-21 02:49:45 UTC
Jaroslav,

What's the status of the z-stream request?

Comment 37 Jaroslav Škarvada 2021-01-21 21:08:11 UTC
(In reply to Luiz Capitulino from comment #34)
> Jaroslav,
> 
> What's the status of the z-stream request?

LGTM, requesting clones.

Comment 50 errata-xmlrpc 2021-05-18 15:05:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (tuned bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1671