Hi Jaroslav:
Do you have any update or workaround for this?

Here are some simple steps to reproduce it:

I installed a fresh RHEL-8 install, and then set tuned to the cpu-partitioning profile.  The /proc/cmdline output looked good.
Then I added spectre_v2=off to the /etc/default/grub file, and regenerated grub.cfg with "grub2-mkconfig -o /boot/grub2/grub.cfg".
Then after the reboot, the /proc/cmdline file shows the "skew_tick=1" flag got truncated to "=1".

The output is below.
Any update?
Thanks,
Joe


# cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-65.el8.x86_64 root=/dev/mapper/rhel_perf132-root ro crashkernel=auto resume=/dev/mapper/rhel_perf132-swap rd.lvm.lv=rhel_perf132/root rd.lvm.lv=rhel_perf132/swap skew_tick=1 nohz=on nohz_full=2-55 rcu_nocbs=2-55 tuned.non_isolcpus=00000003 intel_pstate=disable nosoftlockup

# vi /etc/default/grub 
# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found Red Hat Enterprise Linux Server 7.6 (Maipo) on /dev/mapper/rhel73_perf132-root
rFound Red Hat Enterprise Linux Server 7.4 (Maipo) on /dev/mapper/rhel74_perf132-root
eboodone
# reboot

$ ssh root.lab.eng.bos.redhat.com
root.lab.eng.bos.redhat.com's password: 

Last login: Tue Feb 12 13:07:46 2019 from 10.18.17.131
# cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-65.el8.x86_64 root=/dev/mapper/rhel_perf132-root ro crashkernel=auto resume=/dev/mapper/rhel_perf132-swap rd.lvm.lv=rhel_perf132/root rd.lvm.lv=rhel_perf132/swap spectre_v2=off =1 nohz=on nohz_full=2-55 rcu_nocbs=2-55 tuned.non_isolcpus=00000003 intel_pstate=disable nosoftlockup

Hi Joe, thanks for the report. I can reproduce it without any problems. You should be able to workaround it by disabling "BLS": set "GRUB_ENABLE_BLSCFG"  to "false" in /etc/default/grub and re-run grub2-mkconfig -o /boot/grub2/grub.cfg.

Do you guys think this also affects the real-time profiles?

I think so, yes. Although, note that you'd need to manually edit /etc/default/grub.

The way I see it now is that there's a problem both in Tuned's bootloader plugin and in GRUB2. I'll try to provide an update soon.

OK, so I'm adjusting the BZ summary to reflect that.

Thanks for changing the title.  Given what we know, the new wording is more appropriate.

I recall also seeing this when I used the network-latency tuned profile.  That profile also sets skew_tick=1.

Joe

(In reply to Ondřej Lysoněk from comment #4)
> I think so, yes. Although, note that you'd need to manually edit
> /etc/default/grub.

Correction: there's no need to edit the file. All you need to do is run grub2-mkconfig.

(In reply to Ondřej Lysoněk from comment #4)
> The way I see it now is that there's a problem both in Tuned's bootloader
> plugin and in GRUB2. I'll try to provide an update soon.

The problem in Tuned I was talking about is the fact that the $tuned_params grubenv variable can get appended both to the value of the $kernelopts grubenv variable, and also to the 'options' parameter in the BLS entries. The result of that would be that $tuned_params gets appended to the kernel commandline twice.

However, after a discussion with Jaroslav, we think that this should not cause problems and the kernel should be able to deal with it.

So the remaining issue is in GRUB2. When a grubenv variable gets used both in $kernelopts in grubenv, and in 'options' in the BLS entries, the value of that variable gets corrupted. You can find a reproducer that does not involve Tuned below. I can reproduce the problem on a fresh non-EFI RHEL-8 installation.

Version-Release number of selected component (if applicable):
grub2-common-2.02-66.el8.noarch

Steps to reproduce:
grub2-editenv /boot/grub2/grubenv set 'myvar=skew_tick=1'

# Make '$myvar' appear in the value of kernelopts in /boot/grub2/grubenv
echo 'GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT:+$GRUB_CMDLINE_LINUX_DEFAULT }\$myvar"' >> /etc/default/grub
grub2-mkconfig -o /etc/grub2.cfg

# Make '$myvar' appear in 'options' in the BLS entry
sed -e 's/^\(options.*\)$/\1 $myvar/' -i /boot/loader/entries/591d7280467543a2bd433dccc22b02ef-4.18.0-64.el8.x86_64.conf

reboot

cat /proc/cmdline

Actual results:
The kernel command line is:
BOOT_IMAGE=(hd0,msdos1)/boot/vmlinuz-4.18.0-64.el8.x86_64 root=UUID=40bd3e0a-fde8-4ba5-b88f-81545505d1d4 ro console=tty0 console=ttyS0,115200 crashkernel=auto net.ifnames=0 rhgb quiet =1

Expected results:
The kernel command line should be:
BOOT_IMAGE=(hd0,msdos1)/boot/vmlinuz-4.18.0-64.el8.x86_64 root=UUID=40bd3e0a-fde8-4ba5-b88f-81545505d1d4 ro console=tty0 console=ttyS0,115200 crashkernel=auto net.ifnames=0 rhgb quiet skew_tick=1

Changing the priority and severity of this to a high.

The reason is because many of our tuned profiles add "skew_tick=1" to the kernel boot line.   Because of this bug, that variable turns into "=1".  The "skew_tick=1" flag is needed for performance, especially on larger systems.

Please update with any status for when this can be resolved.
Thank you.
Joe Mario

I was able to reproduce the bug by following the steps in Comment 8. The problem is that the variables are expanded to their values but a space character is not added at the end.

After fixing this, it works correctly:

$ grub2-editenv /boot/grub2/grubenv set 'myvar=skew_tick=1'
$ grub2-editenv list | grep myvar
myvar=skew_tick=1

$ echo 'GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT:+$GRUB_CMDLINE_LINUX_DEFAULT }\$myvar"' >> /etc/default/grub
$ grep myvar /etc/default/grub 
GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT:+$GRUB_CMDLINE_LINUX_DEFAULT }\$myvar"

$ grub2-mkconfig -o /etc/grub2-efi.cfg 
Generating grub configuration file ...
Adding boot menu entry for EFI firmware configuration
done

$ grub2-editenv list | grep myvar
myvar=skew_tick=1
kernelopts=root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet console=tty0 console=ttyS0,115200n $myvar

$ grep options /boot/loader/entries/$(cat /etc/machine-id)-$(uname -r).conf
options $kernelopts $tuned_params

$ sed -e 's/^\(options.*\)$/\1 $myvar/' -i /boot/loader/entries/$(cat /etc/machine-id)-$(uname -r).conf

$ grep options /boot/loader/entries/$(cat /etc/machine-id)-$(uname -r).conf
options $kernelopts $tuned_params $myvar

$ reboot

$ cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt2)/vmlinuz-4.18.0-151.el8.x86_64 root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet console=tty0 console=ttyS0,115200n skew_tick=1 skew_tick=1

Hi Javier:
Is there a trick to installing this scratch grub2?

Here are my installed grub* rpms:
  grub2-common.noarch                           1:2.02-76.el8                              @rhel8-AppStream
  grub2-pc.x86_64                               1:2.02-76.el8                              @rhel8-AppStream
  grub2-pc-modules.noarch                       1:2.02-76.el8                              @rhel8-AppStream
  grub2-tools.x86_64                            1:2.02-76.el8                              @rhel8-AppStream
  grub2-tools-efi.x86_64                        1:2.02-76.el8                              @rhel8-AppStream
  grub2-tools-extra.x86_64                      1:2.02-76.el8                              @rhel8-AppStream
  grub2-tools-minimal.x86_64                    1:2.02-76.el8                              @rhel8-AppStream
  grubby.x86_64                                 8.40-37.el8                                @rhel8-AppStream

Here's my repo file:
  # cat brew-task-repo-grub2-2.02-80.el8-scratch.repo
  [brew-task-repo-grub2-2.02-80.el8-scratch]
  name=Repo for Brew scratch build of grub2-2.02-80.el8
  enabled=1
  gpgcheck=0
  baseurl=http://brew-task-repos.usersys.redhat.com/repos/scratch/fmartine/grub2/2.02/80.el8/$basearch/
  module_hotfixes=1

And here are the results of trying to update it.
  # dnf update grub2* 
  Updating Subscription Management repositories.
  Unable to read consumer identity
  This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
  Last metadata expiration check: 0:04:37 ago on Tue 26 Nov 2019 05:33:50 PM EST.
  Error: 
   Problem 1: cannot install the best update candidate for package grub2-pc-1:2.02-76.el8.x86_64
    - nothing provides grub2-pc-modules = 1:2.02-80.el8 needed by grub2-pc-1:2.02-80.el8.x86_64
   Problem 2: package grub2-pc-modules-1:2.02-78.el8.noarch requires grub2-common = 1:2.02-78.el8, but none of the providers can be installed
    - cannot install both grub2-common-1:2.02-78.el8.noarch and grub2-common-1:2.02-80.el8.noarch
    - cannot install the best update candidate for package grub2-pc-modules-1:2.02-76.el8.noarch
    - cannot install the best update candidate for package grub2-common-1:2.02-76.el8.noarch
   Problem 3: problem with installed package grub2-pc-1:2.02-76.el8.x86_64
    - package grub2-pc-1:2.02-76.el8.x86_64 requires grub2-tools = 1:2.02-76.el8, but none of the providers can be installed
    - package grub2-pc-1:2.02-78.el8.x86_64 requires grub2-tools = 1:2.02-78.el8, but none of the providers can be installed
    - package grub2-tools-efi-1:2.02-80.el8.x86_64 obsoletes grub2-tools < 1:2.02-80.el8 provided by grub2-tools-1:2.02-76.el8.x86_64
    - package grub2-tools-efi-1:2.02-80.el8.x86_64 obsoletes grub2-tools < 1:2.02-80.el8 provided by grub2-tools-1:2.02-78.el8.x86_64
    - cannot install the best update candidate for package grub2-tools-efi-1:2.02-76.el8.x86_64
    - nothing provides grub2-pc-modules = 1:2.02-80.el8 needed by grub2-pc-1:2.02-80.el8.x86_64
   Problem 4: problem with installed package grub2-pc-modules-1:2.02-76.el8.noarch
    - package grub2-pc-modules-1:2.02-76.el8.noarch requires grub2-common = 1:2.02-76.el8, but none of the providers can be installed
    - package grub2-pc-modules-1:2.02-78.el8.noarch requires grub2-common = 1:2.02-78.el8, but none of the providers can be installed
    - cannot install both grub2-common-1:2.02-80.el8.noarch and grub2-common-1:2.02-76.el8.noarch
    - cannot install both grub2-common-1:2.02-78.el8.noarch and grub2-common-1:2.02-80.el8.noarch
    - package grub2-tools-extra-1:2.02-80.el8.x86_64 requires grub2-common = 1:2.02-80.el8, but none of the providers can be installed
    - cannot install the best update candidate for package grub2-tools-extra-1:2.02-76.el8.x86_64
  (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

Is there a trick I'm missing?
Thanks,
Joe

Even --allowerasing doesn't help:

  # dnf update grub2* --allowerasing
  Updating Subscription Management repositories.
  Unable to read consumer identity
  This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
  Last metadata expiration check: 0:06:30 ago on Tue 26 Nov 2019 05:33:50 PM EST.
  Error: 
   Problem: problem with installed package grub2-pc-1:2.02-76.el8.x86_64
    - cannot install the best update candidate for package grub2-pc-1:2.02-76.el8.x86_64
    - nothing provides grub2-pc-modules = 1:2.02-80.el8 needed by grub2-pc-1:2.02-80.el8.x86_64
  (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

Hello Joe,

(In reply to Joe Mario from comment #12)
> Hi Javier:
> Is there a trick to installing this scratch grub2?
>

Yes, the trick is the one I mentioned in Comment 11. You need to install both x86_64 and i686 repos.

[snip]

> 
> Here's my repo file:
>   # cat brew-task-repo-grub2-2.02-80.el8-scratch.repo
>   [brew-task-repo-grub2-2.02-80.el8-scratch]
>   name=Repo for Brew scratch build of grub2-2.02-80.el8
>   enabled=1
>   gpgcheck=0
>  
> baseurl=http://brew-task-repos.usersys.redhat.com/repos/scratch/fmartine/
> grub2/2.02/80.el8/$basearch/
>   module_hotfixes=1
>

$basearch will be for x86_64, you need another repo that has baseurl=http://brew-task-repos.usersys.redhat.com/repos/scratch/fmartine/grub2/2.02/80.el8/i686/

The grub2-pc-modules-2.02-80.el8.noarch.rpm package can be found there.

> And here are the results of trying to update it.
>   # dnf update grub2* 
>   Updating Subscription Management repositories.
>   Unable to read consumer identity
>   This system is not registered to Red Hat Subscription Management. You can
> use subscription-manager to register.
>   Last metadata expiration check: 0:04:37 ago on Tue 26 Nov 2019 05:33:50 PM
> EST.
>   Error: 
>    Problem 1: cannot install the best update candidate for package
> grub2-pc-1:2.02-76.el8.x86_64
>     - nothing provides grub2-pc-modules = 1:2.02-80.el8 needed by
> grub2-pc-1:2.02-80.el8.x86_64
>    Problem 2: package grub2-pc-modules-1:2.02-78.el8.noarch requires
> grub2-common = 1:2.02-78.el8, but none of the providers can be installed
>     - cannot install both grub2-common-1:2.02-78.el8.noarch and
> grub2-common-1:2.02-80.el8.noarch
>     - cannot install the best update candidate for package
> grub2-pc-modules-1:2.02-76.el8.noarch
>     - cannot install the best update candidate for package
> grub2-common-1:2.02-76.el8.noarch
>    Problem 3: problem with installed package grub2-pc-1:2.02-76.el8.x86_64
>     - package grub2-pc-1:2.02-76.el8.x86_64 requires grub2-tools =
> 1:2.02-76.el8, but none of the providers can be installed
>     - package grub2-pc-1:2.02-78.el8.x86_64 requires grub2-tools =
> 1:2.02-78.el8, but none of the providers can be installed
>     - package grub2-tools-efi-1:2.02-80.el8.x86_64 obsoletes grub2-tools <
> 1:2.02-80.el8 provided by grub2-tools-1:2.02-76.el8.x86_64
>     - package grub2-tools-efi-1:2.02-80.el8.x86_64 obsoletes grub2-tools <
> 1:2.02-80.el8 provided by grub2-tools-1:2.02-78.el8.x86_64
>     - cannot install the best update candidate for package
> grub2-tools-efi-1:2.02-76.el8.x86_64
>     - nothing provides grub2-pc-modules = 1:2.02-80.el8 needed by
> grub2-pc-1:2.02-80.el8.x86_64
>    Problem 4: problem with installed package
> grub2-pc-modules-1:2.02-76.el8.noarch
>     - package grub2-pc-modules-1:2.02-76.el8.noarch requires grub2-common =
> 1:2.02-76.el8, but none of the providers can be installed
>     - package grub2-pc-modules-1:2.02-78.el8.noarch requires grub2-common =
> 1:2.02-78.el8, but none of the providers can be installed
>     - cannot install both grub2-common-1:2.02-80.el8.noarch and
> grub2-common-1:2.02-76.el8.noarch
>     - cannot install both grub2-common-1:2.02-78.el8.noarch and
> grub2-common-1:2.02-80.el8.noarch
>     - package grub2-tools-extra-1:2.02-80.el8.x86_64 requires grub2-common =
> 1:2.02-80.el8, but none of the providers can be installed
>     - cannot install the best update candidate for package
> grub2-tools-extra-1:2.02-76.el8.x86_64
>   (try to add '--allowerasing' to command line to replace conflicting
> packages or '--skip-broken' to skip uninstallable packages or '--nobest' to
> use not only best candidate packages)
> 
> Is there a trick I'm missing?
> Thanks,
> Joe

Hi Javier:
Thank you.  With your guidance above, I did get the new rpms installed.

# cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-147.el8.x86_64 root=/dev/mapper/rhel_perf130-root ro crashkernel=auto resume=/dev/mapper/rhel_perf130-swap rd.lvm.lv=rhel_perf130/root rd.lvm.lv=rhel_perf130/swap rhgb quiet crashkernel=auto =1 nohz=on nohz_full=1-27,29-55 rcu_nocbs=1-27,29-55 tuned.non_isolcpus=10000001 intel_pstate=disable nosoftlockup

   # rpm -qa |grep grub
   grub2-tools-2.02-80.el8.x86_64
   grub2-tools-minimal-2.02-80.el8.x86_64
   grub2-pc-2.02-80.el8.x86_64
   grub2-tools-extra-2.02-80.el8.x86_64
   grub2-tools-efi-2.02-80.el8.x86_64
   grubby-8.40-37.el8.x86_64
   grub2-pc-modules-2.02-80.el8.noarch
   grub2-common-2.02-80.el8.noarch

But I still see a "=1" on the kernel boot line.  Could the problem still be there or could it be leftover from before I updated the grub2 rpms.

# cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-147.el8.x86_64 root=/dev/mapper/rhel_perf130-root ro crashkernel=auto resume=/dev/mapper/rhel_perf130-swap rd.lvm.lv=rhel_perf130/root rd.lvm.lv=rhel_perf130/swap rhgb quiet crashkernel=auto =1

[root@perf130 ~]# tuned-adm active
Current active profile: network-latency

The network-latency tuned profile adds "skew_tick=1" to the kernel boot line.  I 
The "=1" goes away when I set it back to a tuned profile that doesn't add "skew_tick=1", like throughput-performance.

Please try this test.
1) Set your tuned profile to network-latency.
2) Reboot, and you should see "skew_tick=1" on the kernel command line.
3) Then edit /etc/default/grub and add some flag, like "migrations=off" to the end of the GRUB_CMDLINE_LINUX line.
4) Run "grub2-mkconfig -o /boot/grub2/grub.cfg"
5) Reboot
6) Check the /proc/cmdline.  You should see the "skew_tick=1" got changed to "=1".

Joe

Javier:
In my previous reply (#15), you can ignore that first "cat /proc/cmdline".
That was from the cpu-partitioning profile.  
I then chose to show the problem using the more simple network-latency profile, as I showed with the later "cat /proc/cmdline".

Joe

(In reply to Joe Mario from comment #16)
> Javier:
> In my previous reply (#15), you can ignore that first "cat /proc/cmdline".
> That was from the cpu-partitioning profile.  
> I then chose to show the problem using the more simple network-latency
> profile, as I showed with the later "cat /proc/cmdline".
> 
> Joe

Did you also execute grub2-install as mentioned in Commit 11?

If is a legacy BIOS installation, then you need to run grub2-install /dev/$block_dev (replace $block_dev with the appropriate block device where GRUB is installed in your machine). Otherwise the GRUB and modules won't be updated.

Reproduced on RHEL-8.2-20191120.0 x86_64 with grub2 2.02-78 using steps from comment 4

Hi Javier:
I did not run grub2-install.  I will do that (in a few hours).

Also, how do I determine if this is a "legacy BIOS installation"?

Hi Petr:
In your comment #18, when you said you "Reproduced on RHEL-8.2-20191120.0 x86_64 with grub2 2.02-78 using steps from comment 4", 
did that mean:
  a) that you reproduced the error, 
  b) or that reproduced the success?

There's really nothing in comment 4 that ties back to your wording.  Please clarify.
Thank you.
Joe

(In reply to Joe Mario from comment #19)
> Hi Javier:
> I did not run grub2-install.  I will do that (in a few hours).
> 
> Also, how do I determine if this is a "legacy BIOS installation"?
> 

You can check if the /sys/firmware/efi/ directory exists. If that's not the case then you are booting using legacy BIOS firmware.

Only for EFI systems GRUB is updated on a package upgrade right now. So that's why I mentioned that you need to execute grub2-install to update GRUB.

Hi Javier:
Thank you for your help with running grub2-install.  Once I did that the new grub fixed the problem, but with a harmless blemish.
It seems to be duplicating some of the added kernel command line flags.

Look at this example below, where I show a correct kernel command line (for a case that was never a problem).  Then I set my tuned to network-latency, where tuned adds "skew_tick=1" to the kernel boot line.  Once I reboot, I see "skew_tick=1" got added twice.

Here's the reproducer:
First show a case that was never a problem:

  # tuned-adm profile throughput-performance
  # reboot
  # cat /proc/cmdline
  BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-147.el8.x86_64 root=/dev/mapper/rhel_perf130-root ro crashkernel=auto resume=/dev/mapper/rhel_perf130-swap rd.lvm.lv=rhel_perf130/root rd.lvm.lv=rhel_perf130/swap rhgb quiet crashkernel=auto

Now set tuned to a profile that adds "skew_tick=1" to the kernel command line:

  # grep skew_tick /lib/tuned/*/tuned.conf
  /lib/tuned/network-latency/tuned.conf:cmdline=skew_tick=1
  # tuned-adm profile network-latency
  # reboot
  # cat /proc/cmdline
  BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-147.el8.x86_64 root=/dev/mapper/rhel_perf130-root ro crashkernel=auto resume=/dev/mapper/rhel_perf130-swap rd.lvm.lv=rhel_perf130/root rd.lvm.lv=rhel_perf130/swap rhgb quiet crashkernel=auto skew_tick=1 skew_tick=1

Notice the duplicate skew_tick=1 entries?

This will fix the performance problem, since two "skew_tick=1" entries are better than the "=1" entry. But you may want to fix it.
Thank you.
Joe

(In reply to Joe Mario from comment #21)
> Hi Javier:
> Thank you for your help with running grub2-install.  Once I did that the new
> grub fixed the problem, but with a harmless blemish.
> It seems to be duplicating some of the added kernel command line flags.
> 
> Look at this example below, where I show a correct kernel command line (for
> a case that was never a problem).  Then I set my tuned to network-latency,
> where tuned adds "skew_tick=1" to the kernel boot line.  Once I reboot, I
> see "skew_tick=1" got added twice.
> 
> Here's the reproducer:
> First show a case that was never a problem:
> 
>   # tuned-adm profile throughput-performance
>   # reboot
>   # cat /proc/cmdline
>   BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-147.el8.x86_64
> root=/dev/mapper/rhel_perf130-root ro crashkernel=auto
> resume=/dev/mapper/rhel_perf130-swap rd.lvm.lv=rhel_perf130/root
> rd.lvm.lv=rhel_perf130/swap rhgb quiet crashkernel=auto
> 
> Now set tuned to a profile that adds "skew_tick=1" to the kernel command
> line:
> 
>   # grep skew_tick /lib/tuned/*/tuned.conf
>   /lib/tuned/network-latency/tuned.conf:cmdline=skew_tick=1
>   # tuned-adm profile network-latency
>   # reboot
>   # cat /proc/cmdline
>   BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-147.el8.x86_64
> root=/dev/mapper/rhel_perf130-root ro crashkernel=auto
> resume=/dev/mapper/rhel_perf130-swap rd.lvm.lv=rhel_perf130/root
> rd.lvm.lv=rhel_perf130/swap rhgb quiet crashkernel=auto skew_tick=1
> skew_tick=1
> 
> Notice the duplicate skew_tick=1 entries?
> 
> This will fix the performance problem, since two "skew_tick=1" entries are
> better than the "=1" entry. But you may want to fix it.
> Thank you.
> Joe

Can you share your grubenv and BLS config files (the ones in the /boot/loader/entries directory)?

Hi Javier:
Here are the files:

First the grubenv file.  Note this is with the tuned set to "network-latency".   But if I then set my tuned profile to "throughput-performance", the skew_tick=1 no longer exists in that grubenv file.

# cat /boot/grub2/grubenv
# GRUB Environment Block
kernelopts=root=/dev/mapper/rhel_perf130-root ro crashkernel=auto resume=/dev/mapper/rhel_perf130-swap rd.lvm.lv=rhel_perf130/root rd.lvm.lv=rhel_perf130/swap rhgb quiet crashkernel=auto $tuned_params
boot_success=0
tuned_params=skew_tick=1
tuned_initrd=
saved_entry=1
##########################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################

Second, here are the config files:

# uname -a
Linux perf130.perf.lab.eng.bos.redhat.com 4.18.0-147.el8.x86_64 #1 SMP Thu Sep 26 15:52:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@perf130 ~]# cd /boot/loader/entries
[root@perf130 entries]# ls -altr
total 16
drwxr-xr-x. 3 root root  21 Sep  4  2018 ..
-rw-r--r--  1 root root 395 Nov  8  2018 7cd70c100471491da466008236452d84-0-rescue.conf
-rw-r--r--  1 root root 361 Oct 11 14:22 7cd70c100471491da466008236452d84-4.18.0-145.el8.test.x86_64.conf
-rw-r--r--  1 root root 336 Nov  8 15:58 7cd70c100471491da466008236452d84-4.18.0-147.el8.x86_64.conf
-rw-r--r--  1 root root 366 Nov 13 15:59 7cd70c100471491da466008236452d84-4.18.0-147.0.3.el8_1.x86_64.conf
drwx------. 2 root root 272 Nov 26 08:01 .
[root@perf130 entries]# more *
::::::::::::::
7cd70c100471491da466008236452d84-0-rescue.conf
::::::::::::::
title Red Hat Enterprise Linux (0-rescue-7cd70c100471491da466008236452d84) 8.0 (Ootpa)
version 0-rescue-7cd70c100471491da466008236452d84
linux /vmlinuz-0-rescue-7cd70c100471491da466008236452d84
initrd /initramfs-0-rescue-7cd70c100471491da466008236452d84.img
options $kernelopts
id rhel-0-0-rescue-7cd70c100471491da466008236452d84
grub_users $grub_users
grub_arg --unrestricted
grub_class kernel
::::::::::::::
7cd70c100471491da466008236452d84-4.18.0-145.el8.test.x86_64.conf
::::::::::::::
title Red Hat Enterprise Linux (4.18.0-145.el8.test.x86_64) 8.1 (Ootpa)
version 4.18.0-145.el8.test.x86_64
linux /vmlinuz-4.18.0-145.el8.test.x86_64
initrd /initramfs-4.18.0-145.el8.test.x86_64.img $tuned_initrd
options $kernelopts $tuned_params
id rhel-20191011113001-4.18.0-145.el8.test.x86_64
grub_users $grub_users
grub_arg --unrestricted
grub_class kernel
::::::::::::::
7cd70c100471491da466008236452d84-4.18.0-147.0.3.el8_1.x86_64.conf
::::::::::::::
title Red Hat Enterprise Linux (4.18.0-147.0.3.el8_1.x86_64) 8.0 (Ootpa)
version 4.18.0-147.0.3.el8_1.x86_64
linux /vmlinuz-4.18.0-147.0.3.el8_1.x86_64
initrd /initramfs-4.18.0-147.0.3.el8_1.x86_64.img $tuned_initrd
options $kernelopts $tuned_params
id rhel-20191111130726-4.18.0-147.0.3.el8_1.x86_64
grub_users $grub_users
grub_arg --unrestricted
grub_class kernel
::::::::::::::
7cd70c100471491da466008236452d84-4.18.0-147.el8.x86_64.conf
::::::::::::::
title Red Hat Enterprise Linux (4.18.0-147.el8.x86_64) 8.1 (Ootpa)
version 4.18.0-147.el8.x86_64
linux /vmlinuz-4.18.0-147.el8.x86_64
initrd /initramfs-4.18.0-147.el8.x86_64.img $tuned_initrd
options $kernelopts $tuned_params
id rhel-20190926160149-4.18.0-147.el8.x86_64
grub_users $grub_users
grub_arg --unrestricted
grub_class kernel
[root@perf130 entries]#

I forgot to add, here's my kernel version:

# uname -a
Linux perf130.perf.lab.eng.bos.redhat.com 4.18.0-147.el8.x86_64 #1 SMP Thu Sep 26 15:52:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

I'm not familiar with tuned...

(In reply to Joe Mario from comment #23)

[snip]

> # cat /boot/grub2/grubenv
> # GRUB Environment Block
> kernelopts=root=/dev/mapper/rhel_perf130-root ro crashkernel=auto
> resume=/dev/mapper/rhel_perf130-swap rd.lvm.lv=rhel_perf130/root
> rd.lvm.lv=rhel_perf130/swap rhgb quiet crashkernel=auto $tuned_params


...but you have $tuned_params in your $kernelopts


> 7cd70c100471491da466008236452d84-4.18.0-145.el8.test.x86_64.conf
> ::::::::::::::
> title Red Hat Enterprise Linux (4.18.0-145.el8.test.x86_64) 8.1 (Ootpa)
> version 4.18.0-145.el8.test.x86_64
> linux /vmlinuz-4.18.0-145.el8.test.x86_64
> initrd /initramfs-4.18.0-145.el8.test.x86_64.img $tuned_initrd
> options $kernelopts $tuned_params

...and again in your BLS options field

So it's expected that the skew_tick=1 will be duplicated.

Hi Javier:

I see what you're saying, that tuned_params is duplicated in the /boot/grub2/grubenv file.

I just looked about 10 RHEL-8 systems that use the cpu-partitioning or network-latency or realtime tuned profiles.  I found two others that had the same duplication of tuned_params in the /boot/grub2/grubenv file.  I don't know what steps led to that happening.

Jaroslav or Ondrej:
Do either of you know what sequence of tuned steps might lead to $tuned_params being added to the kernelopts in the grubenv file?

Here's a grubenv from a realtime profile.  Notice the duplication:

   $ more /boot/grub2/grubenv 
   # GRUB Environment Block
   saved_entry=d21ee5f858d84f6c853a6d886e7dbe04-4.18.0-80.11.2.rt9.157.el8_0.x86_64.conf
   kernelopts=root=/dev/mapper/rhel_perf11600-root ro crashkernel=auto resume=/dev/mapper/rhel_perf11600-swap rd.lvm.lv=rhel_perf11600/rootrd.lvm.lv=rhel_perf11600/swap 
   default_hugepagesz=1G iommu=pt intel_iommu=on modprobe.blacklist=be2net,lpfc $tuned_params
   boot_success=0
   tuned_params=skew_tick=1 isolcpus=1-11,13-23 intel_pstate=disable nosoftlockup
   tuned_initrd=

But Javier, even though there's duplication in the grubenv file, it has never caused duplication on the /proc/cmdline before this update grub2.

Joe

(In reply to Joe Mario from comment #26)
> Hi Javier:
> 
> I see what you're saying, that tuned_params is duplicated in the
> /boot/grub2/grubenv file.
> 
> I just looked about 10 RHEL-8 systems that use the cpu-partitioning or
> network-latency or realtime tuned profiles.  I found two others that had the
> same duplication of tuned_params in the /boot/grub2/grubenv file.  I don't
> know what steps led to that happening.
> 
> Jaroslav or Ondrej:
> Do either of you know what sequence of tuned steps might lead to
> $tuned_params being added to the kernelopts in the grubenv file?
> 

AFAIK Tuned is not touching kernelopts in grubenv. This is very probably caused by some different tool. We need reproducer.

> AFAIK Tuned is not touching kernelopts in grubenv. This is very probably caused by some different tool. We need reproducer.

Hi Jaroslav:
Look back in comment 8 of this BZ.  That's where Ondrej discusses tuned doing this.  That the $tuned_params grubenv variable can get appended both to the value of the $kernelopts grubenv variable, and also to the 'options' parameter in the BLS entries. 

I could probably come up with a reproducer, but it looks like you two may have some insight.  Let me know.
Joe

(In reply to Joe Mario from comment #28)
> > AFAIK Tuned is not touching kernelopts in grubenv. This is very probably caused by some different tool. We need reproducer.
> 
> Hi Jaroslav:
> Look back in comment 8 of this BZ.  That's where Ondrej discusses tuned
> doing this.  That the $tuned_params grubenv variable can get appended both
> to the value of the $kernelopts grubenv variable, and also to the 'options'
> parameter in the BLS entries. 
> 
> I could probably come up with a reproducer, but it looks like you two may
> have some insight.  Let me know.
> Joe

Tuned is not writing kernelopts, this is true.

But I think I know what's happening. Tuned sets GRUB_CMDLINE_LINUX_DEFAULT variable in the /etc/default/grub. It seems its content can be then written to the kernelopts by /etc/grub.d/10_linux script which is called by grub2-mkconfig.

Joe, if duplication is problem for you please open separate Tuned bugzilla, I think we can workaround this.

(In reply to Joe Mario from comment #26)

[snip]

> But Javier, even though there's duplication in the grubenv file, it has
> never caused duplication on the /proc/cmdline before this update grub2.
> 

Yes, you were not seeing those duplicated because the variables weren't expanded correctly. In other words, the grub2 bug was masking this bug about duplicated variables.

Jaroslav wrote:
  > But I think I know what's happening. Tuned sets GRUB_CMDLINE_LINUX_DEFAULT variable in the /etc/default/grub. 
  > It seems its content can be then written to the kernelopts by /etc/grub.d/10_linux script which is called by grub2-mkconfig.

Hi Jaroslav:
That's exactly it.  Running grub2-mkconfig while the current tuned setting happens to be one that modifies the kernel boot line is what provokes it.

Here's an example below.
First look at the /etc/default/grub and /boot/grub2/grubenv files on a system where a tuned profile that modifies the kernel boot line has never been used.

  # tuned-adm active
  Current active profile: latency-performance
  # cat /etc/default/grub
  GRUB_TIMEOUT=5
  GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
  GRUB_DEFAULT=saved
  GRUB_DISABLE_SUBMENU=true
  GRUB_TERMINAL_OUTPUT="console"
  GRUB_CMDLINE_LINUX="crashkernel=auto resume=/dev/mapper/rhel8.2_buzz-swap rd.lvm.lv=rhel8.2_buzz/root rd.lvm.lv=rhel8.2_buzz/swap ipv6_disable=1"
  GRUB_DISABLE_RECOVERY="true"
  GRUB_ENABLE_BLSCFG=true

  # cat /boot/grub2/grubenv
  # GRUB Environment Block
  saved_entry=5ed1a02487f742ce9e5abd9c8d3918a1-4.18.0-153.el8.x86_64
  kernelopts=root=/dev/mapper/rhel8.2_buzz-root ro crashkernel=auto resume=/dev/mapper/rhel8.2_buzz-swap rd.lvm.lv=rhel8.2_buzz/root rd.lvm.lv=rhel8.2_buzz/swap ipv6_disable=1 mem=6G
  boot_success=0
  
Now set the network-latency tuned profile, which does add "skew_tick=1" to the kernel boot line.
Notice the changes to /etc/default/grub and /boot/grub2/grubenv for $tuned_params.
It's all good so far.  No duplication (yet).

  # tuned-adm profile network-latency
  # cat /etc/default/grub
  GRUB_TIMEOUT=5
  GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
  GRUB_DEFAULT=saved
  GRUB_DISABLE_SUBMENU=true
  GRUB_TERMINAL_OUTPUT="console"
  GRUB_CMDLINE_LINUX="crashkernel=auto resume=/dev/mapper/rhel8.2_buzz-swap rd.lvm.lv=rhel8.2_buzz/root rd.lvm.lv=rhel8.2_buzz/swap ipv6_disable=1"
  GRUB_DISABLE_RECOVERY="true"
  GRUB_ENABLE_BLSCFG=true
  GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT:+$GRUB_CMDLINE_LINUX_DEFAULT }\$tuned_params"
  GRUB_INITRD_OVERLAY="${GRUB_INITRD_OVERLAY:+$GRUB_INITRD_OVERLAY }\$tuned_initrd"

  # cat /boot/grub2/grubenv
  # GRUB Environment Block
  saved_entry=5ed1a02487f742ce9e5abd9c8d3918a1-4.18.0-153.el8.x86_64
  kernelopts=root=/dev/mapper/rhel8.2_buzz-root ro crashkernel=auto resume=/dev/mapper/rhel8.2_buzz-swap rd.lvm.lv=rhel8.2_buzz/root rd.lvm.lv=rhel8.2_buzz/swap ipv6_disable=1 mem=6G
  boot_success=0
  tuned_params=skew_tick=1
  tuned_initrd=

Now run grub2-mkconfig.    
  # grub2-mkconfig -o /boot/grub2/grub.cfg
  Generating grub configuration file ...
  Found Red Hat Enterprise Linux Server release 5.11 (Tikanga) on /dev/mapper/VolGroup00-LogVol00
  Found Red Hat Enterprise Linux Server release 6.10 (Santiago) on /dev/mapper/rhel69_buzz-lv_root
  Found Red Hat Enterprise Linux Server 7.6 (Maipo) on /dev/mapper/rhel76_buzz-root
  Found Red Hat Enterprise Linux Server 7.7 (Maipo) on /dev/mapper/rhel77_buzz-root
  Found Red Hat Enterprise Linux Server 7.8 Beta (Maipo) on /dev/mapper/rhel78_buzz-root
  Found Red Hat Enterprise Linux 8.1 (Ootpa) on /dev/mapper/rhel8.1_buzz-root
  done

After grub2-mkconfig, /etc/default/grub did not change, but /boot/grub2/grubenv did change, and it now contains the duplication.
That duplication is "$tuned_params" in kernelopts, and a separate "tuned_params=" variable.

  # cat /boot/grub2/grubenv
  # GRUB Environment Block
  saved_entry=5ed1a02487f742ce9e5abd9c8d3918a1-4.18.0-153.el8.x86_64
  kernelopts=root=/dev/mapper/rhel8.2_buzz-root ro crashkernel=auto resume=/dev/mapper/rhel8.2_buzz-swap rd.lvm.lv=rhel8.2_buzz/root rd.lvm.lv=rhel8.2_buzz/swap ipv6_disable=1 $tuned_params
  boot_success=0
  tuned_params=skew_tick=1
  tuned_initrd=
  
I don't believe the duplication will cause a correctness problem, (I could be wrong).  It will look pretty bad though, especially with the many flags added by profiles like cpu-partitioning with long cpu-lists.

Javier, Jaroslav, and Ondrej:

I am happy that Javier's new grub2 does fix the "=1" problem we were seeing.

What component owns resolving the duplication issue?

(In reply to Joe Mario from comment #32)
> Javier, Jaroslav, and Ondrej:
> 
> I am happy that Javier's new grub2 does fix the "=1" problem we were seeing.
> 
> What component owns resolving the duplication issue?

Tuned bug please for the start. I think the problem is in interaction of Tuned and grub2-mkconfig. We definitely want to fix it, but according to the severity of the problem it will probably not get into 8.2.

*** Bug 1778271 has been marked as a duplicate of this bug. ***

Verified
grub2 2.02-81.el8
RHEL-8.2.0-20200227.0

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1869

Is there some way to tell if a system has this problem, programatically?  E.g. is there some way to query a system and ask "What was the version of grub used to do grub2-install?" or "What version of grub was used to do grub2-install?"  I suppose I could query the rpm package db and see what version of grub2-common is installed, but that will only work if grub2-common was not updated before I execute my query, and I cannot guarantee that.  If I can detect if the system was configured with a buggy grub, then I can upgrade grub and run grub2-install /dev/the_device, but I have no way to know that.

Background: I'm working on a kernel_settings Ansible role https://github.com/linux-system-roles/kernel_settings which uses tuned as its implementation.  Our QE and CI test platform uses cloud images.  The rhel81 and Fedora cloud images suffer from this problem.  I'd like to be able to run some command(s) and determine if I need to upgrade grub and run grub2-install /dev/the_device before configuring the bootloader cmdline settings.

*** Bug 1839341 has been marked as a duplicate of this bug. ***

(In reply to Rich Megginson from comment #39)
> Is there some way to tell if a system has this problem, programatically? 
> E.g. is there some way to query a system and ask "What was the version of
> grub used to do grub2-install?" or "What version of grub was used to do
> grub2-install?"  I suppose I could query the rpm package db and see what
> version of grub2-common is installed, but that will only work if
> grub2-common was not updated before I execute my query, and I cannot
> guarantee that.  If I can detect if the system was configured with a buggy
> grub, then I can upgrade grub and run grub2-install /dev/the_device, but I
> have no way to know that.
> 

Unfortunately I don't think there's currently a good way to figure out that.

*** Bug 1677175 has been marked as a duplicate of this bug. ***