Bug 1839341

Summary: skew_tick=1 kernel parameter included in tuned_params, but not making it to /proc/cmdline
Product: Red Hat Enterprise Linux 8 Reporter: broskos
Component: tunedAssignee: Jaroslav Škarvada <jskarvad>
Status: CLOSED DUPLICATE QA Contact: qe-baseos-daemons
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 8.1CC: broskos, jeder, jskarvad, olysonek
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: 8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-28 11:54:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description broskos 2020-05-23 13:50:52 UTC
Description of problem:
OSP 16.0 / RHEL 8.1 deployment, compute node with cpu-partitioning tuned profile set.  

tuned_params contains skew_tick=1 as expected, but /proc/cmdline only includes the '=1' part

[root@central-compute-1 grub2]# tuned-adm active
Current active profile: cpu-partitioning

[root@central-compute-1 grub2]# cat /boot/grub2/grub.cfg | grep tuned_params= 
set tuned_params="skew_tick=1 nohz=on nohz_full=2-35,38-71 rcu_nocbs=2-35,38-71 tuned.non_isolcpus=00000030,00000003 intel_pstate=disable nosoftlockup"


cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos2)/boot/vmlinuz-4.18.0-147.5.1.el8_1.x86_64 root=UUID=63a12e8d-4563-453e-9df5-d62a8fc0f0ea ro console=ttyS0 console=ttyS0,115200n81 no_timer_check crashkernel=auto rhgb quiet default_hugepagesz=1GB hugepagesz=1G hugepages=128 intel_iommu=on iommu=pt =1 nohz=on nohz_full=2-35,38-71 rcu_nocbs=2-35,38-71 tuned.non_isolcpus=00000030,00000003 intel_pstate=disable nosoftlockup

(immediately following iommu=pt)

Version-Release number of selected component (if applicable):
8.1

How reproducible:
happened on all nodes deployed with cpu-partitioning or realtime-virtual-host

Actual results:
cmdline contains =1

Expected results:
cmdline should contain skew_tick=1

Comment 1 Jaroslav Škarvada 2020-05-28 09:39:50 UTC
This is probably dupe of bug 1669252.

Comment 2 Jaroslav Škarvada 2020-05-28 09:40:35 UTC
Could you check with updated version of the grub and report back?

Comment 3 broskos 2020-05-28 10:32:14 UTC
I have the updated version of grub and I still see this issue:

grub2-common-2.02-81.el8.noarch
redhat-release-8.2-1.0.el8.x86_64


cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos2)/boot/vmlinuz-4.18.0-193.1.2.rt13.53.el8_2.x86_64 root=UUID=63a12e8d-4563-453e-9df5-d62a8fc0f0ea ro console=ttyS0 console=ttyS0,115200n81 no_timer_check crashkernel=auto rhgb quiet default_hugepagesz=1GB hugepagesz=1G hugepages=350 nx_huge_pages_recovery_ratio=0 intel_iommu=on iommu=pt elevator=cfq audit=1 nomodeset intel_idle.max_cstate=1 tsc=reliable rcu_nocb_poll=1 clocksource=tsc pcie_aspm.policy=performance =1 isolcpus=managed_irq,domain,2-35,38-71 intel_pstate=disable nosoftlockup tsc=nowatchdog nohz=on nohz_full=2-35,38-71 rcu_nocbs=2-35,38-71

# =1, but no skew_tick=1 (right before isolcpus)

cat /etc/default/grub 
GRUB_TIMEOUT=1
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="console=ttyS0 console=ttyS0,115200n81 no_timer_check  crashkernel=auto rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true
GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT:+$GRUB_CMDLINE_LINUX_DEFAULT }\$tuned_params"
TRIPLEO_HEAT_TEMPLATE_KERNEL_ARGS=" default_hugepagesz=1GB hugepagesz=1G hugepages=350 nx_huge_pages_recovery_ratio=0 intel_iommu=on iommu=pt elevator=cfq audit=1 nomodeset intel_idle.max_cstate=1 tsc=reliable rcu_nocb_poll=1 clocksource=tsc pcie_aspm.policy=performance"
GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX:+$GRUB_CMDLINE_LINUX }${TRIPLEO_HEAT_TEMPLATE_KERNEL_ARGS}"
GRUB_INITRD_OVERLAY="${GRUB_INITRD_OVERLAY:+$GRUB_INITRD_OVERLAY }\$tuned_initrd"

cat /boot/grub2/grubenv
# GRUB Environment Block
saved_entry=0a922a89c7d06319ac184e86ac15b098-4.18.0-193.1.2.rt13.53.el8_2.x86_64
kernelopts=root=UUID=63a12e8d-4563-453e-9df5-d62a8fc0f0ea ro console=ttyS0 console=ttyS0,115200n81 no_timer_check  crashkernel=auto rhgb quiet  default_hugepagesz=1GB hugepagesz=1G hugepages=350 nx_huge_pages_recovery_ratio=0 intel_iommu=on iommu=pt elevator=cfq audit=1 nomodeset intel_idle.max_cstate=1 tsc=reliable rcu_nocb_poll=1 clocksource=tsc pcie_aspm.policy=performance $tuned_params
boot_success=1
tuned_params=skew_tick=1 isolcpus=managed_irq,domain,2-35,38-71 intel_pstate=disable nosoftlockup tsc=nowatchdog nohz=on nohz_full=2-35,38-71 rcu_nocbs=2-35,38-71
tuned_initrd=
boot_indeterminate=0
########################################################################################################################################################################################################################################################################################################################

Comment 4 Jaroslav Škarvada 2020-05-28 10:57:44 UTC
Is it non-efi platform? If yes, you need to also reinstall grub by:
# grub2-install ROOT_BOOT_DEVICE

For example:

# lsblk
NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda    253:0    0  20G  0 disk 
`-vda1 253:1    0  20G  0 part /

# grub2-install /dev/vda
Installing for i386-pc platform.
Installation finished. No error reported.

Comment 5 broskos 2020-05-28 11:47:32 UTC
Thanks, running grub2-install did fix the issue.

the newer version of grub2 was installed as part of a system update 8.1 -> 8.2, pretty much 'dnf update -y'

Any idea why the update didn't take care of the re-install of grub on the root_boot_device?

Comment 6 Jaroslav Škarvada 2020-05-28 11:54:27 UTC
It's probably grub feature. I think grub has never done it. I think it doesn't know what's your ROOT_BOOT_DEVICE and don't want to touch your MBR, because you could have there your own (or 3rd party) bootloader (e.g for multiboot). I proposed KB article about this problem.

Now closing this bugzilla as a dupe.

*** This bug has been marked as a duplicate of bug 1669252 ***