2114681 – Kernel parm needs to be added when a pao performance profile is applied, rcutree.kthread_prio=11

Bug 2114681 - Kernel parm needs to be added when a pao performance profile is applied, rcutree.kthread_prio=11

Summary: Kernel parm needs to be added when a pao performance profile is applied, rcut...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Performance Addon Operator
Sub Component:
Version:	4.11
Hardware:	x86_64
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	4.11.z
Assignee:	Vitaly Grinberg
QA Contact:	Shereen Haj Makhoul
Docs Contact:
URL:
Whiteboard:
Depends On:	2102450
Blocks:
TreeView+	depends on / blocked

Reported:	2022-08-03 05:49 UTC by OpenShift BugZilla Robot
Modified:	2022-09-07 20:49 UTC (History)
CC List:	6 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2022-09-07 20:49:26 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	openshift cluster-node-tuning-operator pull 414	0	None	open	[release-4.11] Bug 2114681: set rcutree.kthread_prio to ksoftirqd prio	2022-08-03 05:50:09 UTC
Red Hat Product Errata	RHSA-2022:6287	0	None	None	None	2022-09-07 20:49:35 UTC

Comment 2 Shereen Haj Makhoul 2022-08-23 15:58:59 UTC

Verification: 

Version: 
OCP: 4.11.1


Steps: 

1. apply the following PP with RT enabled:

apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
  name: performance
spec:
  cpu:
    isolated: "0-2"
    reserved: "3"
  nodeSelector:
    node-role.kubernetes.io/worker: ""
  realTimeKernel:
    enabled: true

2. once nodes are up, verify the kernel parameter was set to 11:

2.1: 
[root@registry ~]# oc get tuned -A
NAMESPACE                                NAME                                     AGE
openshift-cluster-node-tuning-operator   default                                  92m
openshift-cluster-node-tuning-operator   openshift-node-performance-performance   21m
openshift-cluster-node-tuning-operator   rendered                                 92m
[root@registry ~]# oc get tuned openshift-node-performance-performance -n openshift-cluster-node-tuning-operator -o yaml | grep "rcutree.kthread_prio"
      tsc=nowatchdog nosoftlockup nmi_watchdog=0 mce=off skew_tick=1 rcutree.kthread_prio=11\n\n\n\n\n\n\ncmdline_hugepages=+

2.2:

[root@registry ~]# oc get profiles -A
NAMESPACE                                NAME       TUNED                                    APPLIED   DEGRADED   AGE
openshift-cluster-node-tuning-operator   master-0   openshift-control-plane                  True      False      104m
openshift-cluster-node-tuning-operator   master-1   openshift-control-plane                  True      False      104m
openshift-cluster-node-tuning-operator   master-2   openshift-control-plane                  True      False      104m
openshift-cluster-node-tuning-operator   worker-0   openshift-node-performance-performance   True      True       73m
openshift-cluster-node-tuning-operator   worker-1   openshift-node-performance-performance   True      True       72m

[root@registry ~]# oc get profiles worker-0 -n openshift-cluster-node-tuning-operator -o yaml | grep "rcutree.kthread_prio" 
^[[A    nosoftlockup nmi_watchdog=0 mce=off skew_tick=1 rcutree.kthread_prio=11 + +
[root@registry ~]# oc get profiles worker-1 -n openshift-cluster-node-tuning-operator -o yaml | grep "rcutree.kthread_prio" 
    nosoftlockup nmi_watchdog=0 mce=off skew_tick=1 rcutree.kthread_prio=11 + +


2.3: from the nodes:
worker-0:
sh-4.4# cat  /proc/cmdline | grep rcutree.kthread_prio
BOOT_IMAGE=(hd0,gpt3)/ostree/rhcos-61413a411598b33ae4d1db997db327de680a9ca8b5216a7ea19fbbb3e7f78ec2/vmlinuz-4.18.0-372.19.1.rt7.176.el8_6.x86_64 random.trust_cpu=on console=tty0 console=ttyS0,115200n8 ignition.platform.id=metal ostree=/ostree/boot.0/rhcos/61413a411598b33ae4d1db997db327de680a9ca8b5216a7ea19fbbb3e7f78ec2/0 ip=dhcp root=UUID=a14a8f60-de23-4b0b-9ac6-3156fa908bbb rw rootflags=prjquota boot=UUID=1590adf7-8b9d-4647-9c3f-0ab32e304552 skew_tick=1 nohz=on rcu_nocbs=0-2 tuned.non_isolcpus=0000ffff,ffffffff,fffffff8 systemd.cpu_affinity=3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79 intel_iommu=on iommu=pt isolcpus=managed_irq,0-2 nohz_full=0-2 tsc=nowatchdog nosoftlockup nmi_watchdog=0 mce=off skew_tick=1 rcutree.kthread_prio=11 + +
sh-4.4#

worker-1:
sh-4.4# cat  /proc/cmdline | grep rcutree.kthread_prio
BOOT_IMAGE=(hd0,gpt3)/ostree/rhcos-61413a411598b33ae4d1db997db327de680a9ca8b5216a7ea19fbbb3e7f78ec2/vmlinuz-4.18.0-372.19.1.rt7.176.el8_6.x86_64 random.trust_cpu=on console=tty0 console=ttyS0,115200n8 ignition.platform.id=metal ostree=/ostree/boot.0/rhcos/61413a411598b33ae4d1db997db327de680a9ca8b5216a7ea19fbbb3e7f78ec2/0 ip=dhcp root=UUID=5b4a95be-43a2-4b89-b7af-3692bd43170d rw rootflags=prjquota boot=UUID=9fbbe4d8-430a-4369-888a-2f53f5f8ecee skew_tick=1 nohz=on rcu_nocbs=0-2 tuned.non_isolcpus=0000ffff,ffffffff,fffffff8 systemd.cpu_affinity=3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79 intel_iommu=on iommu=pt isolcpus=managed_irq,0-2 nohz_full=0-2 tsc=nowatchdog nosoftlockup nmi_watchdog=0 mce=off skew_tick=1 rcutree.kthread_prio=11 + +

As can be seen, KConfig was updated with the new value rcutree.kthread_prio=11.

Comment 3 Shereen Haj Makhoul 2022-08-23 16:33:22 UTC

moving back to QA as the last say is for OCP-QE

Comment 5 Gowrishankar Rajaiyan 2022-09-01 16:42:40 UTC

Verified as per comment #2.

Comment 7 errata-xmlrpc 2022-09-07 20:49:26 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.11.3 packages and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:6287

Note You need to log in before you can comment on or make changes to this bug.