Description of problem: NTO uses the containerized TuneD daemon on all OpenShift nodes. The TuneD daemon has an issue with failures to apply a profile with multiple inheritance where parents include a common ancestor fails (see rhbz#1825882). NTO needs to ship the version of TuneD that has the fix. Version-Release number of selected component (if applicable): All How reproducible: Always Steps to Reproduce: 1. Create custom profiles for NTO similar to the TuneD reproducer as in rhbz#1825882 Actual results: 2020-04-20 08:00:14,649 INFO tuned.daemon.application: dynamic tuning is globally disabled 2020-04-20 08:00:14,651 INFO tuned.daemon.daemon: using sleep interval of 1 second(s) 2020-04-20 08:00:14,652 INFO tuned.daemon.daemon: Running in automatic mode, checking what profile is recommended for your configuration. 2020-04-20 08:00:14,652 INFO tuned.daemon.daemon: Using 'openshift-realtime' profile 2020-04-20 08:00:14,653 INFO tuned.profiles.loader: loading profile: openshift-realtime 2020-04-20 08:00:14,655 ERROR tuned.daemon.daemon: Cannot set initial profile. No tunings will be enabled: Cannot load profile(s) 'openshift-realtime': Cannot find profile 'network-latency' in '['/etc/tuned', '/usr/lib/tuned']'. 2020-04-20 08:00:14,655 INFO tuned.daemon.controller: starting controller ^C2020-04-20 08:00:17,390 INFO tuned.daemon.controller: terminating controller[root@b82 ~]# ls -l /usr/lib/tuned/network-latency/tuned.conf -rw-r--r--. 1 root root 366 Dec 11 12:13 /usr/lib/tuned/network-latency/tuned.conf Expected results: Profiles successfully applied. Additional info: Upstream TuneD fix https://github.com/redhat-performance/tuned/pull/357
Fixed in 4.8.0-0.nightly-2021-06-13-101614: $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.0-0.nightly-2021-06-13-101614 True False 5h23m Cluster version is 4.8.0-0.nightly-2021-06-13-101614 $ oc project openshift-cluster-node-tuning-operator $ oc get po -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES cluster-node-tuning-operator-6999b9bf7c-mcgjj 1/1 Running 2 5h25m 10.130.0.27 jmencak-xdtq2-master-1.c.openshift-gce-devel.internal <none> <none> tuned-24v47 1/1 Running 0 5h15m 10.0.128.2 jmencak-xdtq2-worker-a-rvtb7.c.openshift-gce-devel.internal <none> <none> tuned-f45ff 1/1 Running 0 5h15m 10.0.128.3 jmencak-xdtq2-worker-b-n9dsh.c.openshift-gce-devel.internal <none> <none> tuned-kz5vm 1/1 Running 0 5h21m 10.0.0.5 jmencak-xdtq2-master-0.c.openshift-gce-devel.internal <none> <none> tuned-ntgm8 1/1 Running 0 5h21m 10.0.0.4 jmencak-xdtq2-master-2.c.openshift-gce-devel.internal <none> <none> tuned-pb4b5 1/1 Running 0 5h21m 10.0.0.3 jmencak-xdtq2-master-1.c.openshift-gce-devel.internal <none> <none> Let's verify this by trying to include two profiles that share the same parent profile "throughput-performance". An example of such profiles are the openshift-node --> openshift --> (virtual-guest) --> throughput-performance and postgresql profiles. $ oc exec tuned-24v47 -- cat /usr/lib/tuned/postgresql/tuned.conf | grep ^include= include=throughput-performance $ oc label no jmencak-xdtq2-worker-a-rvtb7.c.openshift-gce-devel.internal tuned.openshift.io/openshift-node-postgresql= node/jmencak-xdtq2-worker-a-rvtb7.c.openshift-gce-devel.internal labeled $ oc create -f- <<EOF apiVersion: tuned.openshift.io/v1 kind: Tuned metadata: name: openshift-node-postgresql namespace: openshift-cluster-node-tuning-operator spec: profile: - data: | [main] summary=Custom OpenShift node profile for PostgreSQL server include=openshift-node,postgresql name: openshift-node-postgresql recommend: - match: - label: tuned.openshift.io/openshift-node-postgresql priority: 20 profile: openshift-node-postgresql EOF tuned.tuned.openshift.io/openshift-node-postgresql created $ oc get profile NAME TUNED APPLIED DEGRADED AGE jmencak-xdtq2-master-0.c.openshift-gce-devel.internal openshift-control-plane True False 5h34m jmencak-xdtq2-master-1.c.openshift-gce-devel.internal openshift-control-plane True False 5h34m jmencak-xdtq2-master-2.c.openshift-gce-devel.internal openshift-control-plane True False 5h34m jmencak-xdtq2-worker-a-rvtb7.c.openshift-gce-devel.internal openshift-node-postgresql True False 5h29m jmencak-xdtq2-worker-b-n9dsh.c.openshift-gce-devel.internal openshift-node True False 5h29m Profile openshift-node-postgresql was successfully applied, double-check the Pod logs. $ oc logs tuned-24v47 | tail -n 20 I0614 12:17:58.801304 2534 tuned.go:644] active profile (openshift-node) != recommended profile (openshift-node-postgresql) I0614 12:17:58.801347 2534 tuned.go:499] reloading tuned... I0614 12:17:58.801354 2534 tuned.go:502] sending HUP to PID 3402 2021-06-14 12:17:58,801 INFO tuned.daemon.daemon: stopping tuning 2021-06-14 12:17:58,821 INFO tuned.daemon.daemon: terminating Tuned, rolling back all changes 2021-06-14 12:17:58,828 INFO tuned.daemon.daemon: Running in automatic mode, checking what profile is recommended for your configuration. 2021-06-14 12:17:58,829 INFO tuned.daemon.daemon: Using 'openshift-node-postgresql' profile 2021-06-14 12:17:58,830 INFO tuned.profiles.loader: loading profile: openshift-node-postgresql 2021-06-14 12:17:58,869 INFO tuned.daemon.daemon: starting tuning 2021-06-14 12:17:58,873 INFO tuned.plugins.base: instance cpu: assigning devices cpu0, cpu1, cpu2, cpu3 2021-06-14 12:17:58,874 INFO tuned.plugins.plugin_cpu: We are running on an x86 GenuineIntel platform 2021-06-14 12:17:58,876 WARNING tuned.plugins.plugin_cpu: your CPU doesn't support MSR_IA32_ENERGY_PERF_BIAS, ignoring CPU energy performance bias 2021-06-14 12:17:58,879 INFO tuned.plugins.base: instance disk: assigning devices sda 2021-06-14 12:17:58,882 INFO tuned.plugins.base: instance net: assigning devices ens4 2021-06-14 12:17:58,884 INFO tuned.plugins.plugin_cpu: setting new cpu latency 1 2021-06-14 12:17:58,887 INFO tuned.plugins.plugin_sysctl: reapplying system sysctl 2021-06-14 12:17:58,904 INFO tuned.daemon.daemon: static tuning from profile 'openshift-node-postgresql' applied I0614 12:17:59.673871 2534 tuned.go:842] updated Profile jmencak-xdtq2-worker-a-rvtb7.c.openshift-gce-devel.internal stalld=<nil>, bootcmdline: I0614 12:17:59.674143 2534 tuned.go:390] written "/etc/tuned/recommend.d/50-openshift.conf" to set Tuned profile openshift-node-postgresql I0614 12:18:00.827017 2534 tuned.go:655] active and recommended profile (openshift-node-postgresql) match; profile change will not trigger profile reload
Verified on 4.8.0-0.nightly-2021-06-13-101614
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438