Bug 1969832

Summary: Applying a profile with multiple inheritance where parents include a common ancestor fails
Product: OpenShift Container Platform Reporter: Jiří Mencák <jmencak>
Component: Node Tuning OperatorAssignee: Jiří Mencák <jmencak>
Status: CLOSED ERRATA QA Contact: Simon <skordas>
Severity: high Docs Contact:
Priority: high    
Version: 4.8CC: sejug
Target Milestone: ---   
Target Release: 4.8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-07-27 23:12:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jiří Mencák 2021-06-09 09:53:47 UTC 
Description of problem:
NTO uses the containerized TuneD daemon on all OpenShift nodes.  The TuneD daemon has an issue with failures to apply a profile with multiple inheritance where parents include a common ancestor fails (see rhbz#1825882).  NTO needs to ship the version of TuneD that has the fix.

Version-Release number of selected component (if applicable):
All

How reproducible:
Always

Steps to Reproduce:
1. Create custom profiles for NTO similar to the TuneD reproducer as in rhbz#1825882

Actual results:
2020-04-20 08:00:14,649 INFO     tuned.daemon.application: dynamic tuning is globally disabled
2020-04-20 08:00:14,651 INFO     tuned.daemon.daemon: using sleep interval of 1 second(s)
2020-04-20 08:00:14,652 INFO     tuned.daemon.daemon: Running in automatic mode, checking what profile is recommended for your configuration.
2020-04-20 08:00:14,652 INFO     tuned.daemon.daemon: Using 'openshift-realtime' profile
2020-04-20 08:00:14,653 INFO     tuned.profiles.loader: loading profile: openshift-realtime
2020-04-20 08:00:14,655 ERROR    tuned.daemon.daemon: Cannot set initial profile. No tunings will be enabled: Cannot load profile(s) 'openshift-realtime': Cannot find profile 'network-latency' in '['/etc/tuned', '/usr/lib/tuned']'.
2020-04-20 08:00:14,655 INFO     tuned.daemon.controller: starting controller
^C2020-04-20 08:00:17,390 INFO     tuned.daemon.controller: terminating controller[root@b82 ~]# ls -l /usr/lib/tuned/network-latency/tuned.conf 
-rw-r--r--. 1 root root 366 Dec 11 12:13 /usr/lib/tuned/network-latency/tuned.conf


Expected results:
Profiles successfully applied.

Additional info:
Upstream TuneD fix
https://github.com/redhat-performance/tuned/pull/357
Comment 2 Jiří Mencák 2021-06-14 12:31:12 UTC 
Fixed in 4.8.0-0.nightly-2021-06-13-101614:

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-06-13-101614   True        False         5h23m   Cluster version is 4.8.0-0.nightly-2021-06-13-101614

$ oc project openshift-cluster-node-tuning-operator

$ oc get po -o wide
NAME                                            READY   STATUS    RESTARTS   AGE     IP            NODE                                                          NOMINATED NODE   READINESS GATES
cluster-node-tuning-operator-6999b9bf7c-mcgjj   1/1     Running   2          5h25m   10.130.0.27   jmencak-xdtq2-master-1.c.openshift-gce-devel.internal         <none>           <none>
tuned-24v47                                     1/1     Running   0          5h15m   10.0.128.2    jmencak-xdtq2-worker-a-rvtb7.c.openshift-gce-devel.internal   <none>           <none>
tuned-f45ff                                     1/1     Running   0          5h15m   10.0.128.3    jmencak-xdtq2-worker-b-n9dsh.c.openshift-gce-devel.internal   <none>           <none>
tuned-kz5vm                                     1/1     Running   0          5h21m   10.0.0.5      jmencak-xdtq2-master-0.c.openshift-gce-devel.internal         <none>           <none>
tuned-ntgm8                                     1/1     Running   0          5h21m   10.0.0.4      jmencak-xdtq2-master-2.c.openshift-gce-devel.internal         <none>           <none>
tuned-pb4b5                                     1/1     Running   0          5h21m   10.0.0.3      jmencak-xdtq2-master-1.c.openshift-gce-devel.internal         <none>           <none>

Let's verify this by trying to include two profiles that share the same parent profile "throughput-performance".
An example of such profiles are the openshift-node --> openshift --> (virtual-guest) --> throughput-performance
and postgresql profiles.

$ oc exec tuned-24v47 -- cat /usr/lib/tuned/postgresql/tuned.conf | grep ^include=
include=throughput-performance

$ oc label no jmencak-xdtq2-worker-a-rvtb7.c.openshift-gce-devel.internal tuned.openshift.io/openshift-node-postgresql=
node/jmencak-xdtq2-worker-a-rvtb7.c.openshift-gce-devel.internal labeled

$ oc create -f- <<EOF
apiVersion: tuned.openshift.io/v1
kind: Tuned
metadata:
  name: openshift-node-postgresql
  namespace: openshift-cluster-node-tuning-operator
spec:
  profile:
  - data: |
      [main]
      summary=Custom OpenShift node profile for PostgreSQL server
      include=openshift-node,postgresql
    name: openshift-node-postgresql

  recommend:
  - match:
    - label: tuned.openshift.io/openshift-node-postgresql
    priority: 20
    profile: openshift-node-postgresql
EOF
tuned.tuned.openshift.io/openshift-node-postgresql created

$ oc get profile
NAME                                                          TUNED                       APPLIED   DEGRADED   AGE
jmencak-xdtq2-master-0.c.openshift-gce-devel.internal         openshift-control-plane     True      False      5h34m
jmencak-xdtq2-master-1.c.openshift-gce-devel.internal         openshift-control-plane     True      False      5h34m
jmencak-xdtq2-master-2.c.openshift-gce-devel.internal         openshift-control-plane     True      False      5h34m
jmencak-xdtq2-worker-a-rvtb7.c.openshift-gce-devel.internal   openshift-node-postgresql   True      False      5h29m
jmencak-xdtq2-worker-b-n9dsh.c.openshift-gce-devel.internal   openshift-node              True      False      5h29m

Profile openshift-node-postgresql was successfully applied, double-check the Pod logs.

$ oc logs tuned-24v47 | tail -n 20
I0614 12:17:58.801304    2534 tuned.go:644] active profile (openshift-node) != recommended profile (openshift-node-postgresql)
I0614 12:17:58.801347    2534 tuned.go:499] reloading tuned...
I0614 12:17:58.801354    2534 tuned.go:502] sending HUP to PID 3402
2021-06-14 12:17:58,801 INFO     tuned.daemon.daemon: stopping tuning
2021-06-14 12:17:58,821 INFO     tuned.daemon.daemon: terminating Tuned, rolling back all changes
2021-06-14 12:17:58,828 INFO     tuned.daemon.daemon: Running in automatic mode, checking what profile is recommended for your configuration.
2021-06-14 12:17:58,829 INFO     tuned.daemon.daemon: Using 'openshift-node-postgresql' profile
2021-06-14 12:17:58,830 INFO     tuned.profiles.loader: loading profile: openshift-node-postgresql
2021-06-14 12:17:58,869 INFO     tuned.daemon.daemon: starting tuning
2021-06-14 12:17:58,873 INFO     tuned.plugins.base: instance cpu: assigning devices cpu0, cpu1, cpu2, cpu3
2021-06-14 12:17:58,874 INFO     tuned.plugins.plugin_cpu: We are running on an x86 GenuineIntel platform
2021-06-14 12:17:58,876 WARNING  tuned.plugins.plugin_cpu: your CPU doesn't support MSR_IA32_ENERGY_PERF_BIAS, ignoring CPU energy performance bias
2021-06-14 12:17:58,879 INFO     tuned.plugins.base: instance disk: assigning devices sda
2021-06-14 12:17:58,882 INFO     tuned.plugins.base: instance net: assigning devices ens4
2021-06-14 12:17:58,884 INFO     tuned.plugins.plugin_cpu: setting new cpu latency 1
2021-06-14 12:17:58,887 INFO     tuned.plugins.plugin_sysctl: reapplying system sysctl
2021-06-14 12:17:58,904 INFO     tuned.daemon.daemon: static tuning from profile 'openshift-node-postgresql' applied
I0614 12:17:59.673871    2534 tuned.go:842] updated Profile jmencak-xdtq2-worker-a-rvtb7.c.openshift-gce-devel.internal stalld=<nil>, bootcmdline: 
I0614 12:17:59.674143    2534 tuned.go:390] written "/etc/tuned/recommend.d/50-openshift.conf" to set Tuned profile openshift-node-postgresql
I0614 12:18:00.827017    2534 tuned.go:655] active and recommended profile (openshift-node-postgresql) match; profile change will not trigger profile reload
Comment 3 Simon 2021-06-14 17:34:24 UTC 
Verified on 4.8.0-0.nightly-2021-06-13-101614
Comment 6 errata-xmlrpc 2021-07-27 23:12:22 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438