Bug 1707592 - Values from tuned default doesn't apply
Summary: Values from tuned default doesn't apply
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node Tuning Operator
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Jiří Mencák
QA Contact: Mike Fiedler
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-05-07 20:09 UTC by Simon
Modified: 2019-05-08 13:27 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Not a bug, user education.
Clone Of:
Environment:
Last Closed: 2019-05-08 07:59:08 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Tuned default (3.36 KB, text/plain)
2019-05-07 20:09 UTC, Simon
no flags Details
Pod log (11.94 KB, text/plain)
2019-05-07 20:10 UTC, Simon
no flags Details

Description Simon 2019-05-07 20:09:39 UTC
Created attachment 1565354 [details]
Tuned default

Description of problem:
Values from tuned default doesn't apply.

Version-Release number of selected component (if applicable):
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.1.0-0.nightly-2019-05-07-155635   True        False         30m     Cluster version is 4.1.0-0.nightly-2019-05-07-155635

$ oc get co/node-tuning
NAME          VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
node-tuning   4.1.0-0.nightly-2019-05-07-155635   True        False         False      40m


How reproducible:
100%

Steps to Reproduce:
1. Default next-gen installation. 
2. Edit default values:
$ oc edit tuned default -n openshift-cluster-node-tuning-operator
ex.
...
net.netfilter.nf_conntrack_max=1048578
...

3. Save changes, check value on every node:
$ for node in $(oc get nodes --no-headers | cut -d ' ' -f 1); do oc debug node/$node -- chroot /host && sysctl net.netfilter.nf_conntrack_max; done


Actual results:
net.netfilter.nf_conntrack_max = 262144


Expected results:
the same like in tuned default: net.netfilter.nf_conntrack_max=1048578

Comment 1 Simon 2019-05-07 20:10:07 UTC
Created attachment 1565355 [details]
Pod log

Comment 2 Jiří Mencák 2019-05-08 07:59:08 UTC
You're checking the sysctl on your desktop/wherever you're running your "oc debug node" command from.  You need to drop the "&&" in your "oc debug node" one-liner.

I cannot reproduce the issue on exactly the same nightly:

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.1.0-0.nightly-2019-05-07-155635   True        False         9m27s   Cluster version is 4.1.0-0.nightly-2019-05-07-155635

$ oc get nodes
NAME                                         STATUS   ROLES    AGE   VERSION
ip-10-0-130-239.us-west-2.compute.internal   Ready    master   20m   v1.13.4+0af5e0f8b
ip-10-0-143-89.us-west-2.compute.internal    Ready    worker   13m   v1.13.4+0af5e0f8b
ip-10-0-158-40.us-west-2.compute.internal    Ready    worker   12m   v1.13.4+0af5e0f8b
ip-10-0-181-151.us-west-2.compute.internal   Ready    worker   13m   v1.13.4+0af5e0f8b

$ oc project openshift-cluster-node-tuning-operator
Now using project "openshift-cluster-node-tuning-operator" on server "https://api.jmencak.perf-testing.devcluster.openshift.com:6443".

$ oc edit tuned/default
tuned.tuned.openshift.io/default edited

$ date
Wed 08 May 2019 09:14:38 AM CEST

$ oc get pods -o wide
NAME                                            READY   STATUS    RESTARTS   AGE   IP             NODE                                         NOMINATED NODE   READINESS GATES
cluster-node-tuning-operator-54f64947f5-g7fg9   1/1     Running   0          17m   10.128.0.44    ip-10-0-130-239.us-west-2.compute.internal   <none>           <none>
tuned-c4vxk                                     1/1     Running   0          14m   10.0.158.40    ip-10-0-158-40.us-west-2.compute.internal    <none>           <none>
tuned-ksv2v                                     1/1     Running   0          14m   10.0.181.151   ip-10-0-181-151.us-west-2.compute.internal   <none>           <none>
tuned-n7wrx                                     1/1     Running   0          16m   10.0.130.239   ip-10-0-130-239.us-west-2.compute.internal   <none>           <none>
tuned-s48z5                                     1/1     Running   0          14m   10.0.143.89    ip-10-0-143-89.us-west-2.compute.internal    <none>           <none>

$ oc logs tuned-c4vxk | tail -n20
I0508 07:14:47.069790    1667 openshift-tuned.go:638] Remove event on: /var/lib/tuned/profiles-data/..2019_05_08_07_00_18.128954904
I0508 07:14:48.589697    1667 openshift-tuned.go:176] Extracting tuned profiles
I0508 07:14:48.590533    1667 openshift-tuned.go:215] Reloading tuned...
2019-05-08 07:14:48,681 INFO     tuned.daemon.application: dynamic tuning is globally disabled
2019-05-08 07:14:48,686 INFO     tuned.daemon.daemon: using sleep interval of 1 second(s)
2019-05-08 07:14:48,686 INFO     tuned.daemon.daemon: Running in automatic mode, checking what profile is recommended for your configuration.
2019-05-08 07:14:48,688 INFO     tuned.daemon.daemon: Using 'openshift-node' profile
2019-05-08 07:14:48,688 INFO     tuned.profiles.loader: loading profile: openshift-node
2019-05-08 07:14:48,723 WARNING  tuned.daemon.application: Using one shot no deamon mode, most of the functionality will be not available, it can be changed in global config
2019-05-08 07:14:48,723 INFO     tuned.daemon.controller: starting controller
2019-05-08 07:14:48,723 INFO     tuned.daemon.daemon: starting tuning
2019-05-08 07:14:48,725 INFO     tuned.daemon.controller: terminating controller
2019-05-08 07:14:48,725 INFO     tuned.daemon.daemon: stopping tuning
2019-05-08 07:14:48,732 INFO     tuned.plugins.base: instance cpu: assigning devices cpu0, cpu1
2019-05-08 07:14:48,735 WARNING  tuned.plugins.plugin_cpu: your CPU doesn't support MSR_IA32_ENERGY_PERF_BIAS, ignoring CPU energy performance bias
2019-05-08 07:14:48,736 INFO     tuned.plugins.base: instance disk: assigning devices xvda
2019-05-08 07:14:48,736 INFO     tuned.plugins.base: instance net: assigning devices ens3
2019-05-08 07:14:48,999 INFO     tuned.plugins.plugin_sysctl: reapplying system sysctl
2019-05-08 07:14:49,004 INFO     tuned.daemon.daemon: static tuning from profile 'openshift-node' applied
2019-05-08 07:14:49,012 INFO     tuned.daemon.daemon: terminating Tuned

$ for node in $(oc get nodes --no-headers | cut -d ' ' -f 1); do oc debug node/$node -- chroot /host sysctl net.netfilter.nf_conntrack_max; done
Starting pod/ip-10-0-130-239us-west-2computeinternal-debug ...
To use host binaries, run `chroot /host`
net.netfilter.nf_conntrack_max = 1048578

Removing debug pod ...
Starting pod/ip-10-0-143-89us-west-2computeinternal-debug ...
To use host binaries, run `chroot /host`
net.netfilter.nf_conntrack_max = 1048578

Removing debug pod ...
Starting pod/ip-10-0-158-40us-west-2computeinternal-debug ...
To use host binaries, run `chroot /host`
net.netfilter.nf_conntrack_max = 1048578

Removing debug pod ...
Starting pod/ip-10-0-181-151us-west-2computeinternal-debug ...
To use host binaries, run `chroot /host`
net.netfilter.nf_conntrack_max = 1048578

Removing debug pod ...

Also, please note that:
1) tuned-profiles/tuned-recommend ConfigMap change propagation to tuned pods might take a while
2) the default tuned CR is not really meant for user changes and you should use custom tuned CRs instead

Closing the bug, please re-open if the solution doesn't work for you.

Comment 3 Simon 2019-05-08 13:27:54 UTC
http://www.reactiongifs.com/r/whid.gif
you have absolutely right! My bad. Sorry for taking your time.


Note You need to log in before you can comment on or make changes to this bug.