Bug 1825322 - Sysctl parameters set by tuned can not be overwritten by parameters set via /etc/sysctl{.conf,.d}
Summary: Sysctl parameters set by tuned can not be overwritten by parameters set via /...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node Tuning Operator
Version: 4.3.z
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.5.0
Assignee: jmencak
QA Contact: Simon
URL:
Whiteboard:
Depends On:
Blocks: 1825375
TreeView+ depends on / blocked
 
Reported: 2020-04-17 17:19 UTC by Ryan Howe
Modified: 2020-07-13 17:28 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Tuned pods did not mount /etc/sysctl.{conf,d/} from the host. Consequence: Settings provided by the host can be overriden by tuned profiles even though this is no longer the standard behaviour of tuned as shipped in RHEL 7.3. Fix: Mount /etc/sysctl.{conf,d/} from the host in tuned pods. Result: Tuned profiles no longer override the host sysctl settings in /etc/sysctl.{conf,d/}.
Clone Of:
: 1825375 (view as bug list)
Environment:
Last Closed: 2020-07-13 17:28:35 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:2409 None None None 2020-07-13 17:28:58 UTC

Description Ryan Howe 2020-04-17 17:19:49 UTC
Description of problem:

Admin can not overwrite a sysctl that is being set by tuned because the tuned pod does not mount /etc/sysctl. 

Starting with Red Hat Enterprise Linux 7.3, Tuned no longer overrides system sysctl settings by default.  https://salsa.debian.org/debian/tuned/-/commit/d1986f20f9913cc2fbfb2e83243c2e7f268d4f77

Its not fully supported in 4.3 to add custom profiles to the tune operator. So at this time there is not a supported way to update a sysctl being set by the tuned operator. 


Version-Release number of selected component (if applicable):
4.3 

How reproducible:
100 

Steps to Reproduce:
1. Set sysctl for  kernel.pid_max on the host directly of via MCO 
2. Tuned will over write the parameter based on what is set in the tuned openshift profile 

Actual results:
value set based on openshift tune profile 

Expected results:
value to be set based on host /etc/sysctl settings

Comment 4 jmencak 2020-04-20 07:08:37 UTC
Fixed in 4.5.0-0.nightly-2020-04-19-234910 and later.

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.0-0.nightly-2020-04-19-234910   True        False         13m     Cluster version is 4.5.0-0.nightly-2020-04-19-234910

$ oc project openshift-cluster-node-tuning-operator
Now using project "openshift-cluster-node-tuning-operator" on server "https://api.jm20200420.perf-testing.devcluster.openshift.com:6443".

$ oc get pods
NAME                                            READY   STATUS    RESTARTS   AGE
cluster-node-tuning-operator-5c5cbdccc4-qld9f   1/1     Running   0          22m
tuned-245fx                                     1/1     Running   0          16m
tuned-8fgz2                                     1/1     Running   0          4m18s
tuned-dqjqg                                     1/1     Running   0          16m
tuned-sc7dk                                     1/1     Running   0          16m
tuned-txxh6                                     1/1     Running   0          4m23s

$ worker1=$(oc get nodes --no-headers | awk '/worker/ {print $1}' | head -n1)
$ worker1_tuned=$(oc get pods -o wide --no-headers | grep $worker1 | cut -d' ' -f1)

$ echo "chroot /host /bin/bash -c 'cat /etc/sysctl.d/inotify.conf'" | oc debug node/$worker1
Starting pod/ip-10-0-137-136eu-west-1computeinternal-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.137.136
If you don't see a command prompt, try pressing enter.

fs.inotify.max_user_watches = 65536
fs.inotify.max_user_instances = 8192

Removing debug pod ...

$ oc exec $worker1_tuned -- mount | grep /etc/sysctl
/dev/mapper/coreos-luks-root-nocrypt on /etc/sysctl.d type xfs (ro,relatime,seclabel,attr2,inode64,prjquota)
/dev/mapper/coreos-luks-root-nocrypt on /etc/sysctl.conf type xfs (ro,relatime,seclabel,attr2,inode64,prjquota)

$ echo "chroot /host /bin/bash -c 'sysctl kernel.pid_max'" | oc debug node/$worker1
Starting pod/ip-10-0-137-136eu-west-1computeinternal-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.137.136
If you don't see a command prompt, try pressing enter.
kernel.pid_max = 4194304

Removing debug pod ...

$ oc create -f- <<EOF
apiVersion: tuned.openshift.io/v1
kind: Tuned
metadata:
  name: override
  namespace: openshift-cluster-node-tuning-operator
spec:
  profile:
  - data: |
      [main]
      summary=Testing no-override of /etc/sysctl.d/*.conf parameters
      [sysctl]
      kernel.pid_max=1048576
      fs.inotify.max_user_watches=16384
    name: override
  recommend:
  - match:
    - label: tuned.openshift.io/override
    priority: 20
    profile: override
EOF

$ oc label node $worker1 tuned.openshift.io/override=
node/ip-10-0-137-136.eu-west-1.compute.internal labeled

$ oc logs $worker1_tuned | tail -n9
I0420 06:52:01.418926    3188 tuned.go:432] sending HUP to PID 5383
2020-04-20 06:52:01,419 INFO     tuned.daemon.daemon: stopping tuning
2020-04-20 06:52:02,318 INFO     tuned.daemon.daemon: terminating Tuned, rolling back all changes
2020-04-20 06:52:02,325 INFO     tuned.daemon.daemon: Running in automatic mode, checking what profile is recommended for your configuration.
2020-04-20 06:52:02,326 INFO     tuned.daemon.daemon: Using 'override' profile
2020-04-20 06:52:02,327 INFO     tuned.profiles.loader: loading profile: override
2020-04-20 06:52:02,328 INFO     tuned.daemon.daemon: starting tuning
2020-04-20 06:52:02,329 INFO     tuned.plugins.plugin_sysctl: reapplying system sysctl
2020-04-20 06:52:02,330 INFO     tuned.daemon.daemon: static tuning from profile 'override' applied

$ echo "chroot /host /bin/bash -c 'sysctl fs.inotify.max_user_instances'" | oc debug node/$worker1
Starting pod/ip-10-0-137-136eu-west-1computeinternal-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.137.136
If you don't see a command prompt, try pressing enter.
fs.inotify.max_user_instances = 8192

Removing debug pod ...

# fs.inotify.max_user_instances=8192 from the host was not overriden by the tuned profile override, let's see if the kernel.pid_max was changed;
# it should, as there is no host override for that setting

$ echo "chroot /host /bin/bash -c 'sysctl kernel.pid_max'" | oc debug node/$worker1
Starting pod/ip-10-0-137-136eu-west-1computeinternal-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.137.136
If you don't see a command prompt, try pressing enter.
kernel.pid_max = 1048576

Removing debug pod ...

Comment 5 Simon 2020-04-20 16:32:29 UTC
VERIFICATION POSITIVE!

oc get clusterversions.config.openshift.io
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.0-0.nightly-2020-04-20-062128   True        False         63m     Cluster version is 4.5.0-0.nightly-2020-04-20-062128

oc project openshift-cluster-node-tuning-operator
Now using project "openshift-cluster-node-tuning-operator" on server "https://api.skordas-nto-420.perf-testing.devcluster.openshift.com:6443".

$ oc get pods
NAME                                            READY   STATUS    RESTARTS   AGE
cluster-node-tuning-operator-5b4b95b55d-wx4t9   1/1     Running   0          98m
tuned-4hkp2                                     1/1     Running   0          90m
tuned-b5msp                                     1/1     Running   0          90m
tuned-dqftw                                     1/1     Running   0          90m
tuned-lwz7g                                     1/1     Running   0          81m
tuned-sd6mg                                     1/1     Running   0          81m
tuned-spggg                                     1/1     Running   0          81m

$ worker1=$(oc get nodes --no-headers | awk '/worker/ {print $1}' | head -n1)

$ worker1_tuned=$(oc get pods -o wide --no-headers | grep $worker1 | cut -d' ' -f1)

$ echo $worker1
ip-10-0-131-171.us-east-2.compute.internal

$ echo $worker1_tuned
tuned-spggg

$ echo "chroot /host /bin/bash -c 'cat /etc/sysctl.d/inotify.conf'" | oc debug node/$worker1
Starting pod/ip-10-0-131-171us-east-2computeinternal-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.131.171
If you don't see a command prompt, try pressing enter.

fs.inotify.max_user_watches = 65536
fs.inotify.max_user_instances = 8192

Removing debug pod ...

$ oc exec $worker1_tuned -- mount | grep /etc/sysctl
/dev/mapper/coreos-luks-root-nocrypt on /etc/sysctl.d type xfs (ro,relatime,seclabel,attr2,inode64,prjquota)
/dev/mapper/coreos-luks-root-nocrypt on /etc/sysctl.conf type xfs (ro,relatime,seclabel,attr2,inode64,prjquota)

$ echo "chroot /host /bin/bash -c 'sysctl kernel.pid_max'" | oc debug node/$worker1
Starting pod/ip-10-0-131-171us-east-2computeinternal-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.131.171
If you don't see a command prompt, try pressing enter.
kernel.pid_max = 4194304

Removing debug pod ...

$ oc create -f- <<EOF
> apiVersion: tuned.openshift.io/v1
> kind: Tuned
> metadata:
>   name: override
>   namespace: openshift-cluster-node-tuning-operator
> spec:
>   profile:
>   - data: |
>       [main]
>       summary=Testing no-override of /etc/sysctl.d/*.conf parameters
>       [sysctl]
>       kernel.pid_max=1048576
>       fs.inotify.max_user_watches=16384
>     name: override
>   recommend:
>   - match:
>     - label: tuned.openshift.io/override
>     priority: 20
>     profile: override
> EOF
tuned.tuned.openshift.io/override created

$ oc label node $worker1 tuned.openshift.io/override=
node/ip-10-0-131-171.us-east-2.compute.internal labeled

oc logs $worker1_tuned | tail -n9
I0420 16:13:15.572422    2252 tuned.go:432] sending HUP to PID 2867
2020-04-20 16:13:15,572 INFO     tuned.daemon.daemon: stopping tuning
2020-04-20 16:13:16,158 INFO     tuned.daemon.daemon: terminating Tuned, rolling back all changes
2020-04-20 16:13:16,166 INFO     tuned.daemon.daemon: Running in automatic mode, checking what profile is recommended for your configuration.
2020-04-20 16:13:16,166 INFO     tuned.daemon.daemon: Using 'override' profile
2020-04-20 16:13:16,167 INFO     tuned.profiles.loader: loading profile: override
2020-04-20 16:13:16,167 INFO     tuned.daemon.daemon: starting tuning
2020-04-20 16:13:16,168 INFO     tuned.plugins.plugin_sysctl: reapplying system sysctl
2020-04-20 16:13:16,169 INFO     tuned.daemon.daemon: static tuning from profile 'override' applied

$ echo "chroot /host /bin/bash -c 'sysctl fs.inotify.max_user_instances'" | oc debug node/$worker1
Starting pod/ip-10-0-131-171us-east-2computeinternal-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.131.171
If you don't see a command prompt, try pressing enter.
fs.inotify.max_user_instances = 8192

Removing debug pod ...

$ echo "chroot /host /bin/bash -c 'sysctl kernel.pid_max'" | oc debug node/$worker1
Starting pod/ip-10-0-131-171us-east-2computeinternal-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.131.171
If you don't see a command prompt, try pressing enter.
kernel.pid_max = 1048576

Removing debug pod ...

$ echo "chroot /host /bin/bash -c 'sysctl fs.inotify.max_user_watches'" | oc debug node/$worker1
Starting pod/ip-10-0-131-171us-east-2computeinternal-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.131.171
If you don't see a command prompt, try pressing enter.
fs.inotify.max_user_watches = 65536

Removing debug pod ...


# kernel.pid_max = 1048576 - from override tuned profile
# fs.inotify.max_user_watches = 65536 - from /etc/sysctl even this is changed in override tuned profile :)

Comment 6 errata-xmlrpc 2020-07-13 17:28:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.