Hide Forgot
Description of problem: Admin can not overwrite a sysctl that is being set by tuned because the tuned pod does not mount /etc/sysctl. Starting with Red Hat Enterprise Linux 7.3, Tuned no longer overrides system sysctl settings by default. https://salsa.debian.org/debian/tuned/-/commit/d1986f20f9913cc2fbfb2e83243c2e7f268d4f77 Its not fully supported in 4.3 to add custom profiles to the tune operator. So at this time there is not a supported way to update a sysctl being set by the tuned operator. Version-Release number of selected component (if applicable): 4.3 How reproducible: 100 Steps to Reproduce: 1. Set sysctl for kernel.pid_max on the host directly of via MCO 2. Tuned will over write the parameter based on what is set in the tuned openshift profile Actual results: value set based on openshift tune profile Expected results: value to be set based on host /etc/sysctl settings
Fixed in 4.5.0-0.nightly-2020-04-19-234910 and later. $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.5.0-0.nightly-2020-04-19-234910 True False 13m Cluster version is 4.5.0-0.nightly-2020-04-19-234910 $ oc project openshift-cluster-node-tuning-operator Now using project "openshift-cluster-node-tuning-operator" on server "https://api.jm20200420.perf-testing.devcluster.openshift.com:6443". $ oc get pods NAME READY STATUS RESTARTS AGE cluster-node-tuning-operator-5c5cbdccc4-qld9f 1/1 Running 0 22m tuned-245fx 1/1 Running 0 16m tuned-8fgz2 1/1 Running 0 4m18s tuned-dqjqg 1/1 Running 0 16m tuned-sc7dk 1/1 Running 0 16m tuned-txxh6 1/1 Running 0 4m23s $ worker1=$(oc get nodes --no-headers | awk '/worker/ {print $1}' | head -n1) $ worker1_tuned=$(oc get pods -o wide --no-headers | grep $worker1 | cut -d' ' -f1) $ echo "chroot /host /bin/bash -c 'cat /etc/sysctl.d/inotify.conf'" | oc debug node/$worker1 Starting pod/ip-10-0-137-136eu-west-1computeinternal-debug ... To use host binaries, run `chroot /host` Pod IP: 10.0.137.136 If you don't see a command prompt, try pressing enter. fs.inotify.max_user_watches = 65536 fs.inotify.max_user_instances = 8192 Removing debug pod ... $ oc exec $worker1_tuned -- mount | grep /etc/sysctl /dev/mapper/coreos-luks-root-nocrypt on /etc/sysctl.d type xfs (ro,relatime,seclabel,attr2,inode64,prjquota) /dev/mapper/coreos-luks-root-nocrypt on /etc/sysctl.conf type xfs (ro,relatime,seclabel,attr2,inode64,prjquota) $ echo "chroot /host /bin/bash -c 'sysctl kernel.pid_max'" | oc debug node/$worker1 Starting pod/ip-10-0-137-136eu-west-1computeinternal-debug ... To use host binaries, run `chroot /host` Pod IP: 10.0.137.136 If you don't see a command prompt, try pressing enter. kernel.pid_max = 4194304 Removing debug pod ... $ oc create -f- <<EOF apiVersion: tuned.openshift.io/v1 kind: Tuned metadata: name: override namespace: openshift-cluster-node-tuning-operator spec: profile: - data: | [main] summary=Testing no-override of /etc/sysctl.d/*.conf parameters [sysctl] kernel.pid_max=1048576 fs.inotify.max_user_watches=16384 name: override recommend: - match: - label: tuned.openshift.io/override priority: 20 profile: override EOF $ oc label node $worker1 tuned.openshift.io/override= node/ip-10-0-137-136.eu-west-1.compute.internal labeled $ oc logs $worker1_tuned | tail -n9 I0420 06:52:01.418926 3188 tuned.go:432] sending HUP to PID 5383 2020-04-20 06:52:01,419 INFO tuned.daemon.daemon: stopping tuning 2020-04-20 06:52:02,318 INFO tuned.daemon.daemon: terminating Tuned, rolling back all changes 2020-04-20 06:52:02,325 INFO tuned.daemon.daemon: Running in automatic mode, checking what profile is recommended for your configuration. 2020-04-20 06:52:02,326 INFO tuned.daemon.daemon: Using 'override' profile 2020-04-20 06:52:02,327 INFO tuned.profiles.loader: loading profile: override 2020-04-20 06:52:02,328 INFO tuned.daemon.daemon: starting tuning 2020-04-20 06:52:02,329 INFO tuned.plugins.plugin_sysctl: reapplying system sysctl 2020-04-20 06:52:02,330 INFO tuned.daemon.daemon: static tuning from profile 'override' applied $ echo "chroot /host /bin/bash -c 'sysctl fs.inotify.max_user_instances'" | oc debug node/$worker1 Starting pod/ip-10-0-137-136eu-west-1computeinternal-debug ... To use host binaries, run `chroot /host` Pod IP: 10.0.137.136 If you don't see a command prompt, try pressing enter. fs.inotify.max_user_instances = 8192 Removing debug pod ... # fs.inotify.max_user_instances=8192 from the host was not overriden by the tuned profile override, let's see if the kernel.pid_max was changed; # it should, as there is no host override for that setting $ echo "chroot /host /bin/bash -c 'sysctl kernel.pid_max'" | oc debug node/$worker1 Starting pod/ip-10-0-137-136eu-west-1computeinternal-debug ... To use host binaries, run `chroot /host` Pod IP: 10.0.137.136 If you don't see a command prompt, try pressing enter. kernel.pid_max = 1048576 Removing debug pod ...
VERIFICATION POSITIVE! oc get clusterversions.config.openshift.io NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.5.0-0.nightly-2020-04-20-062128 True False 63m Cluster version is 4.5.0-0.nightly-2020-04-20-062128 oc project openshift-cluster-node-tuning-operator Now using project "openshift-cluster-node-tuning-operator" on server "https://api.skordas-nto-420.perf-testing.devcluster.openshift.com:6443". $ oc get pods NAME READY STATUS RESTARTS AGE cluster-node-tuning-operator-5b4b95b55d-wx4t9 1/1 Running 0 98m tuned-4hkp2 1/1 Running 0 90m tuned-b5msp 1/1 Running 0 90m tuned-dqftw 1/1 Running 0 90m tuned-lwz7g 1/1 Running 0 81m tuned-sd6mg 1/1 Running 0 81m tuned-spggg 1/1 Running 0 81m $ worker1=$(oc get nodes --no-headers | awk '/worker/ {print $1}' | head -n1) $ worker1_tuned=$(oc get pods -o wide --no-headers | grep $worker1 | cut -d' ' -f1) $ echo $worker1 ip-10-0-131-171.us-east-2.compute.internal $ echo $worker1_tuned tuned-spggg $ echo "chroot /host /bin/bash -c 'cat /etc/sysctl.d/inotify.conf'" | oc debug node/$worker1 Starting pod/ip-10-0-131-171us-east-2computeinternal-debug ... To use host binaries, run `chroot /host` Pod IP: 10.0.131.171 If you don't see a command prompt, try pressing enter. fs.inotify.max_user_watches = 65536 fs.inotify.max_user_instances = 8192 Removing debug pod ... $ oc exec $worker1_tuned -- mount | grep /etc/sysctl /dev/mapper/coreos-luks-root-nocrypt on /etc/sysctl.d type xfs (ro,relatime,seclabel,attr2,inode64,prjquota) /dev/mapper/coreos-luks-root-nocrypt on /etc/sysctl.conf type xfs (ro,relatime,seclabel,attr2,inode64,prjquota) $ echo "chroot /host /bin/bash -c 'sysctl kernel.pid_max'" | oc debug node/$worker1 Starting pod/ip-10-0-131-171us-east-2computeinternal-debug ... To use host binaries, run `chroot /host` Pod IP: 10.0.131.171 If you don't see a command prompt, try pressing enter. kernel.pid_max = 4194304 Removing debug pod ... $ oc create -f- <<EOF > apiVersion: tuned.openshift.io/v1 > kind: Tuned > metadata: > name: override > namespace: openshift-cluster-node-tuning-operator > spec: > profile: > - data: | > [main] > summary=Testing no-override of /etc/sysctl.d/*.conf parameters > [sysctl] > kernel.pid_max=1048576 > fs.inotify.max_user_watches=16384 > name: override > recommend: > - match: > - label: tuned.openshift.io/override > priority: 20 > profile: override > EOF tuned.tuned.openshift.io/override created $ oc label node $worker1 tuned.openshift.io/override= node/ip-10-0-131-171.us-east-2.compute.internal labeled oc logs $worker1_tuned | tail -n9 I0420 16:13:15.572422 2252 tuned.go:432] sending HUP to PID 2867 2020-04-20 16:13:15,572 INFO tuned.daemon.daemon: stopping tuning 2020-04-20 16:13:16,158 INFO tuned.daemon.daemon: terminating Tuned, rolling back all changes 2020-04-20 16:13:16,166 INFO tuned.daemon.daemon: Running in automatic mode, checking what profile is recommended for your configuration. 2020-04-20 16:13:16,166 INFO tuned.daemon.daemon: Using 'override' profile 2020-04-20 16:13:16,167 INFO tuned.profiles.loader: loading profile: override 2020-04-20 16:13:16,167 INFO tuned.daemon.daemon: starting tuning 2020-04-20 16:13:16,168 INFO tuned.plugins.plugin_sysctl: reapplying system sysctl 2020-04-20 16:13:16,169 INFO tuned.daemon.daemon: static tuning from profile 'override' applied $ echo "chroot /host /bin/bash -c 'sysctl fs.inotify.max_user_instances'" | oc debug node/$worker1 Starting pod/ip-10-0-131-171us-east-2computeinternal-debug ... To use host binaries, run `chroot /host` Pod IP: 10.0.131.171 If you don't see a command prompt, try pressing enter. fs.inotify.max_user_instances = 8192 Removing debug pod ... $ echo "chroot /host /bin/bash -c 'sysctl kernel.pid_max'" | oc debug node/$worker1 Starting pod/ip-10-0-131-171us-east-2computeinternal-debug ... To use host binaries, run `chroot /host` Pod IP: 10.0.131.171 If you don't see a command prompt, try pressing enter. kernel.pid_max = 1048576 Removing debug pod ... $ echo "chroot /host /bin/bash -c 'sysctl fs.inotify.max_user_watches'" | oc debug node/$worker1 Starting pod/ip-10-0-131-171us-east-2computeinternal-debug ... To use host binaries, run `chroot /host` Pod IP: 10.0.131.171 If you don't see a command prompt, try pressing enter. fs.inotify.max_user_watches = 65536 Removing debug pod ... # kernel.pid_max = 1048576 - from override tuned profile # fs.inotify.max_user_watches = 65536 - from /etc/sysctl even this is changed in override tuned profile :)
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409