Description of problem:

Tuned sysctl plugin wrongly processes /etc/sysctl.conf twice therefore undoing settings previously applied in /etc/sysctl.d/99-tripleo.conf.

Code:

def _apply_system_sysctl():
        files = {}
        for d in SYSCTL_CONFIG_DIRS:                                              
                try:
                        flist = os.listdir(d)
                except OSError:
                        continue
                for fname in flist:
                        if not fname.endswith(".conf"):
                                continue
                        if fname not in files:
                                files[fname] = d

        for fname in sorted(files.keys()):            
                d = files[fname]
                path = "%s/%s" % (d, fname)
                _apply_sysctl_config_file(path)                                  <----- SYSCTL_CONFIG_DIRS = [ "/run/sysctl.d","/etc/sysctl.d" ]
        _apply_sysctl_config_file("/etc/sysctl.conf")                            <----- And last /etc/sysctl.conf which was already applied via symlink /etc/sysctl.d/99-sysctl.conf


Sample: /etc/sysctl.d from a RHEL 8.4 host
$ ls -al /etc/sysctl.d/
total 28
drwxrwxrwx.  2 yank yank 4096 Jun  1 07:54 .
drwxrwxrwx. 72 yank yank 4096 Jun  8 03:47 ..
-rw-rw-rw-.  1 yank yank  525 Jan 14  2022 50-libreswan.conf
-rw-rw-rw-.  1 yank yank   50 Jun  1 07:54 60-kernel.core_pattern.conf
-rw-rw-rw-.  1 yank yank  143 Jun  1 07:54 60-tf-node-init.conf
lrwxrwxrwx.  1 yank yank   14 Jun  7  2021 99-sysctl.conf -> ../sysctl.conf      
-rw-rw-rw-.  1 yank yank 1378 Jun  1 07:54 99-tripleo.conf                       <----- Settings are overridden when sysctl.conf is wrongly applied the second time.

A review of the systemd-sysctl.service handling and changelogs reveals what should likely be the correct approach:

From: systemd-239/NEWS

        * The systemd-sysctl tool no longer natively reads the file
          /etc/sysctl.conf. If desired, the file should be symlinked
          from /etc/sysctl.d/99-sysctl.conf. Apart from providing
          legacy support by a symlink rather than built-in code, it
          also makes the otherwise hidden order of application of the
          different files visible. (Note that this partly reverts to a
          pre-198 application order of sysctl knobs!)

Which matches the code:

              r = conf_files_list_strv(&files, ".conf", NULL, 0, (const char**) CONF_PATHS_STRV("sysctl.d"));
                if (r < 0) {
                        log_error_errno(r, "Failed to enumerate sysctl.d files: %m");
                        goto finish;
                }


Version-Release number of selected component (if applicable):
- tuned-2.18.0-3.1.20220714git70732a57.el8fdp.noarch

How reproducible:
- Everytime tuned is run with reapply_sysctl = 1

Steps to Reproduce:
1. set vm.nr_hugepages=1024 in /etc/sysctl.conf
2. set vm.nr_hugepages=0 in /etc/sysctl.d/99-tripleo.conf
3. apply with restart of systemd-sysctl.service
4. check sysctl -a | grep hugepages to confirm correctly set to 0
5. restart tuned service
4. check sysctl -a | grep hugepages to confirm incorrectly set to 1024

Actual results:
- Settings that are in 99-tripleo.conf get incorrectly overridden

Expected results:
- Settings match what would be applied by systemd-sysctl.service.

Additional info:
- This could be impacting production workloads.
- This is a hot topic for some RHOSP consulting work and needs to be addressed.