Bug 2213614
| Summary: | tuned sysctl plugin wrongly processes /etc/sysctl.conf twice therefore undoing setting previously applied in /etc/sysctl.d/99-tripleo.conf. | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | coldford <coldford> |
| Component: | tuned | Assignee: | Jaroslav Škarvada <jskarvad> |
| Status: | CLOSED MIGRATED | QA Contact: | Robin Hack <rhack> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 8.4 | CC: | cory.bannister, jeder, jorton, jskarvad |
| Target Milestone: | rc | Keywords: | MigratedToJIRA |
| Target Release: | --- | Flags: | pm-rhel:
mirror+
|
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-09-21 22:39:24 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Hello, I wanted to flag that this tuned bug likely also explains: - https://bugzilla.redhat.com/show_bug.cgi?id=2013341 It should be stressed that this bug definitely has the potential for production impacts. Cory Oldford Note: It also is still present in the upstream master: - See: https://github.com/redhat-performance/tuned/blob/master/tuned/plugins/plugin_sysctl.py#L123 The highlighted line should be removed. It seems that procps-ng sysctl binary has the same exact handling as tuned: $ sudo sysctl --system Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ... Applying /usr/lib/sysctl.d/50-coredump.conf ... Applying /usr/lib/sysctl.d/50-default.conf ... Applying /usr/lib/sysctl.d/50-libkcapi-optmem_max.conf ... Applying /usr/lib/sysctl.d/50-pid-max.conf ... Applying /usr/lib/sysctl.d/60-libvirtd.conf ... Applying /usr/lib/sysctl.d/60-qemu-postcopy-migration.conf ... Applying /etc/sysctl.d/99-sysctl.conf ... <--- SETTINGS FIRST APPLIED Applying /etc/sysctl.d/99-test.conf ... <--- CUSTOM SETTINGS REQUIRED Applying /etc/sysctl.conf ... <--- SETTINGS APPLIED AGAIN So this leads me to some questions/thoughts: - What was the intent behind tripleo choosing /etc/sysctl.d/99-tripleo.conf ? - Why the differing behaviours between systemd-udev and ( sysctl --system, and tuned )? - What is the actual expected behaviour? I agree that this systemd change resolves the long time problem and discussion when the /etc/systcl.conf should be read, but it resolved it the worst possible way - by dropping support for it. Unfortunately, AFAIK there is no standard for it, thus we followed the systctl tool as a standard. If systemd upstream insists on this behavior, which I think is not good and has potential to break things, we would have to add some workaround to TuneD, e.g. evaluate all symlinks and in case the symlink points to the /etc/systcl.conf then not finally read the /etc/sysctl.conf. I think this unnecessarily complicates things. Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug. This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there. Due to differences in account names between systems, some fields were not replicated. Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information. To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer. You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like: "Bugzilla Bug" = 1234567 In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information. |
Description of problem: Tuned sysctl plugin wrongly processes /etc/sysctl.conf twice therefore undoing settings previously applied in /etc/sysctl.d/99-tripleo.conf. Code: def _apply_system_sysctl(): files = {} for d in SYSCTL_CONFIG_DIRS: try: flist = os.listdir(d) except OSError: continue for fname in flist: if not fname.endswith(".conf"): continue if fname not in files: files[fname] = d for fname in sorted(files.keys()): d = files[fname] path = "%s/%s" % (d, fname) _apply_sysctl_config_file(path) <----- SYSCTL_CONFIG_DIRS = [ "/run/sysctl.d","/etc/sysctl.d" ] _apply_sysctl_config_file("/etc/sysctl.conf") <----- And last /etc/sysctl.conf which was already applied via symlink /etc/sysctl.d/99-sysctl.conf Sample: /etc/sysctl.d from a RHEL 8.4 host $ ls -al /etc/sysctl.d/ total 28 drwxrwxrwx. 2 yank yank 4096 Jun 1 07:54 . drwxrwxrwx. 72 yank yank 4096 Jun 8 03:47 .. -rw-rw-rw-. 1 yank yank 525 Jan 14 2022 50-libreswan.conf -rw-rw-rw-. 1 yank yank 50 Jun 1 07:54 60-kernel.core_pattern.conf -rw-rw-rw-. 1 yank yank 143 Jun 1 07:54 60-tf-node-init.conf lrwxrwxrwx. 1 yank yank 14 Jun 7 2021 99-sysctl.conf -> ../sysctl.conf -rw-rw-rw-. 1 yank yank 1378 Jun 1 07:54 99-tripleo.conf <----- Settings are overridden when sysctl.conf is wrongly applied the second time. A review of the systemd-sysctl.service handling and changelogs reveals what should likely be the correct approach: From: systemd-239/NEWS * The systemd-sysctl tool no longer natively reads the file /etc/sysctl.conf. If desired, the file should be symlinked from /etc/sysctl.d/99-sysctl.conf. Apart from providing legacy support by a symlink rather than built-in code, it also makes the otherwise hidden order of application of the different files visible. (Note that this partly reverts to a pre-198 application order of sysctl knobs!) Which matches the code: r = conf_files_list_strv(&files, ".conf", NULL, 0, (const char**) CONF_PATHS_STRV("sysctl.d")); if (r < 0) { log_error_errno(r, "Failed to enumerate sysctl.d files: %m"); goto finish; } Version-Release number of selected component (if applicable): - tuned-2.18.0-3.1.20220714git70732a57.el8fdp.noarch How reproducible: - Everytime tuned is run with reapply_sysctl = 1 Steps to Reproduce: 1. set vm.nr_hugepages=1024 in /etc/sysctl.conf 2. set vm.nr_hugepages=0 in /etc/sysctl.d/99-tripleo.conf 3. apply with restart of systemd-sysctl.service 4. check sysctl -a | grep hugepages to confirm correctly set to 0 5. restart tuned service 4. check sysctl -a | grep hugepages to confirm incorrectly set to 1024 Actual results: - Settings that are in 99-tripleo.conf get incorrectly overridden Expected results: - Settings match what would be applied by systemd-sysctl.service. Additional info: - This could be impacting production workloads. - This is a hot topic for some RHOSP consulting work and needs to be addressed.