Hide Forgot
Description of problem * Verification and validation of an active tuned(8) profile is a two fold issue: 1) $ tuned-adm(8) verify - Does not seem to validate all configuration settings done by an active profile. Ex. while verifying 'realtime' profiles, tuned(8) calls respective 'script.sh verify' function via 'plugin_script.py' plugin. === $ sort -t'.' -k2 /tmp/tuned.log | awk -f tlog.awk |less [tuned.plugins.plugin_script:] calling script '/usr/lib/tuned/realtime/script.sh' with arguments '['verify']' calling script '/usr/lib/tuned/realtime-virtual-host/script.sh' with arguments '['verify']' verify: passed: '['/usr/lib/tuned/realtime/script.sh', '/usr/lib/tuned/realtime-virtual-host/script.sh']' === - But respective script.sh:verify() functions do not validate parameters set by start() functions. 2) $ tuned-adm(8) verify - Does not offer convenient user-interface/interaction for users to be able to see and confirm that tuned-adm(8) verify is working well. - It does not present verification results to a user properly. Verification results get logged to the /var/log/tnued/tuned.log file. === # tuned-adm verify Verification failed, current system settings differ from the preset profile. You can mostly fix this by restarting the Tuned daemon, e.g.: systemctl restart tuned or service tuned restart Sometimes (if some plugins like bootloader are used) a reboot may be required. See tuned log file ('/var/log/tuned/tuned.log') for details. === Version-Release number of selected component (if applicable): $ cat /etc/redhat-release Red Hat Enterprise Linux release 8.4 Beta (Ootpa) $ rpm -q tuned tuned-2.15.0-2.el8.noarch $ $ tuned --versio tuned 2.15.0 How reproducible: 100% Steps to Reproduce: 1. install tuned 2. activate realtime profile 3. run tuned-adm verify Expected results: - Make it easy for users to validate active profile settings with certainty.
Created attachment 1770610 [details] tuned-realtime-profiles-script-verify.patch === 1) $ tuned-adm(8) verify - Does not seem to validate all configuration settings done by an active profile. Ex. while verifying 'realtime' profiles, tuned(8) calls respective 'script.sh verify' function via 'plugin_script.py' plugin. === * This is a proposed patch to address this issue. * One limitation is a Boolean return value from verify() function fails to convey if all tests passed or all tests failed. There could be a case wherein some tests pass and some fail. * Another limitation is, output from these verify() functions does not get logged to the /var/log/tuned/tuned.log file.
Created attachment 1770614 [details] tuned-display-verification-results.patch === 2) $ tuned-adm(8) verify - Does not offer convenient user-interface/interaction for users to be able to see and confirm that tuned-adm(8) verify is working well. - It does not present verification results to a user properly. Verification results get logged to the /var/log/tnued/tuned.log file. === * This is a proposed patch to address above issue. * One limitation is it only reflects verification status for a section (ex. [sysctl]) It does not show results for each parameter verification as logged in the /var/log/tuned/tuned.log file.
Created attachment 1770618 [details] tlog.awk - script to parse tund.log file and display verification results === * One limitation is it only reflects verification status for a section (ex. [sysctl]) It does not show results for each parameter verification as logged in the /var/log/tuned/tuned.log file. === * This is a quick awk(1) script to parse '/var/log/tuned/tuned.log' file and better present the verifications result for each parameter. * $ sort -t'.' -k2 /tmp/tuned.log | awk -f tlog.awk [tuned.plugins.base:] verify: failed: 'kernel.numa_balancing' = 'None', expected '' verify: failed: 'net.core.busy_poll' = 'None', expected '50' verify: failed: 'net.core.busy_read' = 'None', expected '50' verify: passed: device cpu0: 'energy_perf_bias' = 'performance' verify: passed: device cpu10: 'energy_perf_bias' = 'performance' verify: passed: device cpu11: 'energy_perf_bias' = 'performance' ... [tuned.plugins.plugin_net:] unknown channels parameter(s): {'check_net_queue_count'} unknown channels parameter(s): {'check_net_queue_count'} unknown channels parameter(s): {'check_net_queue_count'} unknown channels parameter(s): {'check_net_queue_count'} unknown channels parameter(s): {'check_net_queue_count'} unknown channels parameter(s): {'check_net_queue_count'} ... [tuned.plugins.plugin_scheduler:] verify: failed: 'SMP affinity of IRQ 0' = '[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]', expected '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]' verify: failed: 'SMP affinity of IRQ 40' = '[5]', expected '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]' verify: failed: 'SMP affinity of IRQ 41' = '[7]', expected '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]' verify: failed: 'SMP affinity of IRQ 42' = '[9]', expected '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]' verify: failed: 'SMP affinity of IRQ 43' = '[11]', expected '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]' verify: passed: 'default IRQ SMP affinity' = '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]' verify: passed: 'SMP affinity of IRQ 100' = '[0, 2, 4, 6, 8, 10, 12, 14]' verify: passed: 'SMP affinity of IRQ 10' = '[0, 2, 4, 6, 8, 10, 12, 14]' verify: passed: 'SMP affinity of IRQ 101' = '[0, 2, 4, 6, 8, 10, 12, 14]' verify: passed: 'SMP affinity of IRQ 102' = '[0, 2, 4, 6, 8, 10, 12, 14]' ... [tuned.plugins.plugin_script:] calling script '/usr/lib/tuned/realtime/script.sh' with arguments '['verify']' calling script '/usr/lib/tuned/realtime-virtual-host/script.sh' with arguments '['verify']' verify: passed: '['/usr/lib/tuned/realtime/script.sh', '/usr/lib/tuned/realtime-virtual-host/script.sh']' [tuned.plugins.plugin_sysctl:] Failed to read sysctl parameter 'kernel.numa_balancing', the parameter does not exist Failed to read sysctl parameter 'net.core.busy_poll', the parameter does not exist Failed to read sysctl parameter 'net.core.busy_read', the parameter does not exist === * It is a quick PoC attempt, shall need more work.
(In reply to Prasad J Pandit from comment #1) > 1) $ tuned-adm(8) verify > - Does not seem to validate all configuration settings done by an active profile. > Ex. while verifying 'realtime' profiles, tuned(8) calls respective > 'script.sh verify' function via 'plugin_script.py' plugin. > === Raised a PR for this change -> https://github.com/redhat-performance/tuned/pull/341 @Jaroslav: will use this bug for this change alone. And open another Bug+PR for issue 2) above. Hope that's okay. Thank you.
(In reply to Prasad J Pandit from comment #4) > (In reply to Prasad J Pandit from comment #1) > > 1) $ tuned-adm(8) verify > > - Does not seem to validate all configuration settings done by an active profile. > > Ex. while verifying 'realtime' profiles, tuned(8) calls respective > > 'script.sh verify' function via 'plugin_script.py' plugin. > > === > > Raised a PR for this change > -> https://github.com/redhat-performance/tuned/pull/341 > > > @Jaroslav: will use this bug for this change alone. And open another Bug+PR > for issue 2) above. > Hope that's okay. > > Thank you. Thanks for PR, comments in the PR.
(In reply to Jaroslav Škarvada from comment #5) > Thanks for PR, comments in the PR. Updated the PR with the suggested change to check for old parameter value. Thank you.
Revised PR#345: RHBZ#1947858 profiles: improve realtime verify functions -> https://github.com/redhat-performance/tuned/pull/345
(In reply to Prasad J Pandit from comment #7) >> https://github.com/redhat-performance/tuned/pull/345 Merged upstream: -> https://github.com/redhat-performance/tuned/commit/6407ca44c6c83e66e5ce3b9ba921437d1787b59d
Jaroslav, We need this for 8.5 for sure. But since the intent of this work is to improve debugability for real-time KVM, I'd also like to request a backport for 8.4.z and even to 8.2.z if possible because those two releases are TUS (extended lifecycle support for telco). Can we do that?
(In reply to Luiz Capitulino from comment #9) > Jaroslav, > > We need this for 8.5 for sure. But since the intent of this work is to > improve debugability for real-time KVM, I'd also like to request a backport > for 8.4.z and even to 8.2.z if possible because those two releases are TUS > (extended lifecycle support for telco). Can we do that? I added RPL request for 8.5, despite of it the #345 (comment 7) should get in through the planned 8.5 TuneD rebase (it is already merged upstream). It's bugfix, so I think backport to z-stream shouldn't be problem.
Jaroslav, On further discussing this with the team, we believe it's better to have a full verification solution for the z-stream (which we are working on). So, no need to backport.
* Updates to script.sh/verify() functions and moving [sysfs] parameters to tuned.conf file is fairly small change. * I think this can be backported to earlier versions without waiting for or blocking on => BZ#1956387 tuned: add tuned-adm verify --verbose option. * I've opened/cloned rhel-8.4.z & rhel-8.2.z bugs for this. Hope that's okay. @Jaroslav ...wdyt? Thank you.
* Closed the manually cloned z-stream bugs => BZ#1965068, BZ#1965070.
(In reply to Prasad J Pandit from comment #12) > * Updates to script.sh/verify() functions and moving [sysfs] parameters > to tuned.conf file is fairly small change. > > * I think this can be backported to earlier versions without waiting for > or blocking on => BZ#1956387 tuned: add tuned-adm verify --verbose option. > > * I've opened/cloned rhel-8.4.z & rhel-8.2.z bugs for this. Hope that's okay. > > @Jaroslav ...wdyt? > > Thank you. Resolved through IRC. From devel point of view the backport shouldn't be problem, dev_ack from me.