Bug 1947858 - tuned: validate active profile settings
Summary: tuned: validate active profile settings
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: tuned
Version: 8.5
Hardware: All
OS: Linux
unspecified
medium
Target Milestone: beta
: 8.5
Assignee: Jaroslav Škarvada
QA Contact: Robin Hack
URL:
Whiteboard:
Depends On:
Blocks: 1932086 1965068 1965070
TreeView+ depends on / blocked
 
Reported: 2021-04-09 12:36 UTC by Prasad J Pandit
Modified: 2021-11-10 10:05 UTC (History)
7 users (show)

Fixed In Version: tuned-2.16.0-0.1.rc1.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1965068 1965070 (view as bug list)
Environment:
Last Closed: 2021-11-09 19:58:24 UTC
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)
tuned-realtime-profiles-script-verify.patch (3.21 KB, patch)
2021-04-09 12:53 UTC, Prasad J Pandit
no flags Details | Diff
tuned-display-verification-results.patch (3.17 KB, patch)
2021-04-09 13:18 UTC, Prasad J Pandit
no flags Details | Diff
tlog.awk - script to parse tund.log file and display verification results (955 bytes, application/x-awk)
2021-04-09 13:28 UTC, Prasad J Pandit
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2021:4476 0 None None None 2021-11-09 19:58:50 UTC

Description Prasad J Pandit 2021-04-09 12:36:23 UTC
Description of problem

* Verification and validation of an active tuned(8) profile is a two fold issue:

  1) $ tuned-adm(8) verify 
    - Does not seem to validate all configuration settings done by an active profile.
      Ex. while verifying 'realtime' profiles, tuned(8) calls respective 'script.sh verify'
      function via 'plugin_script.py' plugin.
===
  $ sort -t'.' -k2 /tmp/tuned.log | awk -f tlog.awk |less
  [tuned.plugins.plugin_script:]
    calling script '/usr/lib/tuned/realtime/script.sh' with arguments '['verify']'
    calling script '/usr/lib/tuned/realtime-virtual-host/script.sh' with arguments '['verify']'
    verify: passed: '['/usr/lib/tuned/realtime/script.sh', '/usr/lib/tuned/realtime-virtual-host/script.sh']'
===
   - But respective script.sh:verify() functions do not validate parameters set by start() functions.

  2) $ tuned-adm(8) verify
    - Does not offer convenient user-interface/interaction for users to be able to see and confirm
      that tuned-adm(8) verify is working well.
    - It does not present verification results to a user properly.
      Verification results get logged to the /var/log/tnued/tuned.log file.
===
# tuned-adm verify
Verification failed, current system settings differ from the preset profile.
You can mostly fix this by restarting the Tuned daemon, e.g.:
  systemctl restart tuned
or
  service tuned restart
Sometimes (if some plugins like bootloader are used) a reboot may be required.
See tuned log file ('/var/log/tuned/tuned.log') for details.
===

Version-Release number of selected component (if applicable):

$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.4 Beta (Ootpa)
$ rpm -q tuned
tuned-2.15.0-2.el8.noarch
$ 
$ tuned --versio
tuned 2.15.0


How reproducible: 100%


Steps to Reproduce:
1. install tuned
2. activate realtime profile
3. run tuned-adm verify

Expected results:
  - Make it easy for users to validate active profile settings with certainty.

Comment 1 Prasad J Pandit 2021-04-09 12:53:47 UTC
Created attachment 1770610 [details]
tuned-realtime-profiles-script-verify.patch

===
1) $ tuned-adm(8) verify 
    - Does not seem to validate all configuration settings done by an active profile.
      Ex. while verifying 'realtime' profiles, tuned(8) calls respective 'script.sh verify'
      function via 'plugin_script.py' plugin.
===

* This is a proposed patch to address this issue.

* One limitation is a Boolean return value from verify() function fails to convey if all tests passed or all tests failed. There could be a case wherein some tests pass and some fail.

* Another limitation is, output from these verify() functions does not get logged to the /var/log/tuned/tuned.log file.

Comment 2 Prasad J Pandit 2021-04-09 13:18:20 UTC
Created attachment 1770614 [details]
tuned-display-verification-results.patch

===
 2) $ tuned-adm(8) verify
    - Does not offer convenient user-interface/interaction for users to be able to see and confirm
      that tuned-adm(8) verify is working well.
    - It does not present verification results to a user properly.
      Verification results get logged to the /var/log/tnued/tuned.log file.
===

* This is a proposed patch to address above issue.

* One limitation is it only reflects verification status for a section (ex. [sysctl])
  It does not show results for each parameter verification as logged in the /var/log/tuned/tuned.log file.

Comment 3 Prasad J Pandit 2021-04-09 13:28:46 UTC
Created attachment 1770618 [details]
tlog.awk - script to parse tund.log file and display verification results

===
* One limitation is it only reflects verification status for a section (ex. [sysctl])
  It does not show results for each parameter verification as logged in the /var/log/tuned/tuned.log file.
===

* This is a quick awk(1) script to parse '/var/log/tuned/tuned.log' file and better present the verifications result for each parameter.

* $ sort -t'.' -k2 /tmp/tuned.log | awk -f tlog.awk 
[tuned.plugins.base:]
  verify: failed: 'kernel.numa_balancing' = 'None', expected ''
  verify: failed: 'net.core.busy_poll' = 'None', expected '50'
  verify: failed: 'net.core.busy_read' = 'None', expected '50'
  verify: passed: device cpu0: 'energy_perf_bias' = 'performance'
  verify: passed: device cpu10: 'energy_perf_bias' = 'performance'
  verify: passed: device cpu11: 'energy_perf_bias' = 'performance'
...
[tuned.plugins.plugin_net:]
  unknown channels parameter(s): {'check_net_queue_count'}
  unknown channels parameter(s): {'check_net_queue_count'}
  unknown channels parameter(s): {'check_net_queue_count'}
  unknown channels parameter(s): {'check_net_queue_count'}
  unknown channels parameter(s): {'check_net_queue_count'}
  unknown channels parameter(s): {'check_net_queue_count'}
...
[tuned.plugins.plugin_scheduler:]
  verify: failed: 'SMP affinity of IRQ 0' = '[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]', expected '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]'
  verify: failed: 'SMP affinity of IRQ 40' = '[5]', expected '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]'
  verify: failed: 'SMP affinity of IRQ 41' = '[7]', expected '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]'
  verify: failed: 'SMP affinity of IRQ 42' = '[9]', expected '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]'
  verify: failed: 'SMP affinity of IRQ 43' = '[11]', expected '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]'
  verify: passed: 'default IRQ SMP affinity' = '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]'
  verify: passed: 'SMP affinity of IRQ 100' = '[0, 2, 4, 6, 8, 10, 12, 14]'
  verify: passed: 'SMP affinity of IRQ 10' = '[0, 2, 4, 6, 8, 10, 12, 14]'
  verify: passed: 'SMP affinity of IRQ 101' = '[0, 2, 4, 6, 8, 10, 12, 14]'
  verify: passed: 'SMP affinity of IRQ 102' = '[0, 2, 4, 6, 8, 10, 12, 14]'
...
[tuned.plugins.plugin_script:]
  calling script '/usr/lib/tuned/realtime/script.sh' with arguments '['verify']'
  calling script '/usr/lib/tuned/realtime-virtual-host/script.sh' with arguments '['verify']'
  verify: passed: '['/usr/lib/tuned/realtime/script.sh', '/usr/lib/tuned/realtime-virtual-host/script.sh']'
[tuned.plugins.plugin_sysctl:]
  Failed to read sysctl parameter 'kernel.numa_balancing', the parameter does not exist
  Failed to read sysctl parameter 'net.core.busy_poll', the parameter does not exist
  Failed to read sysctl parameter 'net.core.busy_read', the parameter does not exist
===

* It is a quick PoC attempt, shall need more work.

Comment 4 Prasad J Pandit 2021-04-30 12:35:36 UTC
(In reply to Prasad J Pandit from comment #1)
> 1) $ tuned-adm(8) verify 
>     - Does not seem to validate all configuration settings done by an active profile.
>       Ex. while verifying 'realtime' profiles, tuned(8) calls respective
>       'script.sh verify' function via 'plugin_script.py' plugin.
> ===

Raised a PR for this change
  -> https://github.com/redhat-performance/tuned/pull/341


@Jaroslav: will use this bug for this change alone. And open another Bug+PR for issue 2) above.
Hope that's okay.

Thank you.

Comment 5 Jaroslav Škarvada 2021-04-30 18:13:55 UTC
(In reply to Prasad J Pandit from comment #4)
> (In reply to Prasad J Pandit from comment #1)
> > 1) $ tuned-adm(8) verify 
> >     - Does not seem to validate all configuration settings done by an active profile.
> >       Ex. while verifying 'realtime' profiles, tuned(8) calls respective
> >       'script.sh verify' function via 'plugin_script.py' plugin.
> > ===
> 
> Raised a PR for this change
>   -> https://github.com/redhat-performance/tuned/pull/341
> 
> 
> @Jaroslav: will use this bug for this change alone. And open another Bug+PR
> for issue 2) above.
> Hope that's okay.
> 
> Thank you.

Thanks for PR, comments in the PR.

Comment 6 Prasad J Pandit 2021-05-03 10:07:10 UTC
(In reply to Jaroslav Škarvada from comment #5)
> Thanks for PR, comments in the PR.

Updated the PR with the suggested change to check for old parameter value.

Thank you.

Comment 7 Prasad J Pandit 2021-05-13 08:24:47 UTC
Revised PR#345: RHBZ#1947858 profiles: improve realtime verify functions
 -> https://github.com/redhat-performance/tuned/pull/345

Comment 9 Luiz Capitulino 2021-05-19 00:57:40 UTC
Jaroslav,

We need this for 8.5 for sure. But since the intent of this work is to improve debugability for real-time KVM, I'd also like to request a backport for 8.4.z and even to 8.2.z if possible because those two releases are TUS (extended lifecycle support for telco). Can we do that?

Comment 10 Jaroslav Škarvada 2021-05-19 11:45:49 UTC
(In reply to Luiz Capitulino from comment #9)
> Jaroslav,
> 
> We need this for 8.5 for sure. But since the intent of this work is to
> improve debugability for real-time KVM, I'd also like to request a backport
> for 8.4.z and even to 8.2.z if possible because those two releases are TUS
> (extended lifecycle support for telco). Can we do that?

I added RPL request for 8.5, despite of it the #345 (comment 7) should get in through the planned 8.5 TuneD rebase (it is already merged upstream). It's bugfix, so I think backport to z-stream shouldn't be problem.

Comment 11 Luiz Capitulino 2021-05-20 20:48:00 UTC
Jaroslav,

On further discussing this with the team, we believe it's better to have a full verification solution for the z-stream (which we are working on). So, no need to backport.

Comment 12 Prasad J Pandit 2021-05-26 18:31:04 UTC
* Updates to script.sh/verify() functions and moving [sysfs] parameters
  to tuned.conf file is fairly small change.

* I think this can be backported to earlier versions without waiting for
  or blocking on => BZ#1956387 tuned: add tuned-adm verify --verbose option.

* I've opened/cloned rhel-8.4.z & rhel-8.2.z bugs for this. Hope that's okay.

@Jaroslav ...wdyt?

Thank you.

Comment 13 Prasad J Pandit 2021-05-27 15:01:23 UTC
* Closed the manually cloned z-stream bugs => BZ#1965068, BZ#1965070.

Comment 14 Jaroslav Škarvada 2021-05-27 15:17:49 UTC
(In reply to Prasad J Pandit from comment #12)
> * Updates to script.sh/verify() functions and moving [sysfs] parameters
>   to tuned.conf file is fairly small change.
> 
> * I think this can be backported to earlier versions without waiting for
>   or blocking on => BZ#1956387 tuned: add tuned-adm verify --verbose option.
> 
> * I've opened/cloned rhel-8.4.z & rhel-8.2.z bugs for this. Hope that's okay.
> 
> @Jaroslav ...wdyt?
> 
> Thank you.

Resolved through IRC.

From devel point of view the backport shouldn't be problem, dev_ack from me.

Comment 23 errata-xmlrpc 2021-11-09 19:58:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (tuned bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4476


Note You need to log in before you can comment on or make changes to this bug.