Bug 1947858 - tuned: validate active profile settings
Summary: tuned: validate active profile settings
Keywords:
Status: ASSIGNED
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: tuned
Version: 8.5
Hardware: All
OS: Linux
unspecified
medium
Target Milestone: beta
: 8.5
Assignee: Jaroslav Škarvada
QA Contact: rhel-cs-infra-services-qe
URL:
Whiteboard:
Depends On:
Blocks: 1932086 1965068 1965070
TreeView+ depends on / blocked
 
Reported: 2021-04-09 12:36 UTC by Prasad J Pandit
Modified: 2021-05-27 15:17 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1965068 1965070 (view as bug list)
Environment:
Last Closed:
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)
tuned-realtime-profiles-script-verify.patch (3.21 KB, patch)
2021-04-09 12:53 UTC, Prasad J Pandit
no flags Details | Diff
tuned-display-verification-results.patch (3.17 KB, patch)
2021-04-09 13:18 UTC, Prasad J Pandit
no flags Details | Diff
tlog.awk - script to parse tund.log file and display verification results (955 bytes, application/x-awk)
2021-04-09 13:28 UTC, Prasad J Pandit
no flags Details

Description Prasad J Pandit 2021-04-09 12:36:23 UTC
Description of problem

* Verification and validation of an active tuned(8) profile is a two fold issue:

  1) $ tuned-adm(8) verify 
    - Does not seem to validate all configuration settings done by an active profile.
      Ex. while verifying 'realtime' profiles, tuned(8) calls respective 'script.sh verify'
      function via 'plugin_script.py' plugin.
===
  $ sort -t'.' -k2 /tmp/tuned.log | awk -f tlog.awk |less
  [tuned.plugins.plugin_script:]
    calling script '/usr/lib/tuned/realtime/script.sh' with arguments '['verify']'
    calling script '/usr/lib/tuned/realtime-virtual-host/script.sh' with arguments '['verify']'
    verify: passed: '['/usr/lib/tuned/realtime/script.sh', '/usr/lib/tuned/realtime-virtual-host/script.sh']'
===
   - But respective script.sh:verify() functions do not validate parameters set by start() functions.

  2) $ tuned-adm(8) verify
    - Does not offer convenient user-interface/interaction for users to be able to see and confirm
      that tuned-adm(8) verify is working well.
    - It does not present verification results to a user properly.
      Verification results get logged to the /var/log/tnued/tuned.log file.
===
# tuned-adm verify
Verification failed, current system settings differ from the preset profile.
You can mostly fix this by restarting the Tuned daemon, e.g.:
  systemctl restart tuned
or
  service tuned restart
Sometimes (if some plugins like bootloader are used) a reboot may be required.
See tuned log file ('/var/log/tuned/tuned.log') for details.
===

Version-Release number of selected component (if applicable):

$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.4 Beta (Ootpa)
$ rpm -q tuned
tuned-2.15.0-2.el8.noarch
$ 
$ tuned --versio
tuned 2.15.0


How reproducible: 100%


Steps to Reproduce:
1. install tuned
2. activate realtime profile
3. run tuned-adm verify

Expected results:
  - Make it easy for users to validate active profile settings with certainty.

Comment 1 Prasad J Pandit 2021-04-09 12:53:47 UTC
Created attachment 1770610 [details]
tuned-realtime-profiles-script-verify.patch

===
1) $ tuned-adm(8) verify 
    - Does not seem to validate all configuration settings done by an active profile.
      Ex. while verifying 'realtime' profiles, tuned(8) calls respective 'script.sh verify'
      function via 'plugin_script.py' plugin.
===

* This is a proposed patch to address this issue.

* One limitation is a Boolean return value from verify() function fails to convey if all tests passed or all tests failed. There could be a case wherein some tests pass and some fail.

* Another limitation is, output from these verify() functions does not get logged to the /var/log/tuned/tuned.log file.

Comment 2 Prasad J Pandit 2021-04-09 13:18:20 UTC
Created attachment 1770614 [details]
tuned-display-verification-results.patch

===
 2) $ tuned-adm(8) verify
    - Does not offer convenient user-interface/interaction for users to be able to see and confirm
      that tuned-adm(8) verify is working well.
    - It does not present verification results to a user properly.
      Verification results get logged to the /var/log/tnued/tuned.log file.
===

* This is a proposed patch to address above issue.

* One limitation is it only reflects verification status for a section (ex. [sysctl])
  It does not show results for each parameter verification as logged in the /var/log/tuned/tuned.log file.

Comment 3 Prasad J Pandit 2021-04-09 13:28:46 UTC
Created attachment 1770618 [details]
tlog.awk - script to parse tund.log file and display verification results

===
* One limitation is it only reflects verification status for a section (ex. [sysctl])
  It does not show results for each parameter verification as logged in the /var/log/tuned/tuned.log file.
===

* This is a quick awk(1) script to parse '/var/log/tuned/tuned.log' file and better present the verifications result for each parameter.

* $ sort -t'.' -k2 /tmp/tuned.log | awk -f tlog.awk 
[tuned.plugins.base:]
  verify: failed: 'kernel.numa_balancing' = 'None', expected ''
  verify: failed: 'net.core.busy_poll' = 'None', expected '50'
  verify: failed: 'net.core.busy_read' = 'None', expected '50'
  verify: passed: device cpu0: 'energy_perf_bias' = 'performance'
  verify: passed: device cpu10: 'energy_perf_bias' = 'performance'
  verify: passed: device cpu11: 'energy_perf_bias' = 'performance'
...
[tuned.plugins.plugin_net:]
  unknown channels parameter(s): {'check_net_queue_count'}
  unknown channels parameter(s): {'check_net_queue_count'}
  unknown channels parameter(s): {'check_net_queue_count'}
  unknown channels parameter(s): {'check_net_queue_count'}
  unknown channels parameter(s): {'check_net_queue_count'}
  unknown channels parameter(s): {'check_net_queue_count'}
...
[tuned.plugins.plugin_scheduler:]
  verify: failed: 'SMP affinity of IRQ 0' = '[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]', expected '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]'
  verify: failed: 'SMP affinity of IRQ 40' = '[5]', expected '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]'
  verify: failed: 'SMP affinity of IRQ 41' = '[7]', expected '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]'
  verify: failed: 'SMP affinity of IRQ 42' = '[9]', expected '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]'
  verify: failed: 'SMP affinity of IRQ 43' = '[11]', expected '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]'
  verify: passed: 'default IRQ SMP affinity' = '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]'
  verify: passed: 'SMP affinity of IRQ 100' = '[0, 2, 4, 6, 8, 10, 12, 14]'
  verify: passed: 'SMP affinity of IRQ 10' = '[0, 2, 4, 6, 8, 10, 12, 14]'
  verify: passed: 'SMP affinity of IRQ 101' = '[0, 2, 4, 6, 8, 10, 12, 14]'
  verify: passed: 'SMP affinity of IRQ 102' = '[0, 2, 4, 6, 8, 10, 12, 14]'
...
[tuned.plugins.plugin_script:]
  calling script '/usr/lib/tuned/realtime/script.sh' with arguments '['verify']'
  calling script '/usr/lib/tuned/realtime-virtual-host/script.sh' with arguments '['verify']'
  verify: passed: '['/usr/lib/tuned/realtime/script.sh', '/usr/lib/tuned/realtime-virtual-host/script.sh']'
[tuned.plugins.plugin_sysctl:]
  Failed to read sysctl parameter 'kernel.numa_balancing', the parameter does not exist
  Failed to read sysctl parameter 'net.core.busy_poll', the parameter does not exist
  Failed to read sysctl parameter 'net.core.busy_read', the parameter does not exist
===

* It is a quick PoC attempt, shall need more work.

Comment 4 Prasad J Pandit 2021-04-30 12:35:36 UTC
(In reply to Prasad J Pandit from comment #1)
> 1) $ tuned-adm(8) verify 
>     - Does not seem to validate all configuration settings done by an active profile.
>       Ex. while verifying 'realtime' profiles, tuned(8) calls respective
>       'script.sh verify' function via 'plugin_script.py' plugin.
> ===

Raised a PR for this change
  -> https://github.com/redhat-performance/tuned/pull/341


@Jaroslav: will use this bug for this change alone. And open another Bug+PR for issue 2) above.
Hope that's okay.

Thank you.

Comment 5 Jaroslav Škarvada 2021-04-30 18:13:55 UTC
(In reply to Prasad J Pandit from comment #4)
> (In reply to Prasad J Pandit from comment #1)
> > 1) $ tuned-adm(8) verify 
> >     - Does not seem to validate all configuration settings done by an active profile.
> >       Ex. while verifying 'realtime' profiles, tuned(8) calls respective
> >       'script.sh verify' function via 'plugin_script.py' plugin.
> > ===
> 
> Raised a PR for this change
>   -> https://github.com/redhat-performance/tuned/pull/341
> 
> 
> @Jaroslav: will use this bug for this change alone. And open another Bug+PR
> for issue 2) above.
> Hope that's okay.
> 
> Thank you.

Thanks for PR, comments in the PR.

Comment 6 Prasad J Pandit 2021-05-03 10:07:10 UTC
(In reply to Jaroslav Škarvada from comment #5)
> Thanks for PR, comments in the PR.

Updated the PR with the suggested change to check for old parameter value.

Thank you.

Comment 7 Prasad J Pandit 2021-05-13 08:24:47 UTC
Revised PR#345: RHBZ#1947858 profiles: improve realtime verify functions
 -> https://github.com/redhat-performance/tuned/pull/345

Comment 9 Luiz Capitulino 2021-05-19 00:57:40 UTC
Jaroslav,

We need this for 8.5 for sure. But since the intent of this work is to improve debugability for real-time KVM, I'd also like to request a backport for 8.4.z and even to 8.2.z if possible because those two releases are TUS (extended lifecycle support for telco). Can we do that?

Comment 10 Jaroslav Škarvada 2021-05-19 11:45:49 UTC
(In reply to Luiz Capitulino from comment #9)
> Jaroslav,
> 
> We need this for 8.5 for sure. But since the intent of this work is to
> improve debugability for real-time KVM, I'd also like to request a backport
> for 8.4.z and even to 8.2.z if possible because those two releases are TUS
> (extended lifecycle support for telco). Can we do that?

I added RPL request for 8.5, despite of it the #345 (comment 7) should get in through the planned 8.5 TuneD rebase (it is already merged upstream). It's bugfix, so I think backport to z-stream shouldn't be problem.

Comment 11 Luiz Capitulino 2021-05-20 20:48:00 UTC
Jaroslav,

On further discussing this with the team, we believe it's better to have a full verification solution for the z-stream (which we are working on). So, no need to backport.

Comment 12 Prasad J Pandit 2021-05-26 18:31:04 UTC
* Updates to script.sh/verify() functions and moving [sysfs] parameters
  to tuned.conf file is fairly small change.

* I think this can be backported to earlier versions without waiting for
  or blocking on => BZ#1956387 tuned: add tuned-adm verify --verbose option.

* I've opened/cloned rhel-8.4.z & rhel-8.2.z bugs for this. Hope that's okay.

@Jaroslav ...wdyt?

Thank you.

Comment 13 Prasad J Pandit 2021-05-27 15:01:23 UTC
* Closed the manually cloned z-stream bugs => BZ#1965068, BZ#1965070.

Comment 14 Jaroslav Škarvada 2021-05-27 15:17:49 UTC
(In reply to Prasad J Pandit from comment #12)
> * Updates to script.sh/verify() functions and moving [sysfs] parameters
>   to tuned.conf file is fairly small change.
> 
> * I think this can be backported to earlier versions without waiting for
>   or blocking on => BZ#1956387 tuned: add tuned-adm verify --verbose option.
> 
> * I've opened/cloned rhel-8.4.z & rhel-8.2.z bugs for this. Hope that's okay.
> 
> @Jaroslav ...wdyt?
> 
> Thank you.

Resolved through IRC.

From devel point of view the backport shouldn't be problem, dev_ack from me.


Note You need to log in before you can comment on or make changes to this bug.