+++ This bug was initially created as a clone of Bug #1947858 +++

Description of problem

* Verification and validation of an active tuned(8) profile is a two fold issue:

  1) $ tuned-adm(8) verify 
    - Does not seem to validate all configuration settings done by an active profile.
      Ex. while verifying 'realtime' profiles, tuned(8) calls respective 'script.sh verify'
      function via 'plugin_script.py' plugin.
===
  $ sort -t'.' -k2 /tmp/tuned.log | awk -f tlog.awk |less
  [tuned.plugins.plugin_script:]
    calling script '/usr/lib/tuned/realtime/script.sh' with arguments '['verify']'
    calling script '/usr/lib/tuned/realtime-virtual-host/script.sh' with arguments '['verify']'
    verify: passed: '['/usr/lib/tuned/realtime/script.sh', '/usr/lib/tuned/realtime-virtual-host/script.sh']'
===
   - But respective script.sh:verify() functions do not validate parameters set by start() functions.

  2) $ tuned-adm(8) verify
    - Does not offer convenient user-interface/interaction for users to be able to see and confirm
      that tuned-adm(8) verify is working well.
    - It does not present verification results to a user properly.
      Verification results get logged to the /var/log/tnued/tuned.log file.
===
# tuned-adm verify
Verification failed, current system settings differ from the preset profile.
You can mostly fix this by restarting the Tuned daemon, e.g.:
  systemctl restart tuned
or
  service tuned restart
Sometimes (if some plugins like bootloader are used) a reboot may be required.
See tuned log file ('/var/log/tuned/tuned.log') for details.
===

Version-Release number of selected component (if applicable):

$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.4 Beta (Ootpa)
$ rpm -q tuned
tuned-2.15.0-2.el8.noarch
$ 
$ tuned --versio
tuned 2.15.0


How reproducible: 100%


Steps to Reproduce:
1. install tuned
2. activate realtime profile
3. run tuned-adm verify

Expected results:
  - Make it easy for users to validate active profile settings with certainty.

--- Additional comment from Prasad J Pandit on 2021-04-09 12:53:47 UTC ---

===
1) $ tuned-adm(8) verify 
    - Does not seem to validate all configuration settings done by an active profile.
      Ex. while verifying 'realtime' profiles, tuned(8) calls respective 'script.sh verify'
      function via 'plugin_script.py' plugin.
===

* This is a proposed patch to address this issue.

* One limitation is a Boolean return value from verify() function fails to convey if all tests passed or all tests failed. There could be a case wherein some tests pass and some fail.

* Another limitation is, output from these verify() functions does not get logged to the /var/log/tuned/tuned.log file.

--- Additional comment from Prasad J Pandit on 2021-04-09 13:18:20 UTC ---

===
 2) $ tuned-adm(8) verify
    - Does not offer convenient user-interface/interaction for users to be able to see and confirm
      that tuned-adm(8) verify is working well.
    - It does not present verification results to a user properly.
      Verification results get logged to the /var/log/tnued/tuned.log file.
===

* This is a proposed patch to address above issue.

* One limitation is it only reflects verification status for a section (ex. [sysctl])
  It does not show results for each parameter verification as logged in the /var/log/tuned/tuned.log file.

--- Additional comment from Prasad J Pandit on 2021-04-09 13:28:46 UTC ---

===
* One limitation is it only reflects verification status for a section (ex. [sysctl])
  It does not show results for each parameter verification as logged in the /var/log/tuned/tuned.log file.
===

* This is a quick awk(1) script to parse '/var/log/tuned/tuned.log' file and better present the verifications result for each parameter.

* $ sort -t'.' -k2 /tmp/tuned.log | awk -f tlog.awk 
[tuned.plugins.base:]
  verify: failed: 'kernel.numa_balancing' = 'None', expected ''
  verify: failed: 'net.core.busy_poll' = 'None', expected '50'
  verify: failed: 'net.core.busy_read' = 'None', expected '50'
  verify: passed: device cpu0: 'energy_perf_bias' = 'performance'
  verify: passed: device cpu10: 'energy_perf_bias' = 'performance'
  verify: passed: device cpu11: 'energy_perf_bias' = 'performance'
...
[tuned.plugins.plugin_net:]
  unknown channels parameter(s): {'check_net_queue_count'}
  unknown channels parameter(s): {'check_net_queue_count'}
  unknown channels parameter(s): {'check_net_queue_count'}
  unknown channels parameter(s): {'check_net_queue_count'}
  unknown channels parameter(s): {'check_net_queue_count'}
  unknown channels parameter(s): {'check_net_queue_count'}
...
[tuned.plugins.plugin_scheduler:]
  verify: failed: 'SMP affinity of IRQ 0' = '[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]', expected '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]'
  verify: failed: 'SMP affinity of IRQ 40' = '[5]', expected '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]'
  verify: failed: 'SMP affinity of IRQ 41' = '[7]', expected '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]'
  verify: failed: 'SMP affinity of IRQ 42' = '[9]', expected '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]'
  verify: failed: 'SMP affinity of IRQ 43' = '[11]', expected '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]'
  verify: passed: 'default IRQ SMP affinity' = '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]'
  verify: passed: 'SMP affinity of IRQ 100' = '[0, 2, 4, 6, 8, 10, 12, 14]'
  verify: passed: 'SMP affinity of IRQ 10' = '[0, 2, 4, 6, 8, 10, 12, 14]'
  verify: passed: 'SMP affinity of IRQ 101' = '[0, 2, 4, 6, 8, 10, 12, 14]'
  verify: passed: 'SMP affinity of IRQ 102' = '[0, 2, 4, 6, 8, 10, 12, 14]'
...
[tuned.plugins.plugin_script:]
  calling script '/usr/lib/tuned/realtime/script.sh' with arguments '['verify']'
  calling script '/usr/lib/tuned/realtime-virtual-host/script.sh' with arguments '['verify']'
  verify: passed: '['/usr/lib/tuned/realtime/script.sh', '/usr/lib/tuned/realtime-virtual-host/script.sh']'
[tuned.plugins.plugin_sysctl:]
  Failed to read sysctl parameter 'kernel.numa_balancing', the parameter does not exist
  Failed to read sysctl parameter 'net.core.busy_poll', the parameter does not exist
  Failed to read sysctl parameter 'net.core.busy_read', the parameter does not exist
===

* It is a quick PoC attempt, shall need more work.

--- Additional comment from Prasad J Pandit on 2021-04-30 12:35:36 UTC ---

(In reply to Prasad J Pandit from comment #1)
> 1) $ tuned-adm(8) verify 
>     - Does not seem to validate all configuration settings done by an active profile.
>       Ex. while verifying 'realtime' profiles, tuned(8) calls respective
>       'script.sh verify' function via 'plugin_script.py' plugin.
> ===

Raised a PR for this change
  -> https://github.com/redhat-performance/tuned/pull/341


@Jaroslav: will use this bug for this change alone. And open another Bug+PR for issue 2) above.
Hope that's okay.

Thank you.

--- Additional comment from Jaroslav Škarvada on 2021-04-30 18:13:55 UTC ---

(In reply to Prasad J Pandit from comment #4)
> (In reply to Prasad J Pandit from comment #1)
> > 1) $ tuned-adm(8) verify 
> >     - Does not seem to validate all configuration settings done by an active profile.
> >       Ex. while verifying 'realtime' profiles, tuned(8) calls respective
> >       'script.sh verify' function via 'plugin_script.py' plugin.
> > ===
> 
> Raised a PR for this change
>   -> https://github.com/redhat-performance/tuned/pull/341
> 
> 
> @Jaroslav: will use this bug for this change alone. And open another Bug+PR
> for issue 2) above.
> Hope that's okay.
> 
> Thank you.

Thanks for PR, comments in the PR.

--- Additional comment from Prasad J Pandit on 2021-05-03 10:07:10 UTC ---

(In reply to Jaroslav Škarvada from comment #5)
> Thanks for PR, comments in the PR.

Updated the PR with the suggested change to check for old parameter value.

Thank you.

--- Additional comment from Prasad J Pandit on 2021-05-13 08:24:47 UTC ---

Revised PR#345: RHBZ#1947858 profiles: improve realtime verify functions
 -> https://github.com/redhat-performance/tuned/pull/345

--- Additional comment from Prasad J Pandit on 2021-05-17 05:55:26 UTC ---

(In reply to Prasad J Pandit from comment #7)
>> https://github.com/redhat-performance/tuned/pull/345

Merged upstream:
  -> https://github.com/redhat-performance/tuned/commit/6407ca44c6c83e66e5ce3b9ba921437d1787b59d

--- Additional comment from Luiz Capitulino on 2021-05-19 00:57:40 UTC ---

Jaroslav,

We need this for 8.5 for sure. But since the intent of this work is to improve debugability for real-time KVM, I'd also like to request a backport for 8.4.z and even to 8.2.z if possible because those two releases are TUS (extended lifecycle support for telco). Can we do that?

--- Additional comment from Jaroslav Škarvada on 2021-05-19 11:45:49 UTC ---

(In reply to Luiz Capitulino from comment #9)
> Jaroslav,
> 
> We need this for 8.5 for sure. But since the intent of this work is to
> improve debugability for real-time KVM, I'd also like to request a backport
> for 8.4.z and even to 8.2.z if possible because those two releases are TUS
> (extended lifecycle support for telco). Can we do that?

I added RPL request for 8.5, despite of it the #345 (comment 7) should get in through the planned 8.5 TuneD rebase (it is already merged upstream). It's bugfix, so I think backport to z-stream shouldn't be problem.