Bug 1956387

This bug is to address part 2) -> https://bugzilla.redhat.com/show_bug.cgi?id=1947858#c4

===
[root@virtlab500 tuned]# tuned-adm active
Current active profile: balanced
[root@virtlab500 tuned]# 

[root@virtlab500 tuned]# tuned-adm verify 
Verfication succeeded, current system settings match the preset profile.
See tuned log file ('/var/log/tuned/tuned.log') for details.
[root@virtlab500 tuned]# 

[root@virtlab500 tuned]# tuned-adm verify --verbose |less
[cpu]
 governor=conservative|powersave
 energy_perf_bias=normal
Verification: Pass

[sysctl]
 kernel.sched_min_granularity_ns=3000000
 kernel.sched_wakeup_granularity_ns=4000000
 vm.dirty_ratio=10
 vm.dirty_background_ratio=3
 vm.swappiness=10
 kernel.sched_migration_cost_ns=5000000
 net.core.busy_read=None, expected: 50
 net.core.busy_poll=None, expected: 50
 net.ipv4.tcp_fastopen=3
 kernel.numa_balancing=None, expected: 0
 kernel.hung_task_timeout_secs=600
 kernel.nmi_watchdog=0
 kernel.sched_rt_runtime_us=-1
 vm.stat_interval=10
 kernel.timer_migration=0
Verification: Fail

...
[sysfs]
 /sys/bus/workqueue/devices/writeback/cpumask=f55f
 /sys/devices/virtual/workqueue/cpumask=f55f
 /sys/devices/system/machinecheck/machinecheck*/ignore_ce=1
 /sys/devices/system/machinecheck/machinecheck*/ignore_ce=1
 ...
 /sys/devices/system/machinecheck/machinecheck*/ignore_ce=1
 /sys/devices/system/machinecheck/machinecheck*/ignore_ce=1
 /sys/kernel/mm/ksm/run=2
 /sys/kernel/ktimer_lockless_check=0, expected: 1
Verification: Fail

[modules]
 cpufreq_conservative: verify: failed: 'module 'cpufreq_conservative' is not loaded'
Verification: Fail

[audio]
 []
Verification: Pass

[video]
 radeon_powersave=dpm-balanced, auto
Verification: Pass

[disk]
 []
Verification: Pass

[scsi_host]
 alpm=medium_power
  host0=None
Verification: Fail

See tuned log file ('/var/log/tuned/tuned.log') for details.
===

@Jaroslav:
   current patch enables 'tuned-adm verify' to display output as above.
   before I raise a PR, just wanted to check if you've have any inputs/thoughts/suggestions?

Thank you.

Nice work, can you also share the output of tuned-adm verify without verbose?
Will it only report whether the verification has passed or failed?

Have you also looked into the scheduler verification?

(In reply to Prasad J Pandit from comment #2)
> ===
> [root@virtlab500 tuned]# tuned-adm active
> Current active profile: balanced
> [root@virtlab500 tuned]# 
> 
> [root@virtlab500 tuned]# tuned-adm verify 
> Verfication succeeded, current system settings match the preset profile.
> See tuned log file ('/var/log/tuned/tuned.log') for details.
> [root@virtlab500 tuned]# 
> 
> [root@virtlab500 tuned]# tuned-adm verify --verbose |less
> [cpu]
>  governor=conservative|powersave
>  energy_perf_bias=normal
> Verification: Pass
...

Looks good to me! Thanks.

(In reply to Nitesh Narayan Lal from comment #3)
> Nice work, can you also share the output of tuned-adm verify without verbose?
> Will it only report whether the verification has passed or failed?

Yes, above comment shows an output of the default 'tuned-adm verify' run.
There's no change to default behaviour.

> Have you also looked into the scheduler verification?

Yes, currently it looks as below

[root@virtlab500 tuned]# tuned-adm verify --verbose |less
...
[scheduler]
 ps_whitelist=None
 ps_blacklist=None, expected: ksoftirqd.*;rcuc.*;rcub.*;ktimersoftd.*;.*pmd.*;.*PMD.*;^DPDK;.*qemu-kvm.*
 default_irq_smp_affinity=None, expected: calc
 perf_process_fork=None, expected: false
 isolated_cores=None, expected: 5,7,9,11
Verification: Fail

Thank you.

(In reply to Prasad J Pandit from comment #5)
> (In reply to Nitesh Narayan Lal from comment #3)
> > Nice work, can you also share the output of tuned-adm verify without verbose?
> > Will it only report whether the verification has passed or failed?
> 
> Yes, above comment shows an output of the default 'tuned-adm verify' run.
> There's no change to default behaviour.
> 

Maybe we can improve this as well, for example, right now it shows:

# tuned-adm active
Current active profile: realtime-virtual-host

# tuned-adm verify
Verification failed, current system settings differ from the preset profile.
You can mostly fix this by restarting the Tuned daemon, e.g.:
  systemctl restart tuned
or
  service tuned restart
Sometimes (if some plugins like bootloader are used) a reboot may be required.
See tuned log file ('/var/log/tuned/tuned.log') for details.

The suggestion to restart tuned in itself is pretty vague IMHO.
Instead, it will be useful if we can print something like the number of errors
that we triggered during the verification here.

What do you think?

Thanks

(In reply to Nitesh Narayan Lal from comment #6)
> Maybe we can improve this as well, for example, right now it shows:
... 
> The suggestion to restart tuned in itself is pretty vague IMHO.
> Instead, it will be useful if we can print something like the number of
> errors that we triggered during the verification here.

* True, restarting tuned(8) does not seem an effective solution.
  Sometimes current system settings may differ because the system
  does not support certain option ex.

  => tuned.plugins.plugin_vm: Option 'transparent_hugepages' is not supported on current hardware.

OR if an error/exception occurs while setting some parameter ex.

  ...
   File "/usr/lib/python3.6/site-packages/tuned/plugins/plugin_net.py", line 400, in _set_device_parameters
     if context == "channels" and int(dev_params[next(iter(d))]) == 0:
  => ValueError: invalid literal for int() with base 10: 'n/a'

OR if the kernel does not define certain parameter

  => tuned.plugins.plugin_sysctl: Failed to read sysctl parameter 'kernel.numa_balancing', the parameter does not exist


In such situations, restarting tuned(8) isn't going to be helpful.


* I guess absent kernel parameter or system support error could be result of different kernel versions
  and/or system capabilities. They need to be addressed by tweaking tuned(8) profiles for each supported
  RHEL versions.

  I think it can be addressed as another new bug fix, instead of combining with this one.
  Because tweaking each profile parameter for different RHEL versions may take more time/testing.


...wdyt?

Thank you.

(In reply to Prasad J Pandit from comment #7)
> (In reply to Nitesh Narayan Lal from comment #6)
> > Maybe we can improve this as well, for example, right now it shows:
> ... 
> > The suggestion to restart tuned in itself is pretty vague IMHO.
> > Instead, it will be useful if we can print something like the number of
> > errors that we triggered during the verification here.
> 
> * True, restarting tuned(8) does not seem an effective solution.
>   Sometimes current system settings may differ because the system
>   does not support certain option ex.
> 
>   => tuned.plugins.plugin_vm: Option 'transparent_hugepages' is not
> supported on current hardware.

Exactly

> 
> OR if an error/exception occurs while setting some parameter ex.
> 
>   ...
>    File "/usr/lib/python3.6/site-packages/tuned/plugins/plugin_net.py", line
> 400, in _set_device_parameters
>      if context == "channels" and int(dev_params[next(iter(d))]) == 0:
>   => ValueError: invalid literal for int() with base 10: 'n/a'
> 


This is a bug reported in BZ 1943291:)
The fix is upstream but I don't think it has been backported yet.


> OR if the kernel does not define certain parameter
> 
>   => tuned.plugins.plugin_sysctl: Failed to read sysctl parameter
> 'kernel.numa_balancing', the parameter does not exist
> 
> 
> In such situations, restarting tuned(8) isn't going to be helpful.
> 

Right.

> 
> * I guess absent kernel parameter or system support error could be result of
> different kernel versions
>   and/or system capabilities. They need to be addressed by tweaking tuned(8)
> profiles for each supported
>   RHEL versions.
> 
>   I think it can be addressed as another new bug fix, instead of combining
> with this one.
>   Because tweaking each profile parameter for different RHEL versions may
> take more time/testing.
> 
> 
> ...wdyt?
> 
> Thank you.

Yeah, I agree to fix the individual plugins we should open a new BZ.
However, what we can do here (in your upstream PR) is just improve this
messaging that we get when verification fails without the verbose option.

Thanks

New output with and without --verbose option

===
[root@virtlab500 ~]# 
[root@virtlab500 ~]# tuned-adm verify
Active profile => realtime-virtual-host
Verification failed, current system settings differ from the preset profile.
 * Use 'tuned-adm verify --verbose' option to know differing system settings.
 * Sometimes (if some plugins like bootloader are used) a reboot may be required.
 * See tuned log file ('/var/log/tuned/tuned.log') for details.
===

[root@virtlab500 ~]# 
[root@virtlab500 ~]# tuned-adm verify -v |less
Active profile => realtime-virtual-host
[cpu]
 governor=performance
 energy_perf_bias=performance
Verification: Pass

[sysctl]
 kernel.sched_min_granularity_ns=3000000
 kernel.sched_wakeup_granularity_ns=4000000
 vm.dirty_ratio=10
 vm.dirty_background_ratio=3
 vm.swappiness=10
 kernel.sched_migration_cost_ns=5000000
 net.core.busy_read=None, [Expected: 50]
 net.core.busy_poll=None, [Expected: 50]
 net.ipv4.tcp_fastopen=3
 kernel.numa_balancing=None, [Expected: 0]
 kernel.hung_task_timeout_secs=600
 kernel.nmi_watchdog=0
 kernel.sched_rt_runtime_us=-1
 vm.stat_interval=10
 kernel.timer_migration=0
Verification: Fail

[sysfs]
 /sys/bus/workqueue/devices/writeback/cpumask=f55f
 /sys/devices/virtual/workqueue/cpumask=f55f
 /sys/devices/system/machinecheck/machinecheck*/ignore_ce=1
 /sys/devices/system/machinecheck/machinecheck*/ignore_ce=1
 /sys/devices/system/machinecheck/machinecheck*/ignore_ce=1
...
[irqbalance]
 banned_cpus=None, [Expected: 5,7,9,11]
Verification: Fail

[script]
 scripts: verify: failed: '['/usr/lib/tuned/realtime/script.sh', '/usr/lib/tuned/realtime-virtual-host/script.sh']'                                                                           
Verification: Fail

[scheduler]
 ps_whitelist=None
 ps_blacklist=None, [Expected: ksoftirqd.*;rcuc.*;rcub.*;ktimersoftd.*;.*pmd.*;.*PMD.*;^DPDK;.*qemu-kvm.*]                                                                                    
 default_irq_smp_affinity=None, [Expected: calc]
 perf_process_fork=None, [Expected: false]
 isolated_cores=None, [Expected: 5,7,9,11]
Verification: Fail

Verification failed, current system settings differ from the preset profile.
 * Sometimes (if some plugins like bootloader are used) a reboot may be required.
 * See tuned log file ('/var/log/tuned/tuned.log') for details.
===

Looks good, thanks.

PR#342 -> https://github.com/redhat-performance/tuned/pull/342

Revised PR#346: RHBZ#1956387 tuned-adm: add --verbose option to verify command
  -> https://github.com/redhat-performance/tuned/pull/346

Brew scratch build with above patch
  -> https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=36728362

It may help with the PR#346 review.

Revised new scratch build
  ->  https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=36942161

Revised new PR #350
  -> https://github.com/redhat-performance/tuned/pull/350

Latest scratch build
  -> https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=36984097

Revised scratch build with plugin_modules.py fix update

  -> https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=36991631

Retain the return (ret) value behaviour as before.

Revised PR#351
  -> https://github.com/redhat-performance/tuned/pull/351

Revised PR#351
  -> https://github.com/redhat-performance/tuned/pull/351

Scratch build
  -> https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=37513286

Rebased PR#351 to v2.16.0
  -> https://github.com/redhat-performance/tuned/pull/351

Scratch build
  -> https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=38991907

It's being worked on in the PR.

Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.