RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1947858 - tuned: validate active profile settings
Summary: tuned: validate active profile settings
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: tuned
Version: 8.5
Hardware: All
OS: Linux
unspecified
medium
Target Milestone: beta
: 8.5
Assignee: Jaroslav Škarvada
QA Contact: Robin Hack
URL:
Whiteboard:
Depends On:
Blocks: 1932086 1965068 1965070
TreeView+ depends on / blocked
 
Reported: 2021-04-09 12:36 UTC by Prasad Pandit
Modified: 2021-11-10 10:05 UTC (History)
7 users (show)

Fixed In Version: tuned-2.16.0-0.1.rc1.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1965068 1965070 (view as bug list)
Environment:
Last Closed: 2021-11-09 19:58:24 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
tuned-realtime-profiles-script-verify.patch (3.21 KB, patch)
2021-04-09 12:53 UTC, Prasad Pandit 		no flags Details | Diff
tuned-display-verification-results.patch (3.17 KB, patch)
2021-04-09 13:18 UTC, Prasad Pandit 		no flags Details | Diff
tlog.awk - script to parse tund.log file and display verification results (955 bytes, application/x-awk)
2021-04-09 13:28 UTC, Prasad Pandit 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2021:4476 0 None None None 2021-11-09 19:58:50 UTC

Description Prasad Pandit 2021-04-09 12:36:23 UTC 
Description of problem

* Verification and validation of an active tuned(8) profile is a two fold issue:

  1) $ tuned-adm(8) verify 
    - Does not seem to validate all configuration settings done by an active profile.
      Ex. while verifying 'realtime' profiles, tuned(8) calls respective 'script.sh verify'
      function via 'plugin_script.py' plugin.
===
  $ sort -t'.' -k2 /tmp/tuned.log | awk -f tlog.awk |less
  [tuned.plugins.plugin_script:]
    calling script '/usr/lib/tuned/realtime/script.sh' with arguments '['verify']'
    calling script '/usr/lib/tuned/realtime-virtual-host/script.sh' with arguments '['verify']'
    verify: passed: '['/usr/lib/tuned/realtime/script.sh', '/usr/lib/tuned/realtime-virtual-host/script.sh']'
===
   - But respective script.sh:verify() functions do not validate parameters set by start() functions.

  2) $ tuned-adm(8) verify
    - Does not offer convenient user-interface/interaction for users to be able to see and confirm
      that tuned-adm(8) verify is working well.
    - It does not present verification results to a user properly.
      Verification results get logged to the /var/log/tnued/tuned.log file.
===
# tuned-adm verify
Verification failed, current system settings differ from the preset profile.
You can mostly fix this by restarting the Tuned daemon, e.g.:
  systemctl restart tuned
or
  service tuned restart
Sometimes (if some plugins like bootloader are used) a reboot may be required.
See tuned log file ('/var/log/tuned/tuned.log') for details.
===

Version-Release number of selected component (if applicable):

$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.4 Beta (Ootpa)
$ rpm -q tuned
tuned-2.15.0-2.el8.noarch
$ 
$ tuned --versio
tuned 2.15.0


How reproducible: 100%


Steps to Reproduce:
1. install tuned
2. activate realtime profile
3. run tuned-adm verify

Expected results:
  - Make it easy for users to validate active profile settings with certainty.
Comment 1 Prasad Pandit 2021-04-09 12:53:47 UTC 
Created attachment 1770610 [details]
tuned-realtime-profiles-script-verify.patch

===
1) $ tuned-adm(8) verify 
    - Does not seem to validate all configuration settings done by an active profile.
      Ex. while verifying 'realtime' profiles, tuned(8) calls respective 'script.sh verify'
      function via 'plugin_script.py' plugin.
===

* This is a proposed patch to address this issue.

* One limitation is a Boolean return value from verify() function fails to convey if all tests passed or all tests failed. There could be a case wherein some tests pass and some fail.

* Another limitation is, output from these verify() functions does not get logged to the /var/log/tuned/tuned.log file.
Comment 2 Prasad Pandit 2021-04-09 13:18:20 UTC 
Created attachment 1770614 [details]
tuned-display-verification-results.patch

===
 2) $ tuned-adm(8) verify
    - Does not offer convenient user-interface/interaction for users to be able to see and confirm
      that tuned-adm(8) verify is working well.
    - It does not present verification results to a user properly.
      Verification results get logged to the /var/log/tnued/tuned.log file.
===

* This is a proposed patch to address above issue.

* One limitation is it only reflects verification status for a section (ex. [sysctl])
  It does not show results for each parameter verification as logged in the /var/log/tuned/tuned.log file.
Comment 3 Prasad Pandit 2021-04-09 13:28:46 UTC 
Created attachment 1770618 [details]
tlog.awk - script to parse tund.log file and display verification results

===
* One limitation is it only reflects verification status for a section (ex. [sysctl])
  It does not show results for each parameter verification as logged in the /var/log/tuned/tuned.log file.
===

* This is a quick awk(1) script to parse '/var/log/tuned/tuned.log' file and better present the verifications result for each parameter.

* $ sort -t'.' -k2 /tmp/tuned.log | awk -f tlog.awk 
[tuned.plugins.base:]
  verify: failed: 'kernel.numa_balancing' = 'None', expected ''
  verify: failed: 'net.core.busy_poll' = 'None', expected '50'
  verify: failed: 'net.core.busy_read' = 'None', expected '50'
  verify: passed: device cpu0: 'energy_perf_bias' = 'performance'
  verify: passed: device cpu10: 'energy_perf_bias' = 'performance'
  verify: passed: device cpu11: 'energy_perf_bias' = 'performance'
...
[tuned.plugins.plugin_net:]
  unknown channels parameter(s): {'check_net_queue_count'}
  unknown channels parameter(s): {'check_net_queue_count'}
  unknown channels parameter(s): {'check_net_queue_count'}
  unknown channels parameter(s): {'check_net_queue_count'}
  unknown channels parameter(s): {'check_net_queue_count'}
  unknown channels parameter(s): {'check_net_queue_count'}
...
[tuned.plugins.plugin_scheduler:]
  verify: failed: 'SMP affinity of IRQ 0' = '[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]', expected '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]'
  verify: failed: 'SMP affinity of IRQ 40' = '[5]', expected '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]'
  verify: failed: 'SMP affinity of IRQ 41' = '[7]', expected '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]'
  verify: failed: 'SMP affinity of IRQ 42' = '[9]', expected '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]'
  verify: failed: 'SMP affinity of IRQ 43' = '[11]', expected '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]'
  verify: passed: 'default IRQ SMP affinity' = '[0, 1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15]'
  verify: passed: 'SMP affinity of IRQ 100' = '[0, 2, 4, 6, 8, 10, 12, 14]'
  verify: passed: 'SMP affinity of IRQ 10' = '[0, 2, 4, 6, 8, 10, 12, 14]'
  verify: passed: 'SMP affinity of IRQ 101' = '[0, 2, 4, 6, 8, 10, 12, 14]'
  verify: passed: 'SMP affinity of IRQ 102' = '[0, 2, 4, 6, 8, 10, 12, 14]'
...
[tuned.plugins.plugin_script:]
  calling script '/usr/lib/tuned/realtime/script.sh' with arguments '['verify']'
  calling script '/usr/lib/tuned/realtime-virtual-host/script.sh' with arguments '['verify']'
  verify: passed: '['/usr/lib/tuned/realtime/script.sh', '/usr/lib/tuned/realtime-virtual-host/script.sh']'
[tuned.plugins.plugin_sysctl:]
  Failed to read sysctl parameter 'kernel.numa_balancing', the parameter does not exist
  Failed to read sysctl parameter 'net.core.busy_poll', the parameter does not exist
  Failed to read sysctl parameter 'net.core.busy_read', the parameter does not exist
===

* It is a quick PoC attempt, shall need more work.
Comment 4 Prasad Pandit 2021-04-30 12:35:36 UTC 
(In reply to Prasad J Pandit from comment #1)
> 1) $ tuned-adm(8) verify 
>     - Does not seem to validate all configuration settings done by an active profile.
>       Ex. while verifying 'realtime' profiles, tuned(8) calls respective
>       'script.sh verify' function via 'plugin_script.py' plugin.
> ===

Raised a PR for this change
  -> https://github.com/redhat-performance/tuned/pull/341


@Jaroslav: will use this bug for this change alone. And open another Bug+PR for issue 2) above.
Hope that's okay.

Thank you.
Comment 5 Jaroslav Škarvada 2021-04-30 18:13:55 UTC 
(In reply to Prasad J Pandit from comment #4)
> (In reply to Prasad J Pandit from comment #1)
> > 1) $ tuned-adm(8) verify 
> >     - Does not seem to validate all configuration settings done by an active profile.
> >       Ex. while verifying 'realtime' profiles, tuned(8) calls respective
> >       'script.sh verify' function via 'plugin_script.py' plugin.
> > ===
> 
> Raised a PR for this change
>   -> https://github.com/redhat-performance/tuned/pull/341
> 
> 
> @Jaroslav: will use this bug for this change alone. And open another Bug+PR
> for issue 2) above.
> Hope that's okay.
> 
> Thank you.

Thanks for PR, comments in the PR.
Comment 6 Prasad Pandit 2021-05-03 10:07:10 UTC 
(In reply to Jaroslav Škarvada from comment #5)
> Thanks for PR, comments in the PR.

Updated the PR with the suggested change to check for old parameter value.

Thank you.
Comment 7 Prasad Pandit 2021-05-13 08:24:47 UTC 
Revised PR#345: RHBZ#1947858 profiles: improve realtime verify functions
 -> https://github.com/redhat-performance/tuned/pull/345
Comment 8 Prasad Pandit 2021-05-17 05:55:26 UTC 
(In reply to Prasad J Pandit from comment #7)
>> https://github.com/redhat-performance/tuned/pull/345

Merged upstream:
  -> https://github.com/redhat-performance/tuned/commit/6407ca44c6c83e66e5ce3b9ba921437d1787b59d
Comment 9 Luiz Capitulino 2021-05-19 00:57:40 UTC 
Jaroslav,

We need this for 8.5 for sure. But since the intent of this work is to improve debugability for real-time KVM, I'd also like to request a backport for 8.4.z and even to 8.2.z if possible because those two releases are TUS (extended lifecycle support for telco). Can we do that?
Comment 10 Jaroslav Škarvada 2021-05-19 11:45:49 UTC 
(In reply to Luiz Capitulino from comment #9)
> Jaroslav,
> 
> We need this for 8.5 for sure. But since the intent of this work is to
> improve debugability for real-time KVM, I'd also like to request a backport
> for 8.4.z and even to 8.2.z if possible because those two releases are TUS
> (extended lifecycle support for telco). Can we do that?

I added RPL request for 8.5, despite of it the #345 (comment 7) should get in through the planned 8.5 TuneD rebase (it is already merged upstream). It's bugfix, so I think backport to z-stream shouldn't be problem.
Comment 11 Luiz Capitulino 2021-05-20 20:48:00 UTC 
Jaroslav,

On further discussing this with the team, we believe it's better to have a full verification solution for the z-stream (which we are working on). So, no need to backport.
Comment 12 Prasad Pandit 2021-05-26 18:31:04 UTC 
* Updates to script.sh/verify() functions and moving [sysfs] parameters
  to tuned.conf file is fairly small change.

* I think this can be backported to earlier versions without waiting for
  or blocking on => BZ#1956387 tuned: add tuned-adm verify --verbose option.

* I've opened/cloned rhel-8.4.z & rhel-8.2.z bugs for this. Hope that's okay.

@Jaroslav ...wdyt?

Thank you.
Comment 13 Prasad Pandit 2021-05-27 15:01:23 UTC 
* Closed the manually cloned z-stream bugs => BZ#1965068, BZ#1965070.
Comment 14 Jaroslav Škarvada 2021-05-27 15:17:49 UTC 
(In reply to Prasad J Pandit from comment #12)
> * Updates to script.sh/verify() functions and moving [sysfs] parameters
>   to tuned.conf file is fairly small change.
> 
> * I think this can be backported to earlier versions without waiting for
>   or blocking on => BZ#1956387 tuned: add tuned-adm verify --verbose option.
> 
> * I've opened/cloned rhel-8.4.z & rhel-8.2.z bugs for this. Hope that's okay.
> 
> @Jaroslav ...wdyt?
> 
> Thank you.

Resolved through IRC.

From devel point of view the backport shouldn't be problem, dev_ack from me.
Comment 23 errata-xmlrpc 2021-11-09 19:58:24 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (tuned bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4476
Note You need to log in before you can comment on or make changes to this bug.