RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1252153 - verification of realtime-virtual-host profile fails
Summary: verification of realtime-virtual-host profile fails
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: tuned
Version: 7.2
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: rc
: ---
Assignee: Jaroslav Škarvada
QA Contact: Tereza Cerna
URL:
Whiteboard:
Depends On:
Blocks: 1282565
TreeView+ depends on / blocked
 
Reported: 2015-08-10 20:19 UTC by Jeff Nelson
Modified: 2016-11-04 07:24 UTC (History)
7 users (show)

Fixed In Version: tuned-2.5.1-5.el7
Doc Type: Bug Fix
Doc Text:
Cause: Previously the 'tuned-adm verify' command, verifying whether the current system settings matches the requested Tuned profile, counted missing sysctl settings as failure. Consequence: Verification of the realtime profile used together with the realtime kernel could fail. Also verification of any custom profile requested setting of sysctl value which is not supported on the platform could fail. Also from the Tuned output it wasn't apparent why it failed. Fix: Tuned code was improved. Result: Now the missing sysctl values (which are not supported by the platform) are ingored during the verification and not causing the verification to fail (warning is logged to the log). Also in the Tuned output, there is now included the location of the log file with more detailed information.
Clone Of:
: 1282565 (view as bug list)
Environment:
Last Closed: 2016-11-04 07:24:00 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
tuned logfile (58.12 KB, text/plain)
2015-08-10 20:19 UTC, Jeff Nelson 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:2479 0 normal SHIPPED_LIVE tuned bug fix and enhancement update 2016-11-03 14:08:42 UTC

Description Jeff Nelson 2015-08-10 20:19:02 UTC 
Created attachment 1061236 [details]
tuned logfile

Description of problem:
Verification of the active profile, realtime-virtual-host, fails.

Version-Release number of selected component (if applicable):
# yum list installed | egrep tuned
tuned.noarch                          2.5.1-1.el7             @beaker-Server/7.2
tuned-profiles-nfv.noarch             2.5.1-1.el7             @/tuned-profiles-nfv-2.5.1-1.el7.noarch
tuned-profiles-realtime.noarch        2.5.1-1.el7             @beaker-Server-RT 


How reproducible:
100/100

Steps to Reproduce:
1. tuned-adm profile realtime-virtual-host
2. tuned-adm verify


Actual results:
Verification failed, current system settings differ from the preset profile.
See tuned.log for details. You can mostly fix this by Tuned restart, e.g.:
  service tuned restart


Expected results:
Verification works.

Additional info:
1. It is not obvious from error message that tuned.log is actually /var/log/tuned/tuned.log

2. tuned.log contains two ERRORS which appear to be responsible for the failure:

2015-08-10 15:55:07,232 ERROR    tuned.utils.commands: Executing /sbin/sysctl error: sysctl: cannot stat /proc/sys/kernel/numa_balancing: No such file or directory

and

2015-08-10 15:56:25,122 ERROR    tuned.plugins.base: verify: failed: kernel.numa_balancing = None, expected 0

3. additional details
# uname -a
Linux localhost.localdomain 3.10.0-302.rt56.174.el7.x86_64 #1 SMP PREEMPT RT Mon Aug 3 15:01:33 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-3.10.0-302.rt56.174.el7.x86_64 root=/dev/mapper/rhel_virtlab420-root ro crashkernel=auto rd.lvm.lv=rhel_virtlab420/root rd.lvm.lv=rhel_virtlab420/swap console=ttyS1,115200 default_hugepagesz=1G isolcpus=1,3,5,7,9,11,13,14,15 nohz_full=1,3,5,7,9,11,13,14,15 intel_pstate=disable nosoftlockup
# lscpu | grep NUMA
NUMA node(s):          2
NUMA node0 CPU(s):     0,2,4,6,8,10,12,14
NUMA node1 CPU(s):     1,3,5,7,9,11,13,15
Comment 4 Jaroslav Škarvada 2015-08-20 12:48:31 UTC 
(In reply to Jeff Nelson from comment #0)
> Additional info:
> 1. It is not obvious from error message that tuned.log is actually
> /var/log/tuned/tuned.log
>
Good catch, fixed by the following upstream commit:
https://git.fedorahosted.org/cgit/tuned.git/commit/?id=5a74141d79152654eeaa22441bfe772085e05103

> 2. tuned.log contains two ERRORS which appear to be responsible for the
> failure:
> 
> 2015-08-10 15:55:07,232 ERROR    tuned.utils.commands: Executing
> /sbin/sysctl error: sysctl: cannot stat /proc/sys/kernel/numa_balancing: No
> such file or directory
> 
> and
> 
> 2015-08-10 15:56:25,122 ERROR    tuned.plugins.base: verify: failed:
> kernel.numa_balancing = None, expected 0
> 
None is there because it is unavailable on the system. It's logically correct, because profile requires it, but it's unavailable, so the verification fails.

We could handle it as 'soft failures' as Jeremy pointed out (there was proposed =? operator in tuned profile to mark such settings), or there is also proposal for --ignore-missing tuned-adm command line paramater (bug 1243807) which will treat all missing/unavailable settings as passed.

I think it doesn't qualify as release blocker, thus moving to RHEL-7.3.
Comment 6 Jaroslav Škarvada 2015-11-11 13:41:07 UTC 
Short term solution is to shut this specific error message off, long term is to implement --ignore-missing option or markings for 'soft failures'.
Comment 8 Jaroslav Škarvada 2015-11-16 17:27:19 UTC 
(In reply to Jaroslav Škarvada from comment #6)
> Short term solution is to shut this specific error message off, long term is
> to implement --ignore-missing option or markings for 'soft failures'.

Short term fix - verification passes for sysctl plugin even if the option doesn't exist (is unsupported by the system). Upstream commit:
https://git.fedorahosted.org/cgit/tuned.git/commit/?id=d49469f7114e1e9f761235a48eb2f4cbdade3f0f
Comment 12 Jeff Nelson 2016-09-15 22:21:54 UTC 
Tereza writes in comment 11
>...
>I think this is example of right behavior. Right? Do you think so?

No, I don't think this is right.

Comment 8 says:
>Short term fix - verification passes for sysctl plugin even if the option >doesn't exist (is unsupported by the system).

What's the difference between the old and new behavior of the command 'tuned-adm verify'? Both print:

>Verification failed, current system settings differ from the preset profile.

If the option doesn't exist it was supposed to be skipped, yet the verification still fails - the printed message says so. The change in the content of the logfile (from ERROR to INFO) does not appear to make any difference.
Comment 14 Jaroslav Škarvada 2016-09-16 07:57:06 UTC 
(In reply to Jeff Nelson from comment #12)
> Tereza writes in comment 11
> >...
> >I think this is example of right behavior. Right? Do you think so?
> 
> No, I don't think this is right.
> 
> Comment 8 says:
> >Short term fix - verification passes for sysctl plugin even if the option >doesn't exist (is unsupported by the system).
> 
> What's the difference between the old and new behavior of the command
> 'tuned-adm verify'? Both print:
> 
> >Verification failed, current system settings differ from the preset profile.
> 
> If the option doesn't exist it was supposed to be skipped, yet the
> verification still fails - the printed message says so. The change in the
> content of the logfile (from ERROR to INFO) does not appear to make any
> difference.

No, it shouldn't fail, there should be only INFO in the log.
Comment 15 Jaroslav Škarvada 2016-09-16 07:58:06 UTC 
(In reply to Jaroslav Škarvada from comment #14)
> (In reply to Jeff Nelson from comment #12)
> > Tereza writes in comment 11
> > >...
> > >I think this is example of right behavior. Right? Do you think so?
> > 
> > No, I don't think this is right.
> > 
> > Comment 8 says:
> > >Short term fix - verification passes for sysctl plugin even if the option >doesn't exist (is unsupported by the system).
> > 
> > What's the difference between the old and new behavior of the command
> > 'tuned-adm verify'? Both print:
> > 
> > >Verification failed, current system settings differ from the preset profile.
> > 
> > If the option doesn't exist it was supposed to be skipped, yet the
> > verification still fails - the printed message says so. The change in the
> > content of the logfile (from ERROR to INFO) does not appear to make any
> > difference.
> 
> No, it shouldn't fail, there should be only INFO in the log.

Probably there are different errors in the test run.
Comment 16 Tereza Cerna 2016-09-16 08:55:31 UTC 
Sorry, ERRORs were caused by my inattention.

=========================================
Verified in:
    tuned-2.7.1-3.el7.noarch
    tuned-profiles-nfv-2.7.1-3.el7.noarch
PASS
=========================================

# tuned-adm profile realtime-virtual-host
# reboot
# tuned-adm verify
Verfication succeeded, current system settings match the preset profile.
See tuned log file ('/var/log/tuned/tuned.log') for details.
# grep numa /var/log/tuned/tuned.log 
2016-09-16 04:53:12,677 INFO     tuned.plugins.base: verify: skipped, missing: 'kernel.numa_balancing'
#

=========================================
Reproduced in:
    tuned-2.5.1-3.el7.noarch
    tuned-profiles-nfv-2.5.1-3.el7.noarch
FAIL
=========================================

# tuned-adm profile realtime-virtual-host
# reboot
# tuned-adm verify
Verification failed, current system settings differ from the preset profile.
See tuned.log for details. You can mostly fix this by Tuned restart, e.g.:
  service tuned restart
# grep numa /var/log/tuned/tuned.log 
2016-09-16 04:37:12,413 ERROR    tuned.plugins.base: verify: failed: kernel.numa_balancing = None, expected 0
#
Comment 18 errata-xmlrpc 2016-11-04 07:24:00 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2479.html
Note You need to log in before you can comment on or make changes to this bug.