Bug 1252153 - verification of realtime-virtual-host profile fails
verification of realtime-virtual-host profile fails
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: tuned (Show other bugs)
7.2
Unspecified Unspecified
high Severity unspecified
: rc
: ---
Assigned To: Jaroslav Škarvada
Tereza Cerna
: ZStream
Depends On:
Blocks: 1282565
  Show dependency treegraph
 
Reported: 2015-08-10 16:19 EDT by Jeff Nelson
Modified: 2016-11-04 03:24 EDT (History)
7 users (show)

See Also:
Fixed In Version: tuned-2.5.1-5.el7
Doc Type: Bug Fix
Doc Text:
Cause: Previously the 'tuned-adm verify' command, verifying whether the current system settings matches the requested Tuned profile, counted missing sysctl settings as failure. Consequence: Verification of the realtime profile used together with the realtime kernel could fail. Also verification of any custom profile requested setting of sysctl value which is not supported on the platform could fail. Also from the Tuned output it wasn't apparent why it failed. Fix: Tuned code was improved. Result: Now the missing sysctl values (which are not supported by the platform) are ingored during the verification and not causing the verification to fail (warning is logged to the log). Also in the Tuned output, there is now included the location of the log file with more detailed information.
Story Points: ---
Clone Of:
: 1282565 (view as bug list)
Environment:
Last Closed: 2016-11-04 03:24:00 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
tuned logfile (58.12 KB, text/plain)
2015-08-10 16:19 EDT, Jeff Nelson
no flags Details

  None (edit)
Description Jeff Nelson 2015-08-10 16:19:02 EDT
Created attachment 1061236 [details]
tuned logfile

Description of problem:
Verification of the active profile, realtime-virtual-host, fails.

Version-Release number of selected component (if applicable):
# yum list installed | egrep tuned
tuned.noarch                          2.5.1-1.el7             @beaker-Server/7.2
tuned-profiles-nfv.noarch             2.5.1-1.el7             @/tuned-profiles-nfv-2.5.1-1.el7.noarch
tuned-profiles-realtime.noarch        2.5.1-1.el7             @beaker-Server-RT 


How reproducible:
100/100

Steps to Reproduce:
1. tuned-adm profile realtime-virtual-host
2. tuned-adm verify


Actual results:
Verification failed, current system settings differ from the preset profile.
See tuned.log for details. You can mostly fix this by Tuned restart, e.g.:
  service tuned restart


Expected results:
Verification works.

Additional info:
1. It is not obvious from error message that tuned.log is actually /var/log/tuned/tuned.log

2. tuned.log contains two ERRORS which appear to be responsible for the failure:

2015-08-10 15:55:07,232 ERROR    tuned.utils.commands: Executing /sbin/sysctl error: sysctl: cannot stat /proc/sys/kernel/numa_balancing: No such file or directory

and

2015-08-10 15:56:25,122 ERROR    tuned.plugins.base: verify: failed: kernel.numa_balancing = None, expected 0

3. additional details
# uname -a
Linux localhost.localdomain 3.10.0-302.rt56.174.el7.x86_64 #1 SMP PREEMPT RT Mon Aug 3 15:01:33 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-3.10.0-302.rt56.174.el7.x86_64 root=/dev/mapper/rhel_virtlab420-root ro crashkernel=auto rd.lvm.lv=rhel_virtlab420/root rd.lvm.lv=rhel_virtlab420/swap console=ttyS1,115200 default_hugepagesz=1G isolcpus=1,3,5,7,9,11,13,14,15 nohz_full=1,3,5,7,9,11,13,14,15 intel_pstate=disable nosoftlockup
# lscpu | grep NUMA
NUMA node(s):          2
NUMA node0 CPU(s):     0,2,4,6,8,10,12,14
NUMA node1 CPU(s):     1,3,5,7,9,11,13,15
Comment 4 Jaroslav Škarvada 2015-08-20 08:48:31 EDT
(In reply to Jeff Nelson from comment #0)
> Additional info:
> 1. It is not obvious from error message that tuned.log is actually
> /var/log/tuned/tuned.log
>
Good catch, fixed by the following upstream commit:
https://git.fedorahosted.org/cgit/tuned.git/commit/?id=5a74141d79152654eeaa22441bfe772085e05103

> 2. tuned.log contains two ERRORS which appear to be responsible for the
> failure:
> 
> 2015-08-10 15:55:07,232 ERROR    tuned.utils.commands: Executing
> /sbin/sysctl error: sysctl: cannot stat /proc/sys/kernel/numa_balancing: No
> such file or directory
> 
> and
> 
> 2015-08-10 15:56:25,122 ERROR    tuned.plugins.base: verify: failed:
> kernel.numa_balancing = None, expected 0
> 
None is there because it is unavailable on the system. It's logically correct, because profile requires it, but it's unavailable, so the verification fails.

We could handle it as 'soft failures' as Jeremy pointed out (there was proposed =? operator in tuned profile to mark such settings), or there is also proposal for --ignore-missing tuned-adm command line paramater (bug 1243807) which will treat all missing/unavailable settings as passed.

I think it doesn't qualify as release blocker, thus moving to RHEL-7.3.
Comment 6 Jaroslav Škarvada 2015-11-11 08:41:07 EST
Short term solution is to shut this specific error message off, long term is to implement --ignore-missing option or markings for 'soft failures'.
Comment 8 Jaroslav Škarvada 2015-11-16 12:27:19 EST
(In reply to Jaroslav Škarvada from comment #6)
> Short term solution is to shut this specific error message off, long term is
> to implement --ignore-missing option or markings for 'soft failures'.

Short term fix - verification passes for sysctl plugin even if the option doesn't exist (is unsupported by the system). Upstream commit:
https://git.fedorahosted.org/cgit/tuned.git/commit/?id=d49469f7114e1e9f761235a48eb2f4cbdade3f0f
Comment 12 Jeff Nelson 2016-09-15 18:21:54 EDT
Tereza writes in comment 11
>...
>I think this is example of right behavior. Right? Do you think so?

No, I don't think this is right.

Comment 8 says:
>Short term fix - verification passes for sysctl plugin even if the option >doesn't exist (is unsupported by the system).

What's the difference between the old and new behavior of the command 'tuned-adm verify'? Both print:

>Verification failed, current system settings differ from the preset profile.

If the option doesn't exist it was supposed to be skipped, yet the verification still fails - the printed message says so. The change in the content of the logfile (from ERROR to INFO) does not appear to make any difference.
Comment 14 Jaroslav Škarvada 2016-09-16 03:57:06 EDT
(In reply to Jeff Nelson from comment #12)
> Tereza writes in comment 11
> >...
> >I think this is example of right behavior. Right? Do you think so?
> 
> No, I don't think this is right.
> 
> Comment 8 says:
> >Short term fix - verification passes for sysctl plugin even if the option >doesn't exist (is unsupported by the system).
> 
> What's the difference between the old and new behavior of the command
> 'tuned-adm verify'? Both print:
> 
> >Verification failed, current system settings differ from the preset profile.
> 
> If the option doesn't exist it was supposed to be skipped, yet the
> verification still fails - the printed message says so. The change in the
> content of the logfile (from ERROR to INFO) does not appear to make any
> difference.

No, it shouldn't fail, there should be only INFO in the log.
Comment 15 Jaroslav Škarvada 2016-09-16 03:58:06 EDT
(In reply to Jaroslav Škarvada from comment #14)
> (In reply to Jeff Nelson from comment #12)
> > Tereza writes in comment 11
> > >...
> > >I think this is example of right behavior. Right? Do you think so?
> > 
> > No, I don't think this is right.
> > 
> > Comment 8 says:
> > >Short term fix - verification passes for sysctl plugin even if the option >doesn't exist (is unsupported by the system).
> > 
> > What's the difference between the old and new behavior of the command
> > 'tuned-adm verify'? Both print:
> > 
> > >Verification failed, current system settings differ from the preset profile.
> > 
> > If the option doesn't exist it was supposed to be skipped, yet the
> > verification still fails - the printed message says so. The change in the
> > content of the logfile (from ERROR to INFO) does not appear to make any
> > difference.
> 
> No, it shouldn't fail, there should be only INFO in the log.

Probably there are different errors in the test run.
Comment 16 Tereza Cerna 2016-09-16 04:55:31 EDT
Sorry, ERRORs were caused by my inattention.

=========================================
Verified in:
    tuned-2.7.1-3.el7.noarch
    tuned-profiles-nfv-2.7.1-3.el7.noarch
PASS
=========================================

# tuned-adm profile realtime-virtual-host
# reboot
# tuned-adm verify
Verfication succeeded, current system settings match the preset profile.
See tuned log file ('/var/log/tuned/tuned.log') for details.
# grep numa /var/log/tuned/tuned.log 
2016-09-16 04:53:12,677 INFO     tuned.plugins.base: verify: skipped, missing: 'kernel.numa_balancing'
#

=========================================
Reproduced in:
    tuned-2.5.1-3.el7.noarch
    tuned-profiles-nfv-2.5.1-3.el7.noarch
FAIL
=========================================

# tuned-adm profile realtime-virtual-host
# reboot
# tuned-adm verify
Verification failed, current system settings differ from the preset profile.
See tuned.log for details. You can mostly fix this by Tuned restart, e.g.:
  service tuned restart
# grep numa /var/log/tuned/tuned.log 
2016-09-16 04:37:12,413 ERROR    tuned.plugins.base: verify: failed: kernel.numa_balancing = None, expected 0
#
Comment 18 errata-xmlrpc 2016-11-04 03:24:00 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2479.html

Note You need to log in before you can comment on or make changes to this bug.