Bug 1252153
Summary: | verification of realtime-virtual-host profile fails | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Jeff Nelson <jen> | ||||
Component: | tuned | Assignee: | Jaroslav Škarvada <jskarvad> | ||||
Status: | CLOSED ERRATA | QA Contact: | Tereza Cerna <tcerna> | ||||
Severity: | unspecified | Docs Contact: | |||||
Priority: | high | ||||||
Version: | 7.2 | CC: | bhu, jeder, jen, jscotka, jskarvad, tcerna, tlavigne | ||||
Target Milestone: | rc | Keywords: | ZStream | ||||
Target Release: | --- | ||||||
Hardware: | Unspecified | ||||||
OS: | Unspecified | ||||||
Whiteboard: | |||||||
Fixed In Version: | tuned-2.5.1-5.el7 | Doc Type: | Bug Fix | ||||
Doc Text: |
Cause:
Previously the 'tuned-adm verify' command, verifying whether the current system settings matches the requested Tuned profile, counted missing sysctl settings as failure.
Consequence:
Verification of the realtime profile used together with the realtime kernel could fail. Also verification of any custom profile requested setting of sysctl value which is not supported on the platform could fail. Also from the Tuned output it wasn't apparent why it failed.
Fix:
Tuned code was improved.
Result:
Now the missing sysctl values (which are not supported by the platform) are ingored during the verification and not causing the verification to fail (warning is logged to the log). Also in the Tuned output, there is now included the location of the log file with more detailed information.
|
Story Points: | --- | ||||
Clone Of: | |||||||
: | 1282565 (view as bug list) | Environment: | |||||
Last Closed: | 2016-11-04 07:24:00 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Bug Depends On: | |||||||
Bug Blocks: | 1282565 | ||||||
Attachments: |
|
(In reply to Jeff Nelson from comment #0) > Additional info: > 1. It is not obvious from error message that tuned.log is actually > /var/log/tuned/tuned.log > Good catch, fixed by the following upstream commit: https://git.fedorahosted.org/cgit/tuned.git/commit/?id=5a74141d79152654eeaa22441bfe772085e05103 > 2. tuned.log contains two ERRORS which appear to be responsible for the > failure: > > 2015-08-10 15:55:07,232 ERROR tuned.utils.commands: Executing > /sbin/sysctl error: sysctl: cannot stat /proc/sys/kernel/numa_balancing: No > such file or directory > > and > > 2015-08-10 15:56:25,122 ERROR tuned.plugins.base: verify: failed: > kernel.numa_balancing = None, expected 0 > None is there because it is unavailable on the system. It's logically correct, because profile requires it, but it's unavailable, so the verification fails. We could handle it as 'soft failures' as Jeremy pointed out (there was proposed =? operator in tuned profile to mark such settings), or there is also proposal for --ignore-missing tuned-adm command line paramater (bug 1243807) which will treat all missing/unavailable settings as passed. I think it doesn't qualify as release blocker, thus moving to RHEL-7.3. Short term solution is to shut this specific error message off, long term is to implement --ignore-missing option or markings for 'soft failures'. (In reply to Jaroslav Škarvada from comment #6) > Short term solution is to shut this specific error message off, long term is > to implement --ignore-missing option or markings for 'soft failures'. Short term fix - verification passes for sysctl plugin even if the option doesn't exist (is unsupported by the system). Upstream commit: https://git.fedorahosted.org/cgit/tuned.git/commit/?id=d49469f7114e1e9f761235a48eb2f4cbdade3f0f Tereza writes in comment 11 >... >I think this is example of right behavior. Right? Do you think so? No, I don't think this is right. Comment 8 says: >Short term fix - verification passes for sysctl plugin even if the option >doesn't exist (is unsupported by the system). What's the difference between the old and new behavior of the command 'tuned-adm verify'? Both print: >Verification failed, current system settings differ from the preset profile. If the option doesn't exist it was supposed to be skipped, yet the verification still fails - the printed message says so. The change in the content of the logfile (from ERROR to INFO) does not appear to make any difference. (In reply to Jeff Nelson from comment #12) > Tereza writes in comment 11 > >... > >I think this is example of right behavior. Right? Do you think so? > > No, I don't think this is right. > > Comment 8 says: > >Short term fix - verification passes for sysctl plugin even if the option >doesn't exist (is unsupported by the system). > > What's the difference between the old and new behavior of the command > 'tuned-adm verify'? Both print: > > >Verification failed, current system settings differ from the preset profile. > > If the option doesn't exist it was supposed to be skipped, yet the > verification still fails - the printed message says so. The change in the > content of the logfile (from ERROR to INFO) does not appear to make any > difference. No, it shouldn't fail, there should be only INFO in the log. (In reply to Jaroslav Škarvada from comment #14) > (In reply to Jeff Nelson from comment #12) > > Tereza writes in comment 11 > > >... > > >I think this is example of right behavior. Right? Do you think so? > > > > No, I don't think this is right. > > > > Comment 8 says: > > >Short term fix - verification passes for sysctl plugin even if the option >doesn't exist (is unsupported by the system). > > > > What's the difference between the old and new behavior of the command > > 'tuned-adm verify'? Both print: > > > > >Verification failed, current system settings differ from the preset profile. > > > > If the option doesn't exist it was supposed to be skipped, yet the > > verification still fails - the printed message says so. The change in the > > content of the logfile (from ERROR to INFO) does not appear to make any > > difference. > > No, it shouldn't fail, there should be only INFO in the log. Probably there are different errors in the test run. Sorry, ERRORs were caused by my inattention. ========================================= Verified in: tuned-2.7.1-3.el7.noarch tuned-profiles-nfv-2.7.1-3.el7.noarch PASS ========================================= # tuned-adm profile realtime-virtual-host # reboot # tuned-adm verify Verfication succeeded, current system settings match the preset profile. See tuned log file ('/var/log/tuned/tuned.log') for details. # grep numa /var/log/tuned/tuned.log 2016-09-16 04:53:12,677 INFO tuned.plugins.base: verify: skipped, missing: 'kernel.numa_balancing' # ========================================= Reproduced in: tuned-2.5.1-3.el7.noarch tuned-profiles-nfv-2.5.1-3.el7.noarch FAIL ========================================= # tuned-adm profile realtime-virtual-host # reboot # tuned-adm verify Verification failed, current system settings differ from the preset profile. See tuned.log for details. You can mostly fix this by Tuned restart, e.g.: service tuned restart # grep numa /var/log/tuned/tuned.log 2016-09-16 04:37:12,413 ERROR tuned.plugins.base: verify: failed: kernel.numa_balancing = None, expected 0 # Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-2479.html |
Created attachment 1061236 [details] tuned logfile Description of problem: Verification of the active profile, realtime-virtual-host, fails. Version-Release number of selected component (if applicable): # yum list installed | egrep tuned tuned.noarch 2.5.1-1.el7 @beaker-Server/7.2 tuned-profiles-nfv.noarch 2.5.1-1.el7 @/tuned-profiles-nfv-2.5.1-1.el7.noarch tuned-profiles-realtime.noarch 2.5.1-1.el7 @beaker-Server-RT How reproducible: 100/100 Steps to Reproduce: 1. tuned-adm profile realtime-virtual-host 2. tuned-adm verify Actual results: Verification failed, current system settings differ from the preset profile. See tuned.log for details. You can mostly fix this by Tuned restart, e.g.: service tuned restart Expected results: Verification works. Additional info: 1. It is not obvious from error message that tuned.log is actually /var/log/tuned/tuned.log 2. tuned.log contains two ERRORS which appear to be responsible for the failure: 2015-08-10 15:55:07,232 ERROR tuned.utils.commands: Executing /sbin/sysctl error: sysctl: cannot stat /proc/sys/kernel/numa_balancing: No such file or directory and 2015-08-10 15:56:25,122 ERROR tuned.plugins.base: verify: failed: kernel.numa_balancing = None, expected 0 3. additional details # uname -a Linux localhost.localdomain 3.10.0-302.rt56.174.el7.x86_64 #1 SMP PREEMPT RT Mon Aug 3 15:01:33 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux # cat /proc/cmdline BOOT_IMAGE=/vmlinuz-3.10.0-302.rt56.174.el7.x86_64 root=/dev/mapper/rhel_virtlab420-root ro crashkernel=auto rd.lvm.lv=rhel_virtlab420/root rd.lvm.lv=rhel_virtlab420/swap console=ttyS1,115200 default_hugepagesz=1G isolcpus=1,3,5,7,9,11,13,14,15 nohz_full=1,3,5,7,9,11,13,14,15 intel_pstate=disable nosoftlockup # lscpu | grep NUMA NUMA node(s): 2 NUMA node0 CPU(s): 0,2,4,6,8,10,12,14 NUMA node1 CPU(s): 1,3,5,7,9,11,13,15