Description of problem:

It has been observed that if the "tuned" profiles that can make changes to the kernel command-line are activated, the "tuned-adm verify" command will fail. The failure error messages are dependent on the chosen profile. For ex. if "network-latency" is selected and if the tuned logs are checked, the error message will be :

~~~~~
2019-02-21 01:51:33,877 INFO     tuned.plugins.base: verify: passed: 'kernel.numa_balancing' = ''
2019-02-21 01:51:33,878 INFO     tuned.plugins.base: verify: passed: 'transparent_hugepages' = 'never'
2019-02-21 01:51:33,878 ERROR    tuned.plugins.plugin_bootloader: verify: failed: 'cmdline' = 'set()', expected '{'skew_tick=1'}'
~~~~~

The cause for the issue is due to the missing "whitespace". If we check the output of "/proc/cmdline", it will be as follows :

~~~~~~~
BOOT_IMAGE=(hd0,msdos1)/vmlinuz........systemd.unified_cgroup_hierarchy=1skew_tick=1
                                                                     ^^^^^^^
~~~~~~~

At the very end, we can see that there is no whitespace between the parameters "systemd.unified_cgroup_hierarchy=1" and "skew_tick=1". And this is exactly what the tuned logs complain.


Version-Release number of selected component (if applicable):

OS Release     : Red Hat Enterprise Linux release 8.0 Beta (Ootpa)
Kernel Version : 4.18.0-64.el8.x86_64
tuned Version  : tuned-2.10.0-14.el8.noarch


How reproducible:

100%. Make sure to select any tuned profile that can make changes to the commandline. For ex. cpu-partitioning, network-latency, realtime, realtime-virtual-guest, realtime-virtual-host


Steps to Reproduce:
1. Install RHEL 8 and tuned package on it.
2. Select any of the profiles mentioned above.
3. Verify the same with "tuned-adm active"
4. Now run "tuned-adm verify". The error message will pop up
5. Check and verify the tuned logs (/var/log/tuned/tuned.log)


Actual results:

"tuned-adm verify" fails.


Expected results:

"tuned-adm verify" should execute correctly.


Additional info:

I have tested this with 2 profiles on 2 different servers : cpu-partitioning and network-latency. The results are as follows :

[A] cpu-partitioning

root@dell-rhel8~:#> tuned-adm active
Current active profile: cpu-partitioning


root@dell-rhel8~:#> tuned-adm verify
Verification failed, current system settings differ from the preset profile.
You can mostly fix this by restarting the Tuned daemon, e.g.:
  systemctl restart tuned
or
  service tuned restart
Sometimes (if some plugins like bootloader are used) a reboot may be required.
See tuned log file ('/var/log/tuned/tuned.log') for details.


root@dell-rhel8~:#> tail /var/log/tuned/tuned.log

2019-02-21 12:20:00,161 INFO     tuned.plugins.plugin_script: calling script '/usr/lib/tuned/cpu-partitioning/script.sh' with arguments '['verify']'
2019-02-21 12:20:00,178 INFO     tuned.plugins.plugin_script: verify: passed: '['/usr/lib/tuned/cpu-partitioning/script.sh']'
2019-02-21 12:20:00,180 ERROR    tuned.plugins.plugin_bootloader: verify: failed: 'cmdline' = '{'rcu_nocbs=2,4,6,8,10,12,14', 'tuned.non_isolcpus=0000aaab', 'nosoftlockup', 'nohz_full=2,4,6,8,10,12,14', 'intel_pstate=disable', 'nohz=on'}', expected '{'rcu_nocbs=2,4,6,8,10,12,14', 'tuned.non_isolcpus=0000aaab', 'skew_tick=1', 'nosoftlockup', 'nohz_full=2,4,6,8,10,12,14', 'intel_pstate=disable', 'nohz=on'}'


root@dell-rhel8~:#> cat /proc/cmdline  (Check "console=tty0skew_tick=1" here)
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-64.el8.x86_64 root=/dev/mapper/rhel_dell--r530--52-root ro crashkernel=auto resume=/dev/mapper/rhel_dell--r530--52-swap rd.lvm.lv=rhel_dell-r530-52/root rd.lvm.lv=rhel_dell-r530-52/swap console=ttyS0,115200 console=tty0skew_tick=1 nohz=on nohz_full=2,4,6,8,10,12,14 rcu_nocbs=2,4,6,8,10,12,14 tuned.non_isolcpus=0000aaab intel_pstate=disable nosoftlockup



[B] network-latency

[root@dell-per430-05 ~]# tuned-adm active
Current active profile: network-latency


[root@dell-per430-05 ~]# tuned-adm verify
Verification failed, current system settings differ from the preset profile.
You can mostly fix this by restarting the Tuned daemon, e.g.:
  systemctl restart tuned
or
  service tuned restart
Sometimes (if some plugins like bootloader are used) a reboot may be required.
See tuned log file ('/var/log/tuned/tuned.log') for details.


[root@dell-per430-05 ~]# tail -n3 /var/log/tuned/tuned.log
2019-02-21 01:51:33,877 INFO     tuned.plugins.base: verify: passed: 'kernel.numa_balancing' = ''
2019-02-21 01:51:33,878 INFO     tuned.plugins.base: verify: passed: 'transparent_hugepages' = 'never'
2019-02-21 01:51:33,878 ERROR    tuned.plugins.plugin_bootloader: verify: failed: 'cmdline' = 'set()', expected '{'skew_tick=1'}'


[root@dell-per430-05 ~]# cat /proc/cmdline (Check "systemd.unified_cgroup_hierarchy=1skew_tick=1" here)
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-64.el8.x86_64 root=/dev/mapper/rhel_dell--per430--05-root ro crashkernel=auto resume=/dev/mapper/rhel_dell--per430--05-swap rd.lvm.lv=rhel_dell-per430-05/root rd.lvm.lv=rhel_dell-per430-05/swap systemd.unified_cgroup_hierarchy=1skew_tick=1