Bug 1642209

Just to be clear, I'm not targeting an NFV usecase, just want to use isolated cpus and VCPU pinning for normal usecase.

if we manually try to provision cpu-partitioning profile with the same list specified in the templates,

[root@overcloud-compute-0 tuned]# cat cpu-partitioning-variables.conf
# Examples:
# isolated_cores=2,4-7
isolated_cores=1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
#
# To disable the kernel load balancing in certain isolated CPUs:
# no_rebalance_cores=5-10


[root@overcloud-compute-0 tuned]# tuned-adm profile cpu-partitioning
Cannot load profile(s) 'cpu-partitioning': Assertion 'isolated_cores contains online CPU(s)' failed.

now if we shrink the list,
[root@overcloud-compute-0 tuned]# cat cpu-partitioning-variables.conf
# Examples:
# isolated_cores=2,4-7
isolated_cores=1,3,5,7
#
# To disable the kernel load balancing in certain isolated CPUs:
# no_rebalance_cores=5-10
[root@overcloud-compute-0 tuned]# tuned-adm profile cpu-partitioning
[root@overcloud-compute-0 tuned]# 

so clearly the tuned has issue with this particular cpu list.

now increase the list to have 1,3,5,7,9 and reapply the profile
[root@overcloud-compute-0 tuned]# cat cpu-partitioning-variables.conf
# Examples:
# isolated_cores=2,4-7
isolated_cores=1,3,5,7,9
#
# To disable the kernel load balancing in certain isolated CPUs:
# no_rebalance_cores=5-10

[root@overcloud-compute-0 tuned]# tuned-adm profile cpu-partitioning
Cannot load profile(s) 'cpu-partitioning': Assertion 'isolated_cores contains online CPU(s)' failed.

[root@overcloud-compute-0 tuned]# rpm -qa | grep tuned
tuned-2.9.0-1.el7_5.2.noarch
tuned-profiles-cpu-partitioning-2.9.0-1.el7_5.2.noarch
[root@overcloud-compute-0 tuned]# 

this looks awefully similar to: 
https://bugzilla.redhat.com/show_bug.cgi?id=1432240
[root@overcloud-compute-0 tuned]# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                40
On-line CPU(s) list:   0-39
Thread(s) per core:    2
Core(s) per socket:    10
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 62
Model name:            Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz
Stepping:              4
CPU MHz:               1199.890
CPU max MHz:           3600.0000
CPU min MHz:           1200.0000
BogoMIPS:              6000.07
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              25600K
NUMA node0 CPU(s):     0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
NUMA node1 CPU(s):     1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts

so it only accept certain list, like 1,3 works but 1,3,5,7,9 doesn't

[heat-admin@overcloud-compute-0 ~]$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-3.10.0-862.11.6.el7.x86_64 root=UUID=c4f51144-921c-4f64-a3ef-5bb0cb17cef0 ro console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet skew_tick=1 nohz=on nohz_full=1,3 rcu_nocbs=1,3 tuned.non_isolcpus=000000ff,fffffff5 intel_pstate=disable nosoftlockup

[root@overcloud-compute-0 tuned]# uname -r
3.10.0-862.11.6.el7.x86_64

ok, I think I saw where the problem is,

in https://github.com/redhat-performance/tuned/blob/master/tuned/profiles/functions/function_assertion.py

change the if statement on line 22 to:
 if ''.join(sorted(args[1].split(','))) != ''.join(sorted(args[1].split(','))):

shall take care of this issue, this is assuming both args[1] and args[2] only take comma separated list.

typo, change above line 22 to:
  if ''.join(sorted(args[1].split(','))) != ''.join(sorted(args[2].split(','))):

@Jianzhu, any idea why the file cat /etc/tuned/cpu-partitioning-variables.conf was not populated at all when the deploy failed?

I only examined the issue  with the manual cpu-partitioning setup failure and overlooked the deployment failure.

we might have two issues with this:
1) deployment failure, the best person to answer this is skramaja
2) tuned failure similar to the existing one https://bugzilla.redhat.com/show_bug.cgi?id=1432240 

So shall we do this: 
1) have this BZ fix the deployment failure
2) I will re-open https://bugzilla.redhat.com/show_bug.cgi?id=1432240

Saravanan,

Could you help us understand what is going on, please?

Sai,
From a very high level view i am guessing this BZ is related to https://bugzilla.redhat.com/show_bug.cgi?id=1645412.
This is an issue we see when we are trying to update to RHEL 7.6 on the undercloud.
If you are doing the same on your set up it is likely this could be the issue.

Vijay.

Thanks Vijay.

Saravanan,
We see
[heat-admin@overcloud-compute-0 ~]$ cat /etc/tuned/cpu-partitioning-variables.conf
# Examples:
# isolated_cores=2,4-7
# isolated_cores=2-23
#
# To disable the kernel load balancing in certain isolated CPUs:
# no_rebalance_cores=5-10

So this file was not even populated in spite passing the required params 
 TunedProfileName: "cpu-partitioning"
 IsolCpusList: "1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39"


That is also an issue here right?

Please try these templates, they are passing CI
https://code.engineering.redhat.com/gerrit/gitweb?p=nfv-qe.git;a=tree;f=ospd-13-vxlan-dpdk-odl-ctlplane-dataplane-bonding-hybrid;h=e00ab0a72a45a8847503a9b699b6e3165d5c149c;hb=refs/heads/ci

It is working