RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1556724 - tuned cpu-partitioning profile, disable cstates with processor.max_cstate=0 intel_idle.max_cstate=0
Summary: tuned cpu-partitioning profile, disable cstates with processor.max_cstate=0 i...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: tuned
Version: 7.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Jaroslav Škarvada
QA Contact: qe-baseos-daemons
URL:
Whiteboard:
Depends On:
Blocks: 1549614
TreeView+ depends on / blocked
 
Reported: 2018-03-15 06:57 UTC by Andreas Karis
Modified: 2022-03-13 14:46 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-06-21 12:28:27 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 3384811 0 None None None 2018-03-17 04:32:19 UTC

Description Andreas Karis 2018-03-15 06:57:49 UTC 
Description of problem:
tuned cpu-partitioning profile, disable cstates with processor.max_cstate=0 intel_idle.max_cstate=0

According to: 
https://access.redhat.com/solutions/202743

We disable p-states from the kernel cmdline, we should to the same with c-states

http://dpdk.org/doc/guides/linux_gsg/nic_perf_intel_platform.html#bios-settings
Comment 2 Jaroslav Škarvada 2018-03-15 09:39:45 UTC 
(In reply to Andreas Karis from comment #0)
> Description of problem:
> tuned cpu-partitioning profile, disable cstates with processor.max_cstate=0
> intel_idle.max_cstate=0
> 
We currently lock the CPU to C1, not to C0 according to the feedback from the performance team, btw it also seems that the kernel driver sets the C state to the C1, even if you request C0 from the kernel command line (processor_idle.c).

Tuned sets the C state through the /dev/cpu_dma_latency not through the kernel command line for greater flexibility. I.e. customer can unlock the CPU from the C1 state by simply switching to different Tuned profile and save power, she/he doesn't need to reboot the machine, for details see e.g.:
https://jeremyeder.com/2012/11/14/processor-max_cstate-intel_idle-max_cstate-and-devcpu_dma_latency/

So I don't think we need to change it.
Comment 3 Andreas Karis 2018-03-16 20:41:05 UTC 
Hello,

The cpu-partitioning profile lets CPUs go into C1 state, and that pretty often.

E.g. from a customer system with tuned cpu-partitioning:
~~~
10.009961 sec
	Package	Core	CPU	Avg_MHz	Busy%	Bzy_MHz	TSC_MHz	IRQ	SMI	CPU%c1	CPU%c3	CPU%c6	CPU%c7	CoreTmp	PkgTmp	Pkg%pc2	Pkg%pc3	Pkg%pc6	PkgWatt	RAMWatt	PKG_%	RAM_%
	-	-	-	576	20.58	2800	2198	0	7040	79.42	0.00	0.00	0.00	77	77	0.00	0.00	0.00	175.39	31.82	0.00	0.00
	0	0	0	2797	100.00	2800	2198	0	80	0.00	0.00	0.00	0.00	65	70	0.00	0.00	0.00	82.37	17.63	0.00	0.00
	0	0	44	883	31.56	2800	2198	0	80	68.44
	0	1	1	170	6.06	2800	2198	0	80	93.94	0.00	0.00	0.00	63
	0	1	45	632	22.60	2800	2198	0	80	77.40
	0	2	2	145	5.20	2800	2198	0	80	94.80	0.00	0.00	0.00	66
	0	2	46	254	9.09	2800	2198	0	80	90.91
	0	3	3	209	7.49	2800	2198	0	80	92.51	0.00	0.00	0.00	64
~~~

Or from my lab:
~~~
root@overcloud-compute-0 ~]# turbostat 
	Core	CPU	Avg_MHz	Busy%	Bzy_MHz	TSC_MHz	IRQ	SMI	CPU%c1	CPU%c3	CPU%c6	CPU%c7	CoreTmp	PkgTmp	PkgWatt	RAMWatt	PKG_%	RAM_%
	-	-	722	30.11	2400	2197	8207	0	69.89	0.00	0.00	0.00	69	69	78.53	9.39	0.00	0.00
	0	0	20	0.85	2400	2197	5313	0	99.15	0.00	0.00	0.00	62	69	43.97	3.85	0.00	0.00
	0	20	21	0.89	2400	2197	499	0	99.11
	1	2	2397	100.00	2400	2197	9	0	0.00	0.00	0.00	0.00	66
	1	22	2397	100.00	2400	2197	9	0	0.00
	2	4	2397	100.00	2400	2197	9	0	0.00	0.00	0.00	0.00	69
	2	24	2397	100.00	2400	2197	9	0	0.00
	3	6	2397	100.00	2400	2197	9	0	0.00	0.00	0.00	0.00	68
	3	26	2397	100.00	2400	2197	9	0	0.00
	4	8	2397	100.00	2400	2197	9	0	0.00	0.00	0.00	0.00	67
	4	28	2397	100.00	2400	2197	9	0	0.00
	8	10	0	0.01	2400	2197	6	0	99.99	0.00	0.00	0.00	61
	8	30	0	0.00	2400	2197	4	0	100.00
	9	12	0	0.01	2400	2197	16	0	99.99	0.00	0.00	0.00	61
	9	32	0	0.00	2400	2197	4	0	100.00
	10	14	0	0.00	2399	2197	3	0	100.00	0.00	0.00	0.00	60
	10	34	0	0.01	2400	2197	20	0	99.99
	11	16	0	0.01	2400	2197	14	0	99.99	0.00	0.00	0.00	60
	11	36	0	0.00	2400	2197	4	0	100.00
	12	18	0	0.00	2400	2197	6	0	100.00	0.00	0.00	0.00	60
	12	38	0	0.00	2401	2197	4	0	100.00
	0	1	23	0.98	2400	2197	1419	0	99.02	0.00	0.00	0.00	61	64	34.56	5.54	0.00	0.00
	0	21	37	1.53	2400	2197	687	0	98.47
	1	3	2397	100.00	2400	2197	9	0	0.00	0.00	0.00	0.00	65
	1	23	2397	100.00	2400	2197	9	0	0.00
	2	5	2397	100.00	2400	2197	9	0	0.00	0.00	0.00	0.00	64
~~~

~~~
[root@overcloud-compute-0 ~]# cat /proc/cmdline 
BOOT_IMAGE=/boot/vmlinuz-3.10.0-693.17.1.el7.x86_64 root=UUID=fa414390-f78d-49d4-a164-54615a32977b ro console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=2,4,6,8,10,12,14,16,18,22,24,26,28,30,32,34,36,38,3,5,7,9,11,13,15,17,19,23,25,27,29,31,33,35,37,39 nohz=on nohz_full=2,4,6,8,10,12,14,16,18,22,24,26,28,30,32,34,36,38,3,5,7,9,11,13,15,17,19,23,25,27,29,31,33,35,37,39 rcu_nocbs=2,4,6,8,10,12,14,16,18,22,24,26,28,30,32,34,36,38,3,5,7,9,11,13,15,17,19,23,25,27,29,31,33,35,37,39 tuned.non_isolcpus=00300003 intel_pstate=disable nosoftlockup
[root@overcloud-compute-0 ~]# 
~~~

Only the PMDs don't sleep (which is logical, they poll constantly). However, with the cpu-partitioning profile, none of the CPUs should ever go to C1, or the instance vCPUs won't be able to pull packets fast enough from the virtio ring and will lead to spurious drops on the vhost-user (vhu) ports in OVS. If the instance is using the standard virtio driver, that is.
Comment 4 Andreas Karis 2018-03-16 20:43:28 UTC 
Agreed on locking to C1, I think that locking to C0 doesn't work, anyhow.

Both https://access.redhat.com/solutions/202743 and https://access.redhat.com/articles/65410 recommend processor.max_cstates=1

https://access.redhat.com/articles/65410 additionally recommends idle=poll
~~~
A system may be prevented from entering power-saving states by booting with the processor.max_cstates=1 command line option. Additionally, the idle=poll option may be added for the fastest time out of the idle state. Unfortunately both of these options tend to cause power usage to spike.
~~~

https://www.kernel.org/doc/Documentation/admin-guide/kernel-parameters.txt list the 3 parameters:
~~~
intel_idle.max_cstate=	[KNL,HW,ACPI,X86]
			0	disables intel_idle and fall back on acpi_idle.
			1 to 9	specify maximum depth of C-state.
~~~

~~~
	processor.max_cstate=	[HW,ACPI]
			Limit processor to maximum C-state
			max_cstate=9 overrides any DMI blacklist limit.
~~~

~~~
	idle=		[X86]
			Format: idle=poll, idle=halt, idle=nomwait
			Poll forces a polling idle loop that can slightly
			improve the performance of waking up a idle CPU, but
			will use a lot of power and make the system run hot.
			Not recommended.
			idle=halt: Halt is forced to be used for CPU idle.
			In such case C2/C3 won't be used again.
			idle=nomwait: Disable mwait for CPU C-states
~~~

I just tested out several variations:
~~~
processor.max_cstate=0 intel_idle.max_cstate=0
processor.max_cstate=0 intel_idle.max_cstate=0
processor.max_cstate=1 intel_idle.max_cstate=0 idle=poll
processor.max_cstate=1 intel_idle.max_cstate=0 idle=poll
~~~

Only 2 yield C0 states only:
~~~
processor.max_cstate=0 intel_idle.max_cstate=0 idle=poll
processor.max_cstate=1 intel_idle.max_cstate=0 idle=poll
~~~

The important point here is `idle=poll`.

~~~
[root@overcloud-compute-0 ~]# lscpu  | grep 'Model name'
Model name:            Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
[root@overcloud-compute-0 ~]# cat /proc/cmdline 
BOOT_IMAGE=/boot/vmlinuz-3.10.0-693.17.1.el7.x86_64 root=UUID=fa414390-f78d-49d4-a164-54615a32977b ro console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=2,4,6,8,10,12,14,16,18,22,24,26,28,30,32,34,36,38,3,5,7,9,11,13,15,17,19,23,25,27,29,31,33,35,37,39 processor.max_cstate=1 intel_idle.max_cstate=0 idle=poll nohz=on nohz_full=2,4,6,8,10,12,14,16,18,22,24,26,28,30,32,34,36,38,3,5,7,9,11,13,15,17,19,23,25,27,29,31,33,35,37,39 rcu_nocbs=2,4,6,8,10,12,14,16,18,22,24,26,28,30,32,34,36,38,3,5,7,9,11,13,15,17,19,23,25,27,29,31,33,35,37,39 tuned.non_isolcpus=00300003 intel_pstate=disable nosoftlockup
[root@overcloud-compute-0 ~]# turbostat 1
0.023338 sec
	Core	CPU	Avg_MHz	Busy%	Bzy_MHz	TSC_MHz	IRQ	SMI	CPU%c1	CPU%c3	CPU%c6	CPU%c7	CoreTmp	PkgTmp	PkgWatt	RAMWatt	PKG_%	RAM_%
	-	-	2294	100.00	2400	2103	0	0	0.00	0.00	0.00	0.00	68	71	48.48	5.79	0.00	0.00
	0	0	140	100.00	2400	128	0	0	0.00	0.00	0.00	0.00	64	71	4.19	0.33	0.00	0.00
	0	20	140	100.00	2400	128	0	0	0.00
	1	2	2326	100.00	2400	2132	0	0	0.00	0.00	0.00	0.00	66
	1	22	2425	100.00	2400	2223	0	0	0.00
	2	4	2424	100.00	2400	2222	0	0	0.00	0.00	0.00	0.00	67
	2	24	2424	100.00	2400	2222	0	0	0.00
	3	6	2422	100.00	2400	2220	0	0	0.00	0.00	0.00	0.00	68
	3	26	2421	100.00	2400	2220	0	0	0.00
	4	8	2420	100.00	2400	2218	0	0	0.00	0.00	0.00	0.00	66
	4	28	2419	100.00	2400	2217	0	0	0.00
	8	10	2419	100.00	2400	2218	0	0	0.00	0.00	0.00	0.00	65
	8	30	2418	100.00	2400	2216	0	0	0.00
	9	12	2416	100.00	2400	2215	0	0	0.00	0.00	0.00	0.00	65
	9	32	2415	100.00	2400	2214	0	0	0.00
	10	14	2414	100.00	2400	2213	0	0	0.00	0.00	0.00	0.00	66
	10	34	2413	100.00	2400	2212	0	0	0.00
	11	16	2412	100.00	2400	2211	0	0	0.00	0.00	0.00	0.00	65
	11	36	2411	100.00	2400	2210	0	0	0.00
	12	18	2409	100.00	2400	2209	0	0	0.00	0.00	0.00	0.00	64
	12	38	2409	100.00	2400	2208	0	0	0.00
	0	1	2408	100.00	2400	2207	0	0	0.00	0.00	0.00	0.00	67	70	44.29	5.45	0.00	0.00
	0	21	2409	100.00	2400	2208	0	0	0.00
	1	3	2408	100.00	2400	2207	0	0	0.00	0.00	0.00	0.00	68
	1	23	2407	100.00	2400	2207	0	0	0.00
	2	5	2406	100.00	2400	2205	0	0	0.00	0.00	0.00	0.00	68
	2	25	2405	100.00	2400	2204	0	0	0.00
	3	7	2404	100.00	2400	2204	0	0	0.00	0.00	0.00	0.00	66
	3	27	2404	100.00	2400	2203	0	0	0.00
	4	9	2403	100.00	2400	2203	0	0	0.00	0.00	0.00	0.00	67
	4	29	2402	100.00	2400	2202	0	0	0.00
	8	11	2402	100.00	2400	2202	0	0	0.00	0.00	0.00	0.00	66
	8	31	2402	100.00	2400	2202	0	0	0.00
	9	13	2401	100.00	2400	2201	0	0	0.00	0.00	0.00	0.00	66
	9	33	2401	100.00	2400	2201	0	0	0.00
	10	15	2400	100.00	2400	2200	0	0	0.00	0.00	0.00	0.00	66
	10	35	2399	100.00	2400	2199	0	0	0.00
	11	17	2399	100.00	2400	2199	0	0	0.00	0.00	0.00	0.00	67
	11	37	2398	100.00	2400	2198	0	0	0.00
	12	19	2397	100.00	2400	2197	0	0	0.00	0.00	0.00	0.00	66
	12	39	2396	100.00	2400	2196	0	0	0.00
0.025176 sec
	Core	CPU	Avg_MHz	Busy%	Bzy_MHz	TSC_MHz	IRQ	SMI	CPU%c1	CPU%c3	CPU%c6	CPU%c7	CoreTmp	PkgTmp	PkgWatt	RAMWatt	PKG_%	RAM_%
	-	-	2412	100.00	2400	2211	0	0	0.00	0.00	0.00	0.00	68	71	94.11	9.51	0.00	0.00
	0	0	2429	100.00	2400	2227	0	0	0.00	0.00	0.00	0.00	64	71	49.51	4.02	0.00	0.00
	0	20	2429	100.00	2400	2227	0	0	0.00
	1	2	2428	100.00	2400	2226	0	0	0.00	0.00	0.00	0.00	66
	1	22	2427	100.00	2400	2225	0	0	0.00
	2	4	2426	100.00	2400	2224	0	0	0.00	0.00	0.00	0.00	67
	2	24	2425	100.00	2400	2223	0	0	0.00
	3	6	2424	100.00	2400	2222	0	0	0.00	0.00	0.00	0.00	68
	3	26	2422	100.00	2400	2220	0	0	0.00
	4	8	2421	100.00	2400	2219	0	0	0.00	0.00	0.00	0.00	66
	4	28	2420	100.00	2400	2219	0	0	0.00
	8	10	2419	100.00	2400	2218	0	0	0.00	0.00	0.00	0.00	65
	8	30	2418	100.00	2400	2216	0	0	0.00
	9	12	2417	100.00	2400	2215	0	0	0.00	0.00	0.00	0.00	65
	9	32	2416	100.00	2400	2215	0	0	0.00
	10	14	2415	100.00	2400	2213	0	0	0.00	0.00	0.00	0.00	66
	10	34	2414	100.00	2400	2213	0	0	0.00
	11	16	2413	100.00	2400	2212	0	0	0.00	0.00	0.00	0.00	65
	11	36	2412	100.00	2400	2211	0	0	0.00
	12	18	2410	100.00	2400	2210	0	0	0.00	0.00	0.00	0.00	64
	12	38	2410	100.00	2400	2209	0	0	0.00
	0	1	2409	100.00	2400	2208	0	0	0.00	0.00	0.00	0.00	67	69	44.60	5.49	0.00	0.00
	0	21	2409	100.00	2400	2208	0	0	0.00
	1	3	2408	100.00	2400	2208	0	0	0.00	0.00	0.00	0.00	68
	1	23	2408	100.00	2400	2207	0	0	0.00
	2	5	2406	100.00	2400	2206	0	0	0.00	0.00	0.00	0.00	68
	2	25	2406	100.00	2400	2205	0	0	0.00
	3	7	2405	100.00	2400	2205	0	0	0.00	0.00	0.00	0.00	66
	3	27	2405	100.00	2400	2204	0	0	0.00
	4	9	2404	100.00	2400	2204	0	0	0.00	0.00	0.00	0.00	67
	4	29	2404	100.00	2400	2204	0	0	0.00
	8	11	2403	100.00	2400	2203	0	0	0.00	0.00	0.00	0.00	66
	8	31	2403	100.00	2400	2203	0	0	0.00
	9	13	2402	100.00	2400	2202	0	0	0.00	0.00	0.00	0.00	66
	9	33	2402	100.00	2400	2202	0	0	0.00
	10	15	2401	100.00	2400	2201	0	0	0.00	0.00	0.00	0.00	66
	10	35	2401	100.00	2400	2201	0	0	0.00
	11	17	2400	100.00	2400	2200	0	0	0.00	0.00	0.00	0.00	67
	11	37	2399	100.00	2400	2199	0	0	0.00
	12	19	2399	100.00	2400	2199	0	0	0.00	0.00	0.00	0.00	66
	12	39	2397	100.00	2400	2198	0	0	0.00
[root@overcloud-compute-0 ~]# 
~~~

~~~
[root@overcloud-compute-0 ~]# cat /proc/cmdline 
BOOT_IMAGE=/boot/vmlinuz-3.10.0-693.17.1.el7.x86_64 root=UUID=fa414390-f78d-49d4-a164-54615a32977b ro console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=2,4,6,8,10,12,14,16,18,22,24,26,28,30,32,34,36,38,3,5,7,9,11,13,15,17,19,23,25,27,29,31,33,35,37,39 processor.max_cstate=0 intel_idle.max_cstate=0 idle=poll nohz=on nohz_full=2,4,6,8,10,12,14,16,18,22,24,26,28,30,32,34,36,38,3,5,7,9,11,13,15,17,19,23,25,27,29,31,33,35,37,39 rcu_nocbs=2,4,6,8,10,12,14,16,18,22,24,26,28,30,32,34,36,38,3,5,7,9,11,13,15,17,19,23,25,27,29,31,33,35,37,39 tuned.non_isolcpus=00300003 intel_pstate=disable nosoftlockup
[root@overcloud-compute-0 ~]# turbostat
	Core	CPU	Avg_MHz	Busy%	Bzy_MHz	TSC_MHz	IRQ	SMI	CPU%c1	CPU%c3	CPU%c6	CPU%c7	CoreTmp	PkgTmp	PkgWatt	RAMWatt	PKG_%	RAM_%
	-	-	2397	100.00	2400	2198	13460	0	0.00	0.00	0.00	0.00	66	69	92.05	9.46	0.00	0.00
	0	0	2398	100.00	2400	2198	5147	0	0.00	0.00	0.00	0.00	63	69	48.47	3.90	0.00	0.00
	0	20	2398	100.00	2400	2198	2987	0	0.00
	1	2	2398	100.00	2400	2198	14	0	0.00	0.00	0.00	0.00	65
	1	22	2398	100.00	2400	2198	14	0	0.00
	2	4	2398	100.00	2400	2198	18	0	0.00	0.00	0.00	0.00	66
	2	24	2398	100.00	2400	2198	40	0	0.00
	3	6	2398	100.00	2400	2198	40	0	0.00	0.00	0.00	0.00	64
	3	26	2398	100.00	2400	2198	40	0	0.00
	4	8	2398	100.00	2400	2198	40	0	0.00	0.00	0.00	0.00	64
	4	28	2398	100.00	2400	2198	40	0	0.00
	8	10	2398	100.00	2400	2198	5	0	0.00	0.00	0.00	0.00	63
	8	30	2398	100.00	2400	2198	5	0	0.00
	9	12	2398	100.00	2400	2198	5	0	0.00	0.00	0.00	0.00	64
	9	32	2398	100.00	2400	2198	5	0	0.00
	10	14	2398	100.00	2400	2198	5	0	0.00	0.00	0.00	0.00	64
	10	34	2398	100.00	2400	2198	5	0	0.00
	11	16	2398	100.00	2400	2198	5	0	0.00	0.00	0.00	0.00	64
	11	36	2397	100.00	2400	2198	5	0	0.00
	12	18	2397	100.00	2400	2198	6	0	0.00	0.00	0.00	0.00	62
	12	38	2397	100.00	2400	2198	5	0	0.00
	0	1	2397	100.00	2400	2198	2262	0	0.00	0.00	0.00	0.00	63	66	43.58	5.56	0.00	0.00
	0	21	2397	100.00	2400	2198	2660	0	0.00
	1	3	2397	100.00	2400	2198	20	0	0.00	0.00	0.00	0.00	64
	1	23	2397	100.00	2400	2198	5	0	0.00
	2	5	2397	100.00	2400	2198	5	0	0.00	0.00	0.00	0.00	64
	2	25	2397	100.00	2400	2198	5	0	0.00
	3	7	2397	100.00	2400	2198	5	0	0.00	0.00	0.00	0.00	62
	3	27	2397	100.00	2400	2198	5	0	0.00
	4	9	2397	100.00	2400	2198	5	0	0.00	0.00	0.00	0.00	63
	4	29	2397	100.00	2400	2198	5	0	0.00
	8	11	2397	100.00	2400	2198	5	0	0.00	0.00	0.00	0.00	62
	8	31	2397	100.00	2400	2198	5	0	0.00
	9	13	2397	100.00	2400	2198	5	0	0.00	0.00	0.00	0.00	62
	9	33	2397	100.00	2400	2198	5	0	0.00
	10	15	2397	100.00	2400	2198	5	0	0.00	0.00	0.00	0.00	63
	10	35	2397	100.00	2400	2197	5	0	0.00
	11	17	2397	100.00	2400	2197	5	0	0.00	0.00	0.00	0.00	63
	11	37	2397	100.00	2400	2197	6	0	0.00
	12	19	2397	100.00	2400	2197	5	0	0.00	0.00	0.00	0.00	62
	12	39	2397	100.00	2400	2197	6	0	0.00
	Core	CPU	Avg_MHz	Busy%	Bzy_MHz	TSC_MHz	IRQ	SMI	CPU%c1	CPU%c3	CPU%c6	CPU%c7	CoreTmp	PkgTmp	PkgWatt	RAMWatt	PKG_%	RAM_%
	-	-	2397	100.00	2400	2197	13435	0	0.00	0.00	0.00	0.00	66	69	92.56	9.45	0.00	0.00
	0	0	2397	100.00	2400	2197	5275	0	0.00	0.00	0.00	0.00	63	69	48.45	3.89	0.00	0.00
	0	20	2397	100.00	2400	2197	2523	0	0.00
	1	2	2397	100.00	2400	2197	14	0	0.00	0.00	0.00	0.00	65
	1	22	2397	100.00	2400	2197	14	0	0.00
	2	4	2397	100.00	2400	2197	14	0	0.00	0.00	0.00	0.00	66
	2	24	2397	100.00	2400	2197	14	0	0.00
	3	6	2397	100.00	2400	2197	14	0	0.00	0.00	0.00	0.00	65
	3	26	2397	100.00	2400	2197	14	0	0.00
	4	8	2397	100.00	2400	2197	14	0	0.00	0.00	0.00	0.00	64
	4	28	2397	100.00	2400	2197	14	0	0.00
	8	10	2397	100.00	2400	2197	2	0	0.00	0.00	0.00	0.00	64
	8	30	2397	100.00	2400	2197	2	0	0.00
	9	12	2397	100.00	2400	2197	2	0	0.00	0.00	0.00	0.00	64
	9	32	2397	100.00	2400	2197	2	0	0.00
	10	14	2397	100.00	2400	2197	2	0	0.00	0.00	0.00	0.00	64
	10	34	2397	100.00	2400	2197	2	0	0.00
	11	16	2397	100.00	2400	2197	2	0	0.00	0.00	0.00	0.00	64
	11	36	2397	100.00	2400	2197	2	0	0.00
	12	18	2397	100.00	2400	2197	2	0	0.00	0.00	0.00	0.00	62
	12	38	2397	100.00	2400	2197	2	0	0.00
	0	1	2397	100.00	2400	2197	2642	0	0.00	0.00	0.00	0.00	65	66	44.11	5.56	0.00	0.00
	0	21	2397	100.00	2400	2197	2814	0	0.00
	1	3	2397	100.00	2400	2197	10	0	0.00	0.00	0.00	0.00	65
	1	23	2397	100.00	2400	2197	2	0	0.00
	2	5	2397	100.00	2400	2197	2	0	0.00	0.00	0.00	0.00	65
	2	25	2397	100.00	2400	2197	2	0	0.00
	3	7	2397	100.00	2400	2197	3	0	0.00	0.00	0.00	0.00	63
	3	27	2397	100.00	2400	2197	2	0	0.00
	4	9	2397	100.00	2400	2197	2	0	0.00	0.00	0.00	0.00	64
	4	29	2397	100.00	2400	2197	2	0	0.00
	8	11	2397	100.00	2400	2197	2	0	0.00	0.00	0.00	0.00	62
	8	31	2397	100.00	2400	2197	2	0	0.00
	9	13	2397	100.00	2400	2197	2	0	0.00	0.00	0.00	0.00	63
	9	33	2397	100.00	2400	2197	2	0	0.00
	10	15	2397	100.00	2400	2197	2	0	0.00	0.00	0.00	0.00	63
	10	35	2397	100.00	2400	2197	2	0	0.00
	11	17	2397	100.00	2400	2197	2	0	0.00	0.00	0.00	0.00	64
	11	37	2397	100.00	2400	2197	2	0	0.00
	12	19	2397	100.00	2400	2197	2	0	0.00	0.00	0.00	0.00	63
	12	39	2397	100.00	2400	2197	6	0	0.00
^C
[root@overcloud-compute-0 ~]# 
~~~
Comment 5 Andreas Karis 2018-03-16 20:47:14 UTC 
The question hence is:
* do we desire to lock only at C1, instead of heaving all CPUs remain at C0 constantly? Intel's recommendation: 
http://dpdk.org/doc/guides/linux_gsg/nic_perf_intel_platform.html#bios-settings

Is Intel in the following recommending to disable C3 and C6 only, or all C-states?
~~~
    Before starting consider resetting all BIOS settings to their default.
    Disable all power saving options such as: Power performance tuning, CPU P-State, CPU C3 Report and CPU C6 Report.
    Select Performance as the CPU Power and Performance policy.
    Disable Turbo Boost to ensure the performance scaling increases with the number of cores.
    Set memory frequency to the highest available number, NOT auto.
    Disable all virtualization options when you test the physical function of the NIC, and turn on VT-d if you wants to use VFIO.
~~~

If we allow C1, how can I make sure that I see no spurious TX drops in the following scenario:
* instance with kernel virtio
* OVS DPDK
My understanding is that if the instance vCPUs go into C1, then they'll potentially cause packet drop on the vhu port.
Comment 8 Andreas Karis 2018-03-17 00:03:18 UTC 
Hello,

I followed 

https://jeremyeder.com/2012/11/14/processor-max_cstate-intel_idle-max_cstate-and-devcpu_dma_latency/

which leads to
https://github.com/NetSys/NetBricks/blob/master/scripts/tuning/pmqos-static.py

This script is part of tuned-utils! 

On compute-1, I executed:
~~~
yum install tuned-utils -y
/usr/libexec/tuned/pmqos-static.py cpu_dma_latency=0
~~~

And the CPU C-States now do look good:
~~~
[root@compute-1 ~]# /usr/libexec/tuned/pmqos-static.py cpu_dma_latency=0
[root@compute-1 ~]# turbostat
        Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IRQ     SMI     CPU%c1  CPU%c3  CPU%c6  CPU%c7  CoreTmp PkgTmp  PkgWatt RAMWatt PKG_%   RAM_%
        -       -       2198    100.00  2200    2198    32048   3608    0.00    0.00    0.00    0.00    63      65      178.67  31.67   0.00    0.00
        0       0       2198    100.00  2200    2198    8894    41      0.00    0.00    0.00    0.00    58      62      88.24   17.62   0.00    0.00
        0       44      2198    100.00  2200    2198    3293    41      0.00
        1       1       2198    100.00  2200    2198    6261    41      0.00    0.00    0.00    0.00    60
        1       45      2198    100.00  2200    2198    730     41      0.00
        2       2       2198    100.00  2200    2198    5385    41      0.00    0.00    0.00    0.00    59
        2       46      2198    100.00  2200    2198    627     41      0.00
        3       3       2198    100.00  2200    2198    4142    41      0.00    0.00    0.00    0.00    59
        3       47      2198    100.00  2200    2198    2253    41      0.00
        4       4       2198    100.00  2200    2198    14      41      0.00    0.00    0.00    0.00    60
        4       48      2198    100.00  2200    2198    14      41      0.00
        5       5       2198    100.00  2200    2198    14      41      0.00    0.00    0.00    0.00    60
        5       49      2198    100.00  2200    2198    15      41      0.00
        8       6       2198    100.00  2200    2198    5       41      0.00    0.00    0.00    0.00    60
        8       50      2198    100.00  2200    2198    4       41      0.00
        9       7       2198    100.00  2200    2198    4       41      0.00    0.00    0.00    0.00    59
        9       51      2198    100.00  2200    2198    4       41      0.00
        10      8       2198    100.00  2200    2198    4       41      0.00    0.00    0.00    0.00    59
        10      52      2198    100.00  2200    2198    4       41      0.00
        11      9       2198    100.00  2200    2198    4       41      0.00    0.00    0.00    0.00    60
        11      53      2198    100.00  2200    2198    4       41      0.00
        12      10      2198    100.00  2200    2198    4       41      0.00    0.00    0.00    0.00    59
~~~

If I follow Franck's recommendation, then we should not reach C1 for the cpu-partitioning profile. Or we should have a switch there that allows us to set /usr/libexec/tuned/pmqos-static.py cpu_dma_latency=0

Also note that the website which you referred to earlier explains why processor.max_cstate=1 and processor.max_cstate=0 are the same:
~~~
1) When you set processor.max_cstate=0, the kernel actually silently sets it to 1.

drivers/acpi/processor_idle.c:1086:
 1086 if (max_cstate == 0)
 1087 max_cstate = 1;
~~~

- Andreas
Comment 9 Andreas Karis 2018-03-17 03:48:10 UTC 
I just checked, and none of the tuned profiles actually force C0.

---

[root@overcloud-compute-0 ~]# for i in balanced cpu-partitioning desktop latency-performance network-latency network-throughput powersave throughput-performance virtual-guest virtual-host ; do echo "" ; echo "+++ $i +++" ; tuned-adm profile $i ; turbostat sleep 5 ; done

+++ balanced +++
5.002253 sec
	Core	CPU	Avg_MHz	Busy%	Bzy_MHz	TSC_MHz	IRQ	SMI	CPU%c1	CPU%c3	CPU%c6	CPU%c7	CoreTmp	PkgTmp	PkgWatt	RAMWatt	PKG_%	RAM_%
	-	-	538	19.46	2771	2198	0	0	2.47	0.03	78.03	0.00	64	66	54.65	9.44	0.00	0.00
	0	0	32	1.19	2702	2198	0	0	27.38	0.02	71.40	0.00	59	66	36.83	3.85	0.00	0.00
	0	20	6	0.21	2705	2198	0	0	28.37
	1	2	2684	96.94	2772	2198	0	0	3.06	0.00	0.00	0.00	64
	1	22	2685	96.95	2772	2198	0	0	3.05
	2	4	2685	96.97	2772	2198	0	0	3.03	0.00	0.00	0.00	64
	2	24	2686	96.98	2772	2198	0	0	3.02
	3	6	2685	96.98	2772	2198	0	0	3.02	0.00	0.00	0.00	64
	3	26	2686	96.99	2772	2198	0	0	3.01
	4	8	2685	96.98	2772	2198	0	0	3.02	0.00	0.00	0.00	63
	4	28	2686	96.99	2772	2198	0	0	3.01
	8	10	0	0.00	2628	2198	0	0	0.01	0.00	99.98	0.00	57
	8	30	0	0.00	2588	2198	0	0	0.01
	9	12	0	0.00	2545	2198	0	0	0.01	0.00	99.99	0.00	56
	9	32	0	0.00	2514	2198	0	0	0.01
	10	14	0	0.00	2458	2198	0	0	0.01	0.00	99.99	0.00	55
	10	34	0	0.00	2442	2198	0	0	0.01
	11	16	0	0.00	2437	2198	0	0	0.01	0.00	99.99	0.00	56
	11	36	0	0.00	2444	2198	0	0	0.01
	12	18	0	0.00	2424	2198	0	0	0.01	0.01	99.97	0.00	55
	12	38	0	0.00	2479	2198	0	0	0.01
	0	1	13	0.70	1859	2198	0	0	9.18	0.62	89.50	0.00	52	59	17.82	5.59	0.00	0.00
	0	21	7	0.42	1629	2198	0	0	9.45
	1	3	0	0.00	1670	2198	0	0	0.01	0.00	99.99	0.00	52
	1	23	0	0.00	2171	2198	0	0	0.01
	2	5	0	0.00	1702	2197	0	0	0.01	0.00	99.99	0.00	51
	2	25	0	0.00	2743	2197	0	0	0.01
	3	7	0	0.00	2808	2197	0	0	0.01	0.01	99.98	0.00	51
	3	27	0	0.00	2724	2197	0	0	0.01
	4	9	0	0.00	2769	2197	0	0	0.01	0.00	99.98	0.00	52
	4	29	0	0.00	2724	2197	0	0	0.01
	8	11	0	0.00	2714	2197	0	0	0.01	0.00	99.99	0.00	52
	8	31	0	0.00	2729	2197	0	0	0.01
	9	13	0	0.00	2743	2197	0	0	0.01	0.00	99.99	0.00	51
	9	33	0	0.00	2738	2197	0	0	0.01
	10	15	0	0.00	2738	2197	0	0	0.01	0.00	99.99	0.00	52
	10	35	0	0.00	2693	2197	0	0	0.01
	11	17	0	0.00	2698	2197	0	0	0.01	0.01	99.97	0.00	51
	11	37	0	0.00	2636	2197	0	0	0.01
	12	19	0	0.00	2350	2197	0	0	0.03	0.00	99.96	0.00	52
	12	39	0	0.01	2856	2197	0	0	0.02

+++ cpu-partitioning +++
5.001604 sec
	Core	CPU	Avg_MHz	Busy%	Bzy_MHz	TSC_MHz	IRQ	SMI	CPU%c1	CPU%c3	CPU%c6	CPU%c7	CoreTmp	PkgTmp	PkgWatt	RAMWatt	PKG_%	RAM_%
	-	-	242	10.03	2400	2207	0	0	89.97	0.00	0.00	0.00	59	66	64.85	9.90	0.00	0.00
	0	0	2422	100.00	2400	2220	0	0	0.00	0.00	0.00	0.00	59	66	35.24	4.31	0.00	0.00
	0	20	2419	100.00	2400	2217	0	0	0.00
	1	2	26	1.06	2400	2217	0	0	98.94	0.00	0.00	0.00	58
	1	22	0	0.00	2399	2217	0	0	100.00
	2	4	0	0.00	2399	2217	0	0	100.00	0.00	0.00	0.00	58
	2	24	0	0.00	2404	2217	0	0	100.00
	3	6	0	0.00	2401	2217	0	0	100.00	0.00	0.00	0.00	59
	3	26	0	0.00	2400	2217	0	0	100.00
	4	8	0	0.00	2400	2217	0	0	100.00	0.00	0.00	0.00	58
	4	28	0	0.00	2400	2217	0	0	100.00
	8	10	0	0.00	2401	2217	0	0	100.00	0.00	0.00	0.00	58
	8	30	0	0.00	2401	2217	0	0	100.00
	9	12	0	0.00	2401	2217	0	0	100.00	0.00	0.00	0.00	57
	9	32	0	0.00	2401	2217	0	0	100.00
	10	14	0	0.00	2401	2217	0	0	100.00	0.00	0.00	0.00	57
	10	34	0	0.00	2402	2217	0	0	100.00
	11	16	0	0.00	2401	2217	0	0	100.00	0.00	0.00	0.00	57
	11	36	0	0.00	2401	2217	0	0	100.00
	12	18	0	0.00	2402	2217	0	0	100.00	0.00	0.00	0.00	57
	12	38	0	0.00	2401	2217	0	0	100.00
	0	1	2398	100.00	2400	2198	0	0	0.00	0.00	0.00	0.00	58	60	29.61	5.59	0.00	0.00
	0	21	2397	100.00	2400	2198	0	0	0.00
	1	3	0	0.00	2402	2198	0	0	100.00	0.00	0.00	0.00	56
	1	23	0	0.00	2402	2198	0	0	100.00
	2	5	0	0.00	2401	2198	0	0	100.00	0.00	0.00	0.00	54
	2	25	0	0.00	2401	2198	0	0	100.00
	3	7	0	0.00	2401	2198	0	0	100.00	0.00	0.00	0.00	54
	3	27	0	0.00	2401	2198	0	0	100.00
	4	9	0	0.00	2401	2198	0	0	100.00	0.00	0.00	0.00	54
	4	29	0	0.00	2401	2197	0	0	100.00
	8	11	0	0.00	2401	2197	0	0	100.00	0.00	0.00	0.00	54
	8	31	0	0.00	2403	2197	0	0	100.00
	9	13	0	0.00	2402	2197	0	0	100.00	0.00	0.00	0.00	53
	9	33	0	0.00	2401	2197	0	0	100.00
	10	15	0	0.00	2401	2197	0	0	100.00	0.00	0.00	0.00	54
	10	35	0	0.00	2401	2197	0	0	100.00
	11	17	0	0.00	2401	2197	0	0	100.00	0.00	0.00	0.00	54
	11	37	0	0.00	2401	2197	0	0	100.00
	12	19	0	0.00	2402	2197	0	0	100.00	0.00	0.00	0.00	54
	12	39	0	0.02	2400	2197	0	0	99.98

+++ desktop +++
5.002544 sec
	Core	CPU	Avg_MHz	Busy%	Bzy_MHz	TSC_MHz	IRQ	SMI	CPU%c1	CPU%c3	CPU%c6	CPU%c7	CoreTmp	PkgTmp	PkgWatt	RAMWatt	PKG_%	RAM_%
	-	-	311	10.04	3100	2198	0	0	1.38	0.00	88.58	0.00	62	65	50.74	9.72	0.00	0.00
	0	0	3100	99.99	3100	2200	0	0	0.01	0.00	0.00	0.00	62	65	28.28	3.98	0.00	0.00
	0	20	3098	99.99	3100	2199	0	0	0.01
	1	2	41	1.32	3101	2199	0	0	26.73	0.01	71.94	0.00	56
	1	22	5	0.15	3106	2199	0	0	27.89
	2	4	0	0.00	2809	2199	0	0	0.01	0.01	99.98	0.00	55
	2	24	0	0.00	2758	2199	0	0	0.01
	3	6	0	0.00	2739	2199	0	0	0.01	0.01	99.98	0.00	56
	3	26	0	0.00	2685	2199	0	0	0.01
	4	8	0	0.00	2655	2199	0	0	0.01	0.01	99.98	0.00	56
	4	28	0	0.00	2635	2199	0	0	0.01
	8	10	0	0.00	2636	2199	0	0	0.01	0.00	99.99	0.00	56
	8	30	0	0.00	2603	2199	0	0	0.01
	9	12	0	0.00	2660	2199	0	0	0.01	0.00	99.98	0.00	55
	9	32	0	0.00	2592	2199	0	0	0.01
	10	14	0	0.00	2630	2199	0	0	0.01	0.00	99.99	0.00	54
	10	34	0	0.00	2663	2199	0	0	0.01
	11	16	0	0.00	2764	2199	0	0	0.01	0.00	99.99	0.00	54
	11	36	0	0.00	2799	2199	0	0	0.01
	12	18	0	0.00	2774	2199	0	0	0.01	0.00	99.99	0.00	54
	12	38	0	0.00	2795	2199	0	0	0.01
	0	1	3097	99.99	3100	2198	0	0	0.01	0.00	0.00	0.00	59	59	22.46	5.74	0.00	0.00
	0	21	3096	99.99	3100	2197	0	0	0.01
	1	3	0	0.00	2990	2197	0	0	0.01	0.00	99.99	0.00	53
	1	23	0	0.00	2855	2197	0	0	0.01
	2	5	0	0.00	2797	2197	0	0	0.01	0.00	99.99	0.00	53
	2	25	0	0.00	2733	2197	0	0	0.01
	3	7	0	0.00	2717	2197	0	0	0.01	0.00	99.99	0.00	52
	3	27	0	0.00	2666	2197	0	0	0.01
	4	9	0	0.00	2654	2197	0	0	0.01	0.00	99.99	0.00	52
	4	29	0	0.00	2638	2197	0	0	0.01
	8	11	0	0.00	2664	2197	0	0	0.01	0.00	99.99	0.00	53
	8	31	0	0.00	2648	2197	0	0	0.01
	9	13	0	0.00	2565	2197	0	0	0.01	0.00	99.98	0.00	51
	9	33	0	0.00	2597	2197	0	0	0.01
	10	15	0	0.00	2620	2197	0	0	0.01	0.00	99.99	0.00	53
	10	35	0	0.00	2564	2197	0	0	0.01
	11	17	0	0.00	2679	2197	0	0	0.01	0.00	99.99	0.00	51
	11	37	0	0.00	2794	2197	0	0	0.01
	12	19	0	0.00	2834	2197	0	0	0.05	0.00	99.95	0.00	53
	12	39	0	0.01	2803	2197	0	0	0.04

+++ latency-performance +++
5.001753 sec
	Core	CPU	Avg_MHz	Busy%	Bzy_MHz	TSC_MHz	IRQ	SMI	CPU%c1	CPU%c3	CPU%c6	CPU%c7	CoreTmp	PkgTmp	PkgWatt	RAMWatt	PKG_%	RAM_%
	-	-	242	10.04	2400	2211	0	0	89.96	0.00	0.00	0.00	59	65	64.41	9.89	0.00	0.00
	0	0	2434	100.00	2400	2232	0	0	0.00	0.00	0.00	0.00	59	65	34.95	4.30	0.00	0.00
	0	20	2425	100.00	2400	2223	0	0	0.00
	1	2	26	1.05	2400	2223	0	0	98.95	0.00	0.00	0.00	57
	1	22	0	0.00	2400	2223	0	0	100.00
	2	4	0	0.00	2401	2223	0	0	100.00	0.00	0.00	0.00	57
	2	24	0	0.00	2400	2223	0	0	100.00
	3	6	0	0.00	2402	2223	0	0	100.00	0.00	0.00	0.00	58
	3	26	0	0.00	2402	2223	0	0	100.00
	4	8	0	0.00	2401	2223	0	0	100.00	0.00	0.00	0.00	58
	4	28	0	0.00	2402	2223	0	0	100.00
	8	10	0	0.00	2402	2223	0	0	100.00	0.00	0.00	0.00	57
	8	30	0	0.00	2403	2223	0	0	100.00
	9	12	0	0.00	2398	2223	0	0	100.00	0.00	0.00	0.00	57
	9	32	0	0.00	2401	2223	0	0	100.00
	10	14	0	0.00	2402	2223	0	0	100.00	0.00	0.00	0.00	56
	10	34	0	0.00	2402	2223	0	0	100.00
	11	16	0	0.00	2401	2223	0	0	100.00	0.00	0.00	0.00	56
	11	36	0	0.00	2402	2223	0	0	100.00
	12	18	0	0.00	2401	2223	0	0	100.00	0.00	0.00	0.00	56
	12	38	0	0.00	2401	2223	0	0	100.00
	0	1	2406	100.00	2400	2205	0	0	0.00	0.00	0.00	0.00	57	60	29.45	5.59	0.00	0.00
	0	21	2397	100.00	2400	2197	0	0	0.00
	1	3	0	0.00	2402	2197	0	0	100.00	0.00	0.00	0.00	55
	1	23	0	0.00	2401	2197	0	0	100.00
	2	5	0	0.00	2401	2197	0	0	100.00	0.00	0.00	0.00	54
	2	25	0	0.00	2402	2197	0	0	100.00
	3	7	0	0.00	2401	2197	0	0	100.00	0.00	0.00	0.00	54
	3	27	0	0.00	2401	2197	0	0	100.00
	4	9	0	0.00	2400	2197	0	0	100.00	0.00	0.00	0.00	54
	4	29	0	0.00	2401	2197	0	0	100.00
	8	11	0	0.00	2401	2197	0	0	100.00	0.00	0.00	0.00	54
	8	31	0	0.00	2403	2197	0	0	100.00
	9	13	0	0.00	2401	2197	0	0	100.00	0.00	0.00	0.00	53
	9	33	0	0.00	2402	2197	0	0	100.00
	10	15	0	0.00	2402	2197	0	0	100.00	0.00	0.00	0.00	55
	10	35	0	0.00	2401	2197	0	0	100.00
	11	17	0	0.00	2401	2197	0	0	100.00	0.00	0.00	0.00	54
	11	37	0	0.00	2401	2197	0	0	100.00
	12	19	0	0.00	2400	2197	0	0	100.00	0.00	0.00	0.00	55
	12	39	0	0.02	2400	2197	0	0	99.98

+++ network-latency +++
5.001831 sec
	Core	CPU	Avg_MHz	Busy%	Bzy_MHz	TSC_MHz	IRQ	SMI	CPU%c1	CPU%c3	CPU%c6	CPU%c7	CoreTmp	PkgTmp	PkgWatt	RAMWatt	PKG_%	RAM_%
	-	-	242	10.04	2400	2208	0	0	89.96	0.00	0.00	0.00	59	65	64.71	9.80	0.00	0.00
	0	0	2427	100.00	2400	2224	0	0	0.00	0.00	0.00	0.00	59	65	35.21	4.23	0.00	0.00
	0	20	2420	100.00	2400	2219	0	0	0.00
	1	2	26	1.07	2400	2219	0	0	98.93	0.00	0.00	0.00	57
	1	22	0	0.00	2404	2219	0	0	100.00
	2	4	0	0.00	2401	2219	0	0	100.00	0.00	0.00	0.00	57
	2	24	0	0.00	2401	2219	0	0	100.00
	3	6	0	0.00	2401	2219	0	0	100.00	0.00	0.00	0.00	58
	3	26	0	0.00	2401	2219	0	0	100.00
	4	8	0	0.00	2401	2219	0	0	100.00	0.00	0.00	0.00	58
	4	28	0	0.00	2402	2219	0	0	100.00
	8	10	0	0.00	2402	2219	0	0	100.00	0.00	0.00	0.00	57
	8	30	0	0.00	2401	2219	0	0	100.00
	9	12	0	0.00	2402	2219	0	0	100.00	0.00	0.00	0.00	56
	9	32	0	0.00	2402	2219	0	0	100.00
	10	14	0	0.00	2402	2219	0	0	100.00	0.00	0.00	0.00	57
	10	34	0	0.00	2402	2219	0	0	100.00
	11	16	0	0.00	2401	2219	0	0	100.00	0.00	0.00	0.00	56
	11	36	0	0.00	2398	2219	0	0	100.00
	12	18	0	0.00	2401	2219	0	0	100.00	0.00	0.00	0.00	56
	12	38	0	0.00	2401	2219	0	0	100.00
	0	1	2399	100.00	2400	2199	0	0	0.00	0.00	0.00	0.00	58	61	29.50	5.57	0.00	0.00
	0	21	2397	100.00	2400	2197	0	0	0.00
	1	3	0	0.00	2402	2197	0	0	100.00	0.00	0.00	0.00	56
	1	23	0	0.00	2401	2197	0	0	100.00
	2	5	0	0.00	2401	2197	0	0	100.00	0.00	0.00	0.00	55
	2	25	0	0.00	2401	2197	0	0	100.00
	3	7	0	0.00	2401	2197	0	0	100.00	0.00	0.00	0.00	55
	3	27	0	0.00	2403	2197	0	0	100.00
	4	9	0	0.00	2401	2197	0	0	100.00	0.00	0.00	0.00	55
	4	29	0	0.00	2400	2197	0	0	100.00
	8	11	0	0.00	2402	2197	0	0	100.00	0.00	0.00	0.00	55
	8	31	0	0.00	2402	2197	0	0	100.00
	9	13	0	0.00	2401	2197	0	0	100.00	0.00	0.00	0.00	54
	9	33	0	0.00	2402	2197	0	0	100.00
	10	15	0	0.00	2401	2197	0	0	100.00	0.00	0.00	0.00	55
	10	35	0	0.00	2400	2197	0	0	100.00
	11	17	0	0.00	2401	2197	0	0	100.00	0.00	0.00	0.00	54
	11	37	0	0.00	2401	2197	0	0	100.00
	12	19	0	0.00	2401	2197	0	0	100.00	0.00	0.00	0.00	55
	12	39	0	0.02	2400	2197	0	0	99.98

+++ network-throughput +++
5.040841 sec
	Core	CPU	Avg_MHz	Busy%	Bzy_MHz	TSC_MHz	IRQ	SMI	CPU%c1	CPU%c3	CPU%c6	CPU%c7	CoreTmp	PkgTmp	PkgWatt	RAMWatt	PKG_%	RAM_%
	-	-	311	10.04	3100	2197	0	0	1.38	0.00	88.58	0.00	62	64	50.76	9.86	0.00	0.00
	0	0	3097	99.99	3100	2198	0	0	0.01	0.00	0.00	0.00	62	64	28.26	4.29	0.00	0.00
	0	20	3094	99.99	3100	2196	0	0	0.01
	1	2	41	1.32	3101	2196	0	0	26.71	0.00	71.97	0.00	56
	1	22	5	0.15	3105	2196	0	0	27.87
	2	4	0	0.00	2794	2196	0	0	0.01	0.00	99.99	0.00	55
	2	24	0	0.00	2732	2196	0	0	0.01
	3	6	0	0.00	2719	2196	0	0	0.01	0.00	99.99	0.00	55
	3	26	0	0.00	2665	2196	0	0	0.01
	4	8	0	0.00	2641	2196	0	0	0.01	0.00	99.99	0.00	56
	4	28	0	0.00	2607	2196	0	0	0.01
	8	10	0	0.00	2617	2196	0	0	0.01	0.00	99.99	0.00	56
	8	30	0	0.00	2633	2196	0	0	0.01
	9	12	0	0.00	2673	2196	0	0	0.01	0.00	99.99	0.00	54
	9	32	0	0.00	2658	2196	0	0	0.01
	10	14	0	0.00	2714	2196	0	0	0.01	0.00	99.99	0.00	54
	10	34	0	0.00	2687	2196	0	0	0.01
	11	16	0	0.00	2758	2196	0	0	0.01	0.00	99.99	0.00	54
	11	36	0	0.00	2804	2196	0	0	0.01
	12	18	0	0.00	2752	2196	0	0	0.01	0.00	99.99	0.00	54
	12	38	0	0.00	2752	2196	0	0	0.01
	0	1	3104	99.99	3100	2203	0	0	0.01	0.00	0.00	0.00	60	60	22.50	5.57	0.00	0.00
	0	21	3096	99.99	3100	2198	0	0	0.01
	1	3	0	0.00	2947	2198	0	0	0.01	0.00	99.99	0.00	54
	1	23	0	0.00	2843	2197	0	0	0.01
	2	5	0	0.00	2808	2197	0	0	0.01	0.00	99.99	0.00	53
	2	25	0	0.00	2723	2197	0	0	0.01
	3	7	0	0.00	2711	2197	0	0	0.01	0.00	99.99	0.00	52
	3	27	0	0.00	2675	2197	0	0	0.01
	4	9	0	0.00	2650	2197	0	0	0.01	0.00	99.99	0.00	53
	4	29	0	0.00	2646	2197	0	0	0.01
	8	11	0	0.00	2629	2197	0	0	0.01	0.00	99.99	0.00	53
	8	31	0	0.00	2623	2197	0	0	0.01
	9	13	0	0.00	2702	2197	0	0	0.01	0.00	99.98	0.00	52
	9	33	0	0.00	2662	2197	0	0	0.01
	10	15	0	0.00	2714	2197	0	0	0.01	0.00	99.99	0.00	53
	10	35	0	0.00	2689	2197	0	0	0.01
	11	17	0	0.00	2773	2197	0	0	0.01	0.00	99.99	0.00	53
	11	37	0	0.00	2772	2197	0	0	0.01
	12	19	0	0.00	2833	2197	0	0	0.05	0.01	99.95	0.00	53
	12	39	0	0.01	2795	2197	0	0	0.03

+++ powersave +++
5.003208 sec
	Core	CPU	Avg_MHz	Busy%	Bzy_MHz	TSC_MHz	IRQ	SMI	CPU%c1	CPU%c3	CPU%c6	CPU%c7	CoreTmp	PkgTmp	PkgWatt	RAMWatt	PKG_%	RAM_%
	-	-	311	10.04	3100	2199	0	0	1.38	0.00	88.58	0.00	62	64	50.83	9.88	0.00	0.00
	0	0	3106	99.99	3100	2205	0	0	0.01	0.00	0.00	0.00	62	64	28.35	4.31	0.00	0.00
	0	20	3098	99.99	3100	2199	0	0	0.01
	1	2	41	1.32	3101	2199	0	0	26.70	0.00	71.98	0.00	55
	1	22	5	0.15	3105	2199	0	0	27.87
	2	4	0	0.00	2783	2199	0	0	0.01	0.00	99.99	0.00	55
	2	24	0	0.01	2984	2199	0	0	0.01
	3	6	0	0.00	2743	2199	0	0	0.01	0.00	99.99	0.00	55
	3	26	0	0.00	2687	2199	0	0	0.01
	4	8	0	0.00	2669	2199	0	0	0.01	0.00	99.99	0.00	55
	4	28	0	0.00	2634	2199	0	0	0.01
	8	10	0	0.00	2642	2199	0	0	0.01	0.00	99.99	0.00	55
	8	30	0	0.00	2641	2199	0	0	0.01
	9	12	0	0.00	2668	2199	0	0	0.01	0.00	99.99	0.00	54
	9	32	0	0.00	2611	2199	0	0	0.01
	10	14	0	0.00	2666	2199	0	0	0.01	0.00	99.99	0.00	54
	10	34	0	0.00	2638	2199	0	0	0.01
	11	16	0	0.00	2725	2199	0	0	0.01	0.00	99.99	0.00	54
	11	36	0	0.00	2829	2199	0	0	0.02
	12	18	0	0.00	2795	2199	0	0	0.01	0.00	99.99	0.00	54
	12	38	0	0.00	2794	2199	0	0	0.01
	0	1	3097	99.99	3100	2198	0	0	0.01	0.00	0.00	0.00	60	60	22.48	5.57	0.00	0.00
	0	21	3097	99.99	3100	2198	0	0	0.01
	1	3	0	0.00	3057	2198	0	0	0.01	0.00	99.99	0.00	54
	1	23	0	0.00	3014	2198	0	0	0.01
	2	5	0	0.00	2883	2198	0	0	0.01	0.00	99.99	0.00	53
	2	25	0	0.00	2872	2198	0	0	0.01
	3	7	0	0.01	2884	2198	0	0	0.01	0.01	99.97	0.00	53
	3	27	0	0.00	2818	2198	0	0	0.01
	4	9	0	0.00	2766	2198	0	0	0.01	0.00	99.99	0.00	53
	4	29	0	0.00	2691	2198	0	0	0.01
	8	11	0	0.00	2693	2198	0	0	0.01	0.00	99.98	0.00	54
	8	31	0	0.00	2637	2198	0	0	0.01
	9	13	0	0.00	2590	2198	0	0	0.01	0.00	99.99	0.00	52
	9	33	0	0.00	2598	2198	0	0	0.01
	10	15	0	0.00	2617	2198	0	0	0.01	0.00	99.99	0.00	53
	10	35	0	0.00	2640	2198	0	0	0.01
	11	17	0	0.00	2679	2198	0	0	0.01	0.00	99.99	0.00	52
	11	37	0	0.00	2644	2197	0	0	0.01
	12	19	0	0.00	2739	2197	0	0	0.05	0.00	99.95	0.00	53
	12	39	0	0.01	2942	2197	0	0	0.03

+++ throughput-performance +++
5.054865 sec
	Core	CPU	Avg_MHz	Busy%	Bzy_MHz	TSC_MHz	IRQ	SMI	CPU%c1	CPU%c3	CPU%c6	CPU%c7	CoreTmp	PkgTmp	PkgWatt	RAMWatt	PKG_%	RAM_%
	-	-	311	10.04	3100	2201	0	0	1.38	0.00	88.58	0.00	62	64	50.89	9.85	0.00	0.00
	0	0	3109	99.99	3100	2207	0	0	0.01	0.00	0.00	0.00	62	64	28.38	4.30	0.00	0.00
	0	20	3105	99.99	3100	2204	0	0	0.01
	1	2	41	1.32	3101	2204	0	0	26.73	0.00	71.95	0.00	56
	1	22	5	0.15	3105	2204	0	0	27.90
	2	4	0	0.00	2783	2204	0	0	0.01	0.00	99.99	0.00	55
	2	24	0	0.00	2707	2204	0	0	0.01
	3	6	0	0.00	2710	2204	0	0	0.01	0.00	99.99	0.00	55
	3	26	0	0.00	2658	2204	0	0	0.01
	4	8	0	0.00	2642	2204	0	0	0.01	0.00	99.99	0.00	55
	4	28	0	0.00	2608	2204	0	0	0.01
	8	10	0	0.00	2608	2204	0	0	0.01	0.00	99.99	0.00	55
	8	30	0	0.00	2644	2204	0	0	0.01
	9	12	0	0.00	2670	2204	0	0	0.01	0.00	99.99	0.00	55
	9	32	0	0.00	2640	2204	0	0	0.01
	10	14	0	0.00	2724	2204	0	0	0.01	0.00	99.98	0.00	54
	10	34	0	0.00	2696	2204	0	0	0.01
	11	16	0	0.00	2760	2204	0	0	0.01	0.00	99.99	0.00	54
	11	36	0	0.00	2801	2204	0	0	0.01
	12	18	0	0.00	2772	2204	0	0	0.01	0.00	99.99	0.00	54
	12	38	0	0.00	2765	2204	0	0	0.01
	0	1	3097	99.99	3100	2198	0	0	0.01	0.00	0.00	0.00	60	60	22.51	5.56	0.00	0.00
	0	21	3096	99.99	3100	2197	0	0	0.01
	1	3	0	0.00	2951	2197	0	0	0.01	0.00	99.99	0.00	55
	1	23	0	0.00	2843	2197	0	0	0.01
	2	5	0	0.00	2811	2197	0	0	0.01	0.00	99.99	0.00	53
	2	25	0	0.00	2722	2197	0	0	0.01
	3	7	0	0.00	2724	2197	0	0	0.01	0.00	99.99	0.00	53
	3	27	0	0.00	2671	2197	0	0	0.01
	4	9	0	0.00	2646	2197	0	0	0.01	0.00	99.99	0.00	54
	4	29	0	0.00	2645	2197	0	0	0.01
	8	11	0	0.00	2624	2197	0	0	0.01	0.00	99.99	0.00	54
	8	31	0	0.00	2641	2197	0	0	0.01
	9	13	0	0.00	2679	2197	0	0	0.01	0.00	99.98	0.00	52
	9	33	0	0.00	2664	2197	0	0	0.01
	10	15	0	0.00	2754	2197	0	0	0.01	0.00	99.99	0.00	53
	10	35	0	0.00	2776	2197	0	0	0.01
	11	17	0	0.00	2761	2197	0	0	0.01	0.00	99.99	0.00	52
	11	37	0	0.00	2781	2197	0	0	0.01
	12	19	0	0.00	2813	2197	0	0	0.05	0.00	99.95	0.00	53
	12	39	0	0.02	2842	2197	0	0	0.03

+++ virtual-guest +++
5.055904 sec
	Core	CPU	Avg_MHz	Busy%	Bzy_MHz	TSC_MHz	IRQ	SMI	CPU%c1	CPU%c3	CPU%c6	CPU%c7	CoreTmp	PkgTmp	PkgWatt	RAMWatt	PKG_%	RAM_%
	-	-	310	10.04	3100	2194	0	0	1.39	0.00	88.58	0.00	62	64	50.66	9.79	0.00	0.00
	0	0	3088	99.99	3100	2192	0	0	0.01	0.00	0.00	0.00	62	64	28.20	4.24	0.00	0.00
	0	20	3086	99.99	3100	2190	0	0	0.01
	1	2	41	1.33	3101	2190	0	0	26.95	0.00	71.72	0.00	56
	1	22	5	0.15	3105	2190	0	0	28.13
	2	4	0	0.00	2793	2190	0	0	0.01	0.00	99.99	0.00	55
	2	24	0	0.00	2719	2190	0	0	0.01
	3	6	0	0.00	2723	2190	0	0	0.01	0.00	99.99	0.00	55
	3	26	0	0.00	2658	2190	0	0	0.01
	4	8	0	0.00	2643	2190	0	0	0.01	0.00	99.99	0.00	56
	4	28	0	0.00	2606	2190	0	0	0.01
	8	10	0	0.00	2613	2190	0	0	0.01	0.00	99.99	0.00	55
	8	30	0	0.00	2638	2190	0	0	0.01
	9	12	0	0.00	2655	2190	0	0	0.01	0.00	99.98	0.00	54
	9	32	0	0.00	2660	2190	0	0	0.01
	10	14	0	0.00	2708	2190	0	0	0.01	0.00	99.99	0.00	54
	10	34	0	0.00	2682	2190	0	0	0.01
	11	16	0	0.00	2744	2190	0	0	0.01	0.00	99.99	0.00	54
	11	36	0	0.00	2775	2190	0	0	0.01
	12	18	0	0.00	2753	2190	0	0	0.01	0.00	99.99	0.00	54
	12	38	0	0.00	2766	2190	0	0	0.01
	0	1	3093	99.99	3100	2195	0	0	0.01	0.00	0.00	0.00	60	61	22.45	5.54	0.00	0.00
	0	21	3096	99.99	3100	2197	0	0	0.01
	1	3	0	0.00	2951	2197	0	0	0.01	0.00	99.99	0.00	54
	1	23	0	0.00	2849	2197	0	0	0.01
	2	5	0	0.00	2800	2197	0	0	0.01	0.00	99.99	0.00	53
	2	25	0	0.00	2730	2197	0	0	0.01
	3	7	0	0.00	2708	2197	0	0	0.01	0.00	99.99	0.00	53
	3	27	0	0.00	2678	2197	0	0	0.01
	4	9	0	0.00	2649	2197	0	0	0.01	0.00	99.99	0.00	54
	4	29	0	0.00	2611	2197	0	0	0.01
	8	11	0	0.00	2633	2197	0	0	0.01	0.00	99.99	0.00	54
	8	31	0	0.00	2642	2197	0	0	0.01
	9	13	0	0.00	2714	2197	0	0	0.01	0.00	99.98	0.00	52
	9	33	0	0.00	2688	2197	0	0	0.01
	10	15	0	0.00	2763	2197	0	0	0.01	0.00	99.99	0.00	53
	10	35	0	0.00	2782	2197	0	0	0.01
	11	17	0	0.00	2743	2197	0	0	0.01	0.00	99.99	0.00	52
	11	37	0	0.00	2775	2197	0	0	0.01
	12	19	0	0.00	2809	2197	0	0	0.05	0.00	99.95	0.00	53
	12	39	0	0.01	2802	2197	0	0	0.04

+++ virtual-host +++
5.038866 sec
	Core	CPU	Avg_MHz	Busy%	Bzy_MHz	TSC_MHz	IRQ	SMI	CPU%c1	CPU%c3	CPU%c6	CPU%c7	CoreTmp	PkgTmp	PkgWatt	RAMWatt	PKG_%	RAM_%
	-	-	311	10.04	3100	2198	0	0	1.38	0.00	88.58	0.00	62	64	50.83	9.81	0.00	0.00
	0	0	3098	99.99	3100	2199	0	0	0.01	0.00	0.00	0.00	62	64	28.28	4.25	0.00	0.00
	0	20	3097	99.99	3100	2198	0	0	0.01
	1	2	41	1.32	3101	2198	0	0	26.69	0.00	71.99	0.00	55
	1	22	5	0.15	3105	2198	0	0	27.86
	2	4	0	0.00	2800	2198	0	0	0.01	0.00	99.99	0.00	55
	2	24	0	0.00	2728	2198	0	0	0.01
	3	6	0	0.00	2718	2198	0	0	0.01	0.00	99.99	0.00	55
	3	26	0	0.00	2676	2198	0	0	0.01
	4	8	0	0.00	2639	2198	0	0	0.01	0.00	99.99	0.00	55
	4	28	0	0.00	2619	2198	0	0	0.01
	8	10	0	0.00	2658	2198	0	0	0.01	0.00	99.99	0.00	55
	8	30	0	0.00	2644	2198	0	0	0.01
	9	12	0	0.00	2706	2198	0	0	0.01	0.00	99.98	0.00	54
	9	32	0	0.00	2682	2198	0	0	0.01
	10	14	0	0.00	2727	2198	0	0	0.01	0.00	99.99	0.00	54
	10	34	0	0.00	2694	2198	0	0	0.01
	11	16	0	0.00	2770	2198	0	0	0.01	0.00	99.99	0.00	54
	11	36	0	0.00	2802	2198	0	0	0.01
	12	18	0	0.00	2774	2198	0	0	0.01	0.00	99.99	0.00	53
	12	38	0	0.00	2768	2198	0	0	0.01
	0	1	3099	99.99	3100	2199	0	0	0.01	0.00	0.00	0.00	60	60	22.55	5.56	0.00	0.00
	0	21	3096	99.99	3100	2197	0	0	0.01
	1	3	0	0.00	2953	2197	0	0	0.01	0.00	99.99	0.00	55
	1	23	0	0.00	2854	2197	0	0	0.01
	2	5	0	0.00	2819	2197	0	0	0.01	0.00	99.99	0.00	54
	2	25	0	0.00	2731	2197	0	0	0.01
	3	7	0	0.00	2712	2197	0	0	0.01	0.00	99.99	0.00	53
	3	27	0	0.00	2681	2197	0	0	0.01
	4	9	0	0.00	2646	2197	0	0	0.01	0.00	99.99	0.00	54
	4	29	0	0.00	2639	2197	0	0	0.01
	8	11	0	0.00	2619	2197	0	0	0.01	0.00	99.99	0.00	54
	8	31	0	0.00	2641	2197	0	0	0.01
	9	13	0	0.00	2696	2197	0	0	0.01	0.00	99.98	0.00	52
	9	33	0	0.00	2665	2197	0	0	0.01
	10	15	0	0.00	2760	2197	0	0	0.01	0.00	99.99	0.00	54
	10	35	0	0.00	2776	2197	0	0	0.01
	11	17	0	0.00	2761	2197	0	0	0.01	0.00	99.99	0.00	53
	11	37	0	0.00	2779	2197	0	0	0.01
	12	19	0	0.00	2800	2197	0	0	0.05	0.00	99.95	0.00	53
	12	39	0	0.02	2831	2197	0	0	0.03
[root@overcloud-compute-0 ~]#
Comment 10 Andreas Karis 2018-03-17 04:03:18 UTC 
Finally, one can easily create ones own tuned profile with C0, the question though is if this is desired.

There's probably a reason why latency-performance started using C1, and all other profiles are based upon latency performance.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/performance_tuning_guide/main-analyzeperf-tuned
~~~
latency-performance
    A server profile for typical latency performance tuning. This profile disables dynamic tuning mechanisms and transparent hugepages. It uses the performance governer for p-states through cpuspeed, and sets the I/O scheduler to deadline. Additionally, in Red Hat Enterprise Linux 6.5 and later, the profile requests a cpu_dma_latency value of 1. In Red Hat Enterprise Linux 6.4 and earlier, cpu_dma_latency requested a value of 0. 
~~~

Anyway, here's the way to go to use cpu-partitioning locked to C0:
~~~
[root@overcloud-compute-0 ~]# cat /etc/tuned/cpu-partitioning-disable-cstates/tuned.conf
[main]
include=cpu-partitioning

[cpu]
force_latency=0
~~~

~~~
[root@overcloud-compute-0 ~]# tuned-adm profile cpu-partitioning
[root@overcloud-compute-0 ~]# turbostat sleep 5
5.001639 sec
	Core	CPU	Avg_MHz	Busy%	Bzy_MHz	TSC_MHz	IRQ	SMI	CPU%c1	CPU%c3	CPU%c6	CPU%c7	CoreTmp	PkgTmp	PkgWatt	RAMWatt	PKG_%	RAM_%
	-	-	242	10.04	2400	2209	0	0	89.96	0.00	0.00	0.00	61	65	64.87	9.67	0.00	0.00
	0	0	2430	100.00	2400	2228	0	0	0.00	0.00	0.00	0.00	59	65	35.04	3.92	0.00	0.00
	0	20	2422	100.00	2400	2220	0	0	0.00
	1	2	26	1.06	2400	2220	0	0	98.94	0.00	0.00	0.00	57
	1	22	0	0.00	2401	2220	0	0	100.00
	2	4	0	0.00	2400	2220	0	0	100.00	0.00	0.00	0.00	57
	2	24	0	0.00	2400	2220	0	0	100.00
	3	6	0	0.00	2401	2220	0	0	100.00	0.00	0.00	0.00	58
	3	26	0	0.00	2402	2220	0	0	100.00
	4	8	0	0.00	2402	2220	0	0	100.00	0.00	0.00	0.00	57
	4	28	0	0.00	2400	2220	0	0	100.00
	8	10	0	0.00	2401	2220	0	0	100.00	0.00	0.00	0.00	57
	8	30	0	0.00	2403	2220	0	0	100.00
	9	12	0	0.00	2402	2220	0	0	100.00	0.00	0.00	0.00	57
	9	32	0	0.00	2403	2220	0	0	100.00
	10	14	0	0.00	2401	2220	0	0	100.00	0.00	0.00	0.00	56
	10	34	0	0.00	2401	2220	0	0	100.00
	11	16	0	0.00	2401	2220	0	0	100.00	0.00	0.00	0.00	56
	11	36	0	0.00	2402	2220	0	0	100.00
	12	18	0	0.00	2402	2220	0	0	100.00	0.00	0.00	0.00	56
	12	38	0	0.00	2402	2220	0	0	100.00
	0	1	2403	100.00	2400	2203	0	0	0.00	0.00	0.00	0.00	61	64	29.83	5.75	0.00	0.00
	0	21	2397	100.00	2400	2198	0	0	0.00
	1	3	0	0.00	2402	2198	0	0	100.00	0.00	0.00	0.00	59
	1	23	0	0.00	2401	2198	0	0	100.00
	2	5	0	0.00	2401	2198	0	0	100.00	0.00	0.00	0.00	59
	2	25	0	0.00	2401	2198	0	0	100.00
	3	7	0	0.00	2401	2198	0	0	100.00	0.00	0.00	0.00	58
	3	27	0	0.00	2385	2198	0	0	100.00
	4	9	0	0.00	2402	2197	0	0	100.00	0.00	0.00	0.00	58
	4	29	0	0.00	2403	2197	0	0	100.00
	8	11	0	0.00	2402	2197	0	0	100.00	0.00	0.00	0.00	58
	8	31	0	0.00	2401	2197	0	0	100.00
	9	13	0	0.00	2400	2197	0	0	100.00	0.00	0.00	0.00	57
	9	33	0	0.00	2401	2197	0	0	100.00
	10	15	0	0.00	2402	2197	0	0	100.00	0.00	0.00	0.00	59
	10	35	0	0.00	2401	2197	0	0	100.00
	11	17	0	0.00	2401	2197	0	0	100.00	0.00	0.00	0.00	57
	11	37	0	0.00	2401	2197	0	0	100.00
	12	19	0	0.00	2400	2197	0	0	100.00	0.00	0.00	0.00	58
	12	39	0	0.02	2400	2197	0	0	99.98
[root@overcloud-compute-0 ~]# tuned-adm profile cpu-partitioning-disable-cstates
[root@overcloud-compute-0 ~]# turbostat sleep 5
5.001900 sec
	Core	CPU	Avg_MHz	Busy%	Bzy_MHz	TSC_MHz	IRQ	SMI	CPU%c1	CPU%c3	CPU%c6	CPU%c7	CoreTmp	PkgTmp	PkgWatt	RAMWatt	PKG_%	RAM_%
	-	-	2411	100.00	2400	2210	0	0	0.00	0.00	0.00	0.00	63	67	86.52	9.61	0.00	0.00
	0	0	2426	100.00	2400	2224	0	0	0.00	0.00	0.00	0.00	61	67	45.71	3.90	0.00	0.00
	0	20	2425	100.00	2400	2223	0	0	0.00
	1	2	2425	100.00	2400	2223	0	0	0.00	0.00	0.00	0.00	61
	1	22	2425	100.00	2400	2223	0	0	0.00
	2	4	2425	100.00	2400	2223	0	0	0.00	0.00	0.00	0.00	62
	2	24	2425	100.00	2400	2223	0	0	0.00
	3	6	2425	100.00	2400	2223	0	0	0.00	0.00	0.00	0.00	61
	3	26	2425	100.00	2400	2223	0	0	0.00
	4	8	2425	100.00	2400	2223	0	0	0.00	0.00	0.00	0.00	61
	4	28	2425	100.00	2400	2223	0	0	0.00
	8	10	2425	100.00	2400	2223	0	0	0.00	0.00	0.00	0.00	60
	8	30	2425	100.00	2400	2223	0	0	0.00
	9	12	2425	100.00	2400	2223	0	0	0.00	0.00	0.00	0.00	61
	9	32	2425	100.00	2400	2223	0	0	0.00
	10	14	2425	100.00	2400	2223	0	0	0.00	0.00	0.00	0.00	61
	10	34	2425	100.00	2400	2223	0	0	0.00
	11	16	2425	100.00	2400	2223	0	0	0.00	0.00	0.00	0.00	61
	11	36	2425	100.00	2400	2223	0	0	0.00
	12	18	2425	100.00	2400	2223	0	0	0.00	0.00	0.00	0.00	59
	12	38	2425	100.00	2400	2223	0	0	0.00
	0	1	2403	100.00	2400	2202	0	0	0.00	0.00	0.00	0.00	63	65	40.81	5.71	0.00	0.00
	0	21	2397	100.00	2400	2198	0	0	0.00
	1	3	2397	100.00	2400	2198	0	0	0.00	0.00	0.00	0.00	63
	1	23	2397	100.00	2400	2198	0	0	0.00
	2	5	2397	100.00	2400	2198	0	0	0.00	0.00	0.00	0.00	63
	2	25	2397	100.00	2400	2198	0	0	0.00
	3	7	2397	100.00	2400	2198	0	0	0.00	0.00	0.00	0.00	61
	3	27	2397	100.00	2400	2198	0	0	0.00
	4	9	2397	100.00	2400	2198	0	0	0.00	0.00	0.00	0.00	62
	4	29	2397	100.00	2400	2198	0	0	0.00
	8	11	2397	100.00	2400	2197	0	0	0.00	0.00	0.00	0.00	61
	8	31	2397	100.00	2400	2197	0	0	0.00
	9	13	2397	100.00	2400	2197	0	0	0.00	0.00	0.00	0.00	61
	9	33	2397	100.00	2400	2197	0	0	0.00
	10	15	2397	100.00	2400	2197	0	0	0.00	0.00	0.00	0.00	62
	10	35	2397	100.00	2400	2197	0	0	0.00
	11	17	2397	100.00	2400	2197	0	0	0.00	0.00	0.00	0.00	62
	11	37	2397	100.00	2400	2197	0	0	0.00
	12	19	2397	100.00	2400	2197	0	0	0.00	0.00	0.00	0.00	62
	12	39	2397	100.00	2400	2197	0	0	0.00
[root@overcloud-compute-0 ~]# grep -i affi /etc/systemd/system.conf 
#CPUAffinity=1 2
CPUAffinity=0 1 20 21
~~~
Comment 14 Ondřej Lysoněk 2018-06-05 14:14:46 UTC 
(In reply to Andreas Karis from comment #8)
> If I follow Franck's recommendation, then we should not reach C1 for the
> cpu-partitioning profile. Or we should have a switch there that allows us to
> set /usr/libexec/tuned/pmqos-static.py cpu_dma_latency=0

We could add a configuration file to modify the latency in the underlying latency-performance profile, e.g.:
$ cat /etc/tuned/latency-performance-variables.conf
cpu_dma_latency=1

But given that doing the same thing by overriding the profile is fairly simple, maybe it's not worth it:
> Anyway, here's the way to go to use cpu-partitioning locked to C0:
> [root@overcloud-compute-0 ~]# cat
> /etc/tuned/cpu-partitioning-disable-cstates/tuned.conf
> [main]
> include=cpu-partitioning
> 
> [cpu]
> force_latency=0

As long as it's documented, it should suffice. But I'm fine to add the configuration file.
Comment 15 Jaroslav Škarvada 2018-06-21 12:07:31 UTC 
AFAIK the acpi_idle driver doesn't allow locking to C0, i.e. the effect of the following two boot command lines is the same:
 processor.max_cstate=0 intel_idle.max_cstate=0
 processor.max_cstate=1 intel_idle.max_cstate=0

i.e.:
intel_idle.max_cstate=0 disables intel_idle driver and acpi_idle driver is used
processor.max_cstate=0 if acpi_idle driver is used, it silently changes max_cstate=0 to max_cstate=1, so C1 is used always.

Adding 'idle=poll' will add polling loop which will not allow CPU to sleep, thus it will appear as locked in C0. But this will significantly increase power consumption and the CPU may become very hot. In the kernel documentation there is written that this setting is "not recommended" (idle=poll in e.g. [1]).

Tuned uses different mechanism for locking C-states - the PMQoS. It seems it allows locking CPU to C0, but if the acpi_idle driver doesn't allow it, there is probably good reason for not doing it. There is also a possibility that this behavior is a bug in the acpi_idle driver and that it shouldn't allow PMQoS to force C0.

In Tuned we switched to C1 due to bug 1013085. IIRC there was some feedback from Intel regarding this, but I can't remember more at the moment. Jeremy can probably say more about this change.

If you really need locking to C0, you can customize the Tuned profile. It's easy and documented - there is article about it [2].

Thus I tend to close this bugzilla as wontfix.

[1] https://www.kernel.org/doc/html/v4.17/admin-guide/kernel-parameters.html
[2] https://access.redhat.com/solutions/3384811
Comment 16 Red Hat Bugzilla Rules Engine 2018-06-21 12:28:27 UTC 
Development Management has reviewed and declined this request. You may appeal this decision by reopening this request.
Note You need to log in before you can comment on or make changes to this bug.