Bug 960383

Assign a intel 82576 NIC card to guest and test SMP affinity. Also hits this problem. 

This NIC card supports MSI-X and the detaild PCI information is as following:
# lspci -vvv -s 00:03.0
00:03.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
	Subsystem: Intel Corporation Gigabit ET2 Quad Port Server Adapter
	Physical Slot: 3
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 32 bytes
	Interrupt: pin A routed to IRQ 11
	Region 0: Memory at f1020000 (32-bit, non-prefetchable) [size=128K]
	Region 1: Memory at f1400000 (32-bit, non-prefetchable) [size=4M]
	Region 2: I/O ports at c020 [size=32]
	Region 3: Memory at f1800000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [a0] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Latency L0 <4us, L1 <64us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 16ms to 55ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
		Vector table: BAR=3 offset=00000000
		PBA: BAR=3 offset=00002000
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit-
		Address: 00000000  Data: 0000
	Kernel driver in use: igb
	Kernel modules: igb

Where the assumption that without irqbalancer irq will be distributed is comming from? This is irqbalancer job to distribute interrupts. In some limited scenarious OS can configure interrupt as LowPrio and then it will be distributed by HW, but most often than not such config is not possible or OS chose to not use it. On my physical machins all afinity is set to ffff and no interrupts are distributed.

Gleb, is this a guest kernel issue?  Testing on a RHEL6.4 host, I find that an smp_affinity mask with multiple CPUs set does work with a RHEL6.4 guest, but does not work with a RHEL5 guest.  The name of the guest in comment 0 is rhel7, but I don't know if that reflects what's actually running.  Perhaps something broke upstream?

Hello, Gleb,
(In reply to Gleb Natapov from comment #5)
> Where the assumption that without irqbalancer irq will be distributed is
> comming from? 

I make irq affinity test refering to IRQ-affinity.txt in kernel-doc. About why to stop irqbalance service, mainly refer to some tech doc from internet such as: https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_MRG/2/html/Realtime_Tuning_Guide/sect-Realtime_Tuning_Guide-General_System_Tuning-Interrupt_and_Process_Binding.html

>On my physical machins all afinity is set to ffff and no interrupts are distributed.

I test two cards on physical machines, the result is: 
1) One supports MSI-X, I set ffff and interrupts can only be distributed to CPU0.
2) The other supports MSI, I set ffff and interrupts can be distributed to all cpus.

I assign the NIC card supporting MSI to rhel6.4 guest. 
On guest, I set ffff and interrupts can only be distributed to CPU0.
Add "pci=nomsi" to guest kernel line, set ffff and interrupts can only be distributed to CPU0. 


The host and guest kernel version is:
kernel-2.6.32-358.el6.x86_64
The qemu-kvm version is:
qemu-kvm-0.12.1.2-2.370.el6.x86_64

Best regards,
Ding Huiqing

(In reply to Alex Williamson from comment #6)
> Gleb, is this a guest kernel issue?  Testing on a RHEL6.4 host, I find that
> an smp_affinity mask with multiple CPUs set does work with a RHEL6.4 guest,
> but does not work with a RHEL5 guest.  The name of the guest in comment 0 is
> rhel7, but I don't know if that reflects what's actually running.  Perhaps
> something broke upstream?

I wouldn't call it an issue :) But yes, it depends on how guest kernel
configures interrupt controller. The only way HW can distribute interrupts
is by LowPrio mechanism and the mechanism is very limit and does not allow
distribution to arbitrary set of vcpus (if there is more then 8 cpus anyway).
Probably newer kernels stop even trying, but I cannot tell if this intentional
or a bug.

There is a reason irqbalacer exists.

(In reply to huiqingding from comment #7)
> Hello, Gleb,
> (In reply to Gleb Natapov from comment #5)
> > Where the assumption that without irqbalancer irq will be distributed is
> > comming from? 
> 
> I make irq affinity test refering to IRQ-affinity.txt in kernel-doc.
That document predates beginning of the git. This is ancient history.
 
>  About
> why to stop irqbalance service, mainly refer to some tech doc from internet
> such as:
> https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_MRG/2/
> html/Realtime_Tuning_Guide/sect-Realtime_Tuning_Guide-General_System_Tuning-
> Interrupt_and_Process_Binding.html
This document talks about manually binding interrupts to cpus, which works perfectly fine according to
your test if you do not expect interrupt to be distributed to multiple cpus.

> 
> >On my physical machins all afinity is set to ffff and no interrupts are distributed.
> 
> I test two cards on physical machines, the result is: 
> 1) One supports MSI-X, I set ffff and interrupts can only be distributed to
> CPU0.
> 2) The other supports MSI, I set ffff and interrupts can be distributed to
> all cpus.
It has nothing to do with MSI or MSI-X. What's machines HW (number of sockets/cores) and kernel versions.

This behavior is expected, "-cpu SandyBridge" enables x2apic and KVM emulates just "physical x2apic", which does not allow lowest priority delivery.

(We have lowest priority delivery in "flat" mode only; guest kernel with "nox2apic" parameter will likely use it.)

*** Bug 1029343 has been marked as a duplicate of this bug. ***