Bug 1010264

Summary: smp_affinity broken?
Product: [Fedora] Fedora Reporter: Markus Stockhausen <mst>
Component: kernelAssignee: Neil Horman <nhorman>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 19CC: gansalmon, itamar, jonathan, kernel-maint, madhu.chinakonda, marcelo.barbosa, nhorman, skottler
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-09-20 15:27:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Markus Stockhausen 2013-09-20 12:09:10 UTC 
Description of problem:

my knowledge about IRQ SMP affinity may be a little bit outdated but I found no newer documentation about the topic. In Fedora 19 I'm no longer able to change the interrupt affinity for my intel network card so it gets served by two or more CPUs:

Version-Release number of selected component (if applicable):

Fedora 19 kernel 3.l1.1

How reproducible:

100%

Steps to Reproduce:

1. stop irqbalance daemon
2. get interrupt of network card

[root@colovn1 ~]# cat /proc/interrupts | grep p1
 71:    2542650          0          0         55  PCI-MSI-edge      p49p1-rx-0
 72:         11    4324139          0          0  PCI-MSI-edge      p49p1-tx-0

3. set SMP affinity of IRQ 71 to CPU0 & CPU1

 echo 3 > /proc/irq/71/smp_affinity

4. transfer data over the network and watch the interrupt values. 
 
Actual results:

only CPU0 serves interrupts

Expected results:

CPU0 and CPU1 get interrupts of network card

Additional info:

/sys/class/net/p49p1/queues/rx-0/rps_cpus is not set:

cat /sys/class/net/p49p1/queues/rx-0/rps_cpus
00000000,00000000,00000000,00000000

Card is

[root@colovn1 ~]# lspci | grep -i net
01:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
02:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
Comment 1 Neil Horman 2013-09-20 15:27:55 UTC 
Setting affinity for a given irq to multiple cpus does not imply that irq will be equally serviced by those cpus (quite the opposite in fact).  Setting smp affinity just marks the respective cpus in the mask as elligible to service that cpu.  The actual decision as to which cpu handles the irq is made by hardware, and the hardware policy is to have a given irq serviced by the same cpu (as determined by a bios configured priority value for each cpu in the system).  Only if the highest priority cpu of the elligible cpu set in smp_affinity is busy (handling another interrupt), will another cpu in the set be selected.  This increases the likelyhood of L2 cache hits for instructions in the asserted irqs software handler, which is where the performance increase in setting smp affinity is found.

Unless you want to do something specfic for some very uncommon requirements, just run irqbalance, and the right things should happen.