Prior to this update, irqbalance could not accurately determine the NUMA node it was local to or the device to which an IRQ was sent. The kernel affinity_hint values were created to work around this issue. With this update, irqbalance is now capable of parsing all information about an IRQ provided by the sysfs() function. IRQ balancing now works correctly, and the affinity_hint values are now ignored by default not to distort the irqbalance functionality.
Description of problem:
irqbalance oneshot not working with ixgbe
[root@dell-pet410-01 ~]# uname -a
Linux dell-pet410-01.lab.bos.redhat.com 2.6.32-431.5.1.el6.x86_64 #1 SMP Fri Jan 10 14:46:43 EST 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@dell-pet410-01 ~]# dmidecode | grep -i Product
Product Name: PowerEdge T410
Product Name: 0M638F
[root@dell-pet410-01 ~]# ethtool -i p2p1
driver: ixgbe
version: 3.15.1-k
firmware-version: 0x0f220001
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
Version-Release number of selected component (if applicable):
[root@dell-pet410-01 ~]# ethtool -i p2p1
driver: ixgbe
version: 3.15.1-k
firmware-version: 0x0f220001
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
How reproducible:
before changes
for i in `ls /proc/irq`; do echo -n $(grep " ${i}:" /proc/interrupts | awk '{print $NF}'); echo -n ": "; cat /proc/irq/${i}/smp_affinity; done
timer: ffffffff
i8042: 00005555
: 00005555
p2p1-TxRx-0: 00000001
: 0000ffff
p2p1-TxRx-1: 00000002
p2p1-TxRx-2: 00000004
p2p1-TxRx-3: 00000008
p2p1-TxRx-4: 00000010
p2p1-TxRx-5: 00000020
p2p1-TxRx-6: 00000040
p2p1-TxRx-7: 00000080
p2p1-TxRx-8: 00000100
p2p1-TxRx-9: 00000200
p2p1-TxRx-10: 00000400
i8042: 00005555
p2p1-TxRx-11: 00000800
p2p1-TxRx-12: 00001000
p2p1-TxRx-13: 00002000
p2p1-TxRx-14: 00004000
p2p1-TxRx-15: 00008000
p2p1: 00008000
mlx4-comp-0@pci:0000:04:00.0: 00000100
mlx4-comp-1@pci:0000:04:00.0: 00000200
mlx4-comp-2@pci:0000:04:00.0: 00000004
mlx4-async@pci:0000:04:00.0: 00000008
: 0000ffff
: 0000ffff
em1-0: 00000040
em1-1: 00000080
: 0000ffff
em1-2: 00000020
em1-3: 00000002
em1-4: 00000400
em1-5: 00000800
em1-6: 00001000
em1-7: 00002000
uhci_hcd:usb3: 00000202
uhci_hcd:usb4: 00000404
ehci_hcd:usb1: 00000808
: ffffffff
uhci_hcd:usb8: 00004040
uhci_hcd:usb7: 00002020
ata_piix: 00000404
ata_piix: 00000202
IR-IO-APIC-edge: 00005555
ioc0: 00000101
serial: 00005555
: 00000808
dmar0: 00000001
: 0000ffff
pciehp: 00001010
: 0000ffff
rtc0: 00005555
acpi: 00005555
chkconfig irqbalance off
/etc/init.d/irqbalance stop
[root@dell-pet410-01 ~]# export IRQBALANCE_BANNED_CPUS=ffffffff,fffffffe
[root@dell-pet410-01 ~]# /usr/sbin/irqbalance -d -o
doesn't work
for i in `ls /proc/irq`; do echo -n $(grep " ${i}:" /proc/interrupts | awk '{print $NF}'); echo -n ": "; cat /proc/irq/${i}/smp_affinity; done
timer: ffffffff
i8042: 00005555
: 00000001
p2p1-TxRx-0: 00000001
: 0000ffff
p2p1-TxRx-1: 00000002
p2p1-TxRx-2: 00000004
p2p1-TxRx-3: 00000008
p2p1-TxRx-4: 00000010
p2p1-TxRx-5: 00000020
p2p1-TxRx-6: 00000040
p2p1-TxRx-7: 00000080
p2p1-TxRx-8: 00000100
p2p1-TxRx-9: 00000200
p2p1-TxRx-10: 00000400
i8042: 00005555
p2p1-TxRx-11: 00000800
p2p1-TxRx-12: 00001000
p2p1-TxRx-13: 00002000
p2p1-TxRx-14: 00004000
p2p1-TxRx-15: 00008000
p2p1: 00000001
mlx4-comp-0@pci:0000:04:00.0: 00000001
mlx4-comp-1@pci:0000:04:00.0: 00000001
mlx4-comp-2@pci:0000:04:00.0: 00000001
mlx4-async@pci:0000:04:00.0: 00000001
: 0000ffff
: 0000ffff
em1-0: 00000001
em1-1: 00000001
: 0000ffff
em1-2: 00000001
em1-3: 00000001
em1-4: 00000001
em1-5: 00000001
em1-6: 00000001
em1-7: 00000001
uhci_hcd:usb3: 00000001
uhci_hcd:usb4: 00000001
ehci_hcd:usb1: 00000001
: ffffffff
uhci_hcd:usb8: 00000001
uhci_hcd:usb7: 00000001
ata_piix: 00000001
ata_piix: 00000001
IR-IO-APIC-edge: 00005555
ioc0: 00000001
serial: 00005555
: 00000001
dmar0: 00000001
: 0000ffff
pciehp: 00000001
: 0000ffff
rtc0: 00005555
acpi: 00005555
[
notice all the p2p1 queues don't change
Additional info:
We recently had some past BZs on this see below
Previously, the irqbalance daemon did not consider the NUMA node assignment for an IRQ (interrupt request) for the banned CPU set. Consequently, irqbalance set the affinity incorrectly when the IRQBALANCE_BANNED_IRQS variable was set to a single CPU. In addition, IRQs could not be assigned to a node that had no eligible CPUs. Node assignment has been restricted to nodes that have eligible CPUs as defined by the unbanned_cpus bitmask, thus fixing the bug. As a result, irqbalance now sets affinity properly, and IRQs are assigned to the respective nodes correctly. (BZ#1054590,BZ#1054591)
Prior to this update, the dependency of the irqbalance daemon was set incorrectly referring to a wrong kernel version. As a consequence, irqbalance could not balance IRQs on NUMA systems. With this update, the dependency has been fixed, and IRQs are now balanced correctly on NUMA systems. Note that users of irqbalance packages have to update kernel to 2.6.32-358.2.1 or later in order to use the irqbalance daemon in correct manner. (BZ#1055572, BZ#1055574)
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
http://rhn.redhat.com/errata/RHBA-2014-1504.html
Description of problem: irqbalance oneshot not working with ixgbe [root@dell-pet410-01 ~]# uname -a Linux dell-pet410-01.lab.bos.redhat.com 2.6.32-431.5.1.el6.x86_64 #1 SMP Fri Jan 10 14:46:43 EST 2014 x86_64 x86_64 x86_64 GNU/Linux [root@dell-pet410-01 ~]# dmidecode | grep -i Product Product Name: PowerEdge T410 Product Name: 0M638F [root@dell-pet410-01 ~]# ethtool -i p2p1 driver: ixgbe version: 3.15.1-k firmware-version: 0x0f220001 bus-info: 0000:03:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: no Version-Release number of selected component (if applicable): [root@dell-pet410-01 ~]# ethtool -i p2p1 driver: ixgbe version: 3.15.1-k firmware-version: 0x0f220001 bus-info: 0000:03:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: no How reproducible: before changes for i in `ls /proc/irq`; do echo -n $(grep " ${i}:" /proc/interrupts | awk '{print $NF}'); echo -n ": "; cat /proc/irq/${i}/smp_affinity; done timer: ffffffff i8042: 00005555 : 00005555 p2p1-TxRx-0: 00000001 : 0000ffff p2p1-TxRx-1: 00000002 p2p1-TxRx-2: 00000004 p2p1-TxRx-3: 00000008 p2p1-TxRx-4: 00000010 p2p1-TxRx-5: 00000020 p2p1-TxRx-6: 00000040 p2p1-TxRx-7: 00000080 p2p1-TxRx-8: 00000100 p2p1-TxRx-9: 00000200 p2p1-TxRx-10: 00000400 i8042: 00005555 p2p1-TxRx-11: 00000800 p2p1-TxRx-12: 00001000 p2p1-TxRx-13: 00002000 p2p1-TxRx-14: 00004000 p2p1-TxRx-15: 00008000 p2p1: 00008000 mlx4-comp-0@pci:0000:04:00.0: 00000100 mlx4-comp-1@pci:0000:04:00.0: 00000200 mlx4-comp-2@pci:0000:04:00.0: 00000004 mlx4-async@pci:0000:04:00.0: 00000008 : 0000ffff : 0000ffff em1-0: 00000040 em1-1: 00000080 : 0000ffff em1-2: 00000020 em1-3: 00000002 em1-4: 00000400 em1-5: 00000800 em1-6: 00001000 em1-7: 00002000 uhci_hcd:usb3: 00000202 uhci_hcd:usb4: 00000404 ehci_hcd:usb1: 00000808 : ffffffff uhci_hcd:usb8: 00004040 uhci_hcd:usb7: 00002020 ata_piix: 00000404 ata_piix: 00000202 IR-IO-APIC-edge: 00005555 ioc0: 00000101 serial: 00005555 : 00000808 dmar0: 00000001 : 0000ffff pciehp: 00001010 : 0000ffff rtc0: 00005555 acpi: 00005555 chkconfig irqbalance off /etc/init.d/irqbalance stop [root@dell-pet410-01 ~]# export IRQBALANCE_BANNED_CPUS=ffffffff,fffffffe [root@dell-pet410-01 ~]# /usr/sbin/irqbalance -d -o doesn't work for i in `ls /proc/irq`; do echo -n $(grep " ${i}:" /proc/interrupts | awk '{print $NF}'); echo -n ": "; cat /proc/irq/${i}/smp_affinity; done timer: ffffffff i8042: 00005555 : 00000001 p2p1-TxRx-0: 00000001 : 0000ffff p2p1-TxRx-1: 00000002 p2p1-TxRx-2: 00000004 p2p1-TxRx-3: 00000008 p2p1-TxRx-4: 00000010 p2p1-TxRx-5: 00000020 p2p1-TxRx-6: 00000040 p2p1-TxRx-7: 00000080 p2p1-TxRx-8: 00000100 p2p1-TxRx-9: 00000200 p2p1-TxRx-10: 00000400 i8042: 00005555 p2p1-TxRx-11: 00000800 p2p1-TxRx-12: 00001000 p2p1-TxRx-13: 00002000 p2p1-TxRx-14: 00004000 p2p1-TxRx-15: 00008000 p2p1: 00000001 mlx4-comp-0@pci:0000:04:00.0: 00000001 mlx4-comp-1@pci:0000:04:00.0: 00000001 mlx4-comp-2@pci:0000:04:00.0: 00000001 mlx4-async@pci:0000:04:00.0: 00000001 : 0000ffff : 0000ffff em1-0: 00000001 em1-1: 00000001 : 0000ffff em1-2: 00000001 em1-3: 00000001 em1-4: 00000001 em1-5: 00000001 em1-6: 00000001 em1-7: 00000001 uhci_hcd:usb3: 00000001 uhci_hcd:usb4: 00000001 ehci_hcd:usb1: 00000001 : ffffffff uhci_hcd:usb8: 00000001 uhci_hcd:usb7: 00000001 ata_piix: 00000001 ata_piix: 00000001 IR-IO-APIC-edge: 00005555 ioc0: 00000001 serial: 00005555 : 00000001 dmar0: 00000001 : 0000ffff pciehp: 00000001 : 0000ffff rtc0: 00005555 acpi: 00005555 [ notice all the p2p1 queues don't change Additional info: We recently had some past BZs on this see below Previously, the irqbalance daemon did not consider the NUMA node assignment for an IRQ (interrupt request) for the banned CPU set. Consequently, irqbalance set the affinity incorrectly when the IRQBALANCE_BANNED_IRQS variable was set to a single CPU. In addition, IRQs could not be assigned to a node that had no eligible CPUs. Node assignment has been restricted to nodes that have eligible CPUs as defined by the unbanned_cpus bitmask, thus fixing the bug. As a result, irqbalance now sets affinity properly, and IRQs are assigned to the respective nodes correctly. (BZ#1054590,BZ#1054591) Prior to this update, the dependency of the irqbalance daemon was set incorrectly referring to a wrong kernel version. As a consequence, irqbalance could not balance IRQs on NUMA systems. With this update, the dependency has been fixed, and IRQs are now balanced correctly on NUMA systems. Note that users of irqbalance packages have to update kernel to 2.6.32-358.2.1 or later in order to use the irqbalance daemon in correct manner. (BZ#1055572, BZ#1055574)