Bug 1079109 - irqbalance oneshot not working with ixgbe
Summary: irqbalance oneshot not working with ixgbe
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: irqbalance
Version: 6.6
Hardware: x86_64
OS: Unspecified
urgent
high
Target Milestone: rc
: ---
Assignee: Petr Holasek
QA Contact: Evan McNabb
URL:
Whiteboard:
: 1093105 (view as bug list)
Depends On:
Blocks: 994246 1093105 1093440 1093441
TreeView+ depends on / blocked
 
Reported: 2014-03-21 02:18 UTC by Dave Sullivan
Modified: 2018-12-05 17:48 UTC (History)
5 users (show)

Fixed In Version: irqbalance-1.0.4-9.el6
Doc Type: Bug Fix
Doc Text:
Prior to this update, irqbalance could not accurately determine the NUMA node it was local to or the device to which an IRQ was sent. The kernel affinity_hint values were created to work around this issue. With this update, irqbalance is now capable of parsing all information about an IRQ provided by the sysfs() function. IRQ balancing now works correctly, and the affinity_hint values are now ignored by default not to distort the irqbalance functionality.
Clone Of:
: 1093105 (view as bug list)
Environment:
Last Closed: 2014-10-14 07:10:45 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2014:1504 normal SHIPPED_LIVE irqbalance bug fix update 2014-10-14 01:28:06 UTC

Description Dave Sullivan 2014-03-21 02:18:57 UTC
Description of problem:

irqbalance oneshot not working with ixgbe

[root@dell-pet410-01 ~]# uname -a
Linux dell-pet410-01.lab.bos.redhat.com 2.6.32-431.5.1.el6.x86_64 #1 SMP Fri Jan 10 14:46:43 EST 2014 x86_64 x86_64 x86_64 GNU/Linux

[root@dell-pet410-01 ~]# dmidecode | grep -i Product
	Product Name: PowerEdge T410
	Product Name: 0M638F

[root@dell-pet410-01 ~]# ethtool -i p2p1
driver: ixgbe
version: 3.15.1-k
firmware-version: 0x0f220001
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no


Version-Release number of selected component (if applicable):


[root@dell-pet410-01 ~]# ethtool -i p2p1
driver: ixgbe
version: 3.15.1-k
firmware-version: 0x0f220001
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

How reproducible:

before changes

for i in `ls /proc/irq`; do echo -n $(grep " ${i}:" /proc/interrupts | awk '{print $NF}'); echo -n ": "; cat /proc/irq/${i}/smp_affinity; done

timer: ffffffff
i8042: 00005555
: 00005555
p2p1-TxRx-0: 00000001
: 0000ffff
p2p1-TxRx-1: 00000002
p2p1-TxRx-2: 00000004
p2p1-TxRx-3: 00000008
p2p1-TxRx-4: 00000010
p2p1-TxRx-5: 00000020
p2p1-TxRx-6: 00000040
p2p1-TxRx-7: 00000080
p2p1-TxRx-8: 00000100
p2p1-TxRx-9: 00000200
p2p1-TxRx-10: 00000400
i8042: 00005555
p2p1-TxRx-11: 00000800
p2p1-TxRx-12: 00001000
p2p1-TxRx-13: 00002000
p2p1-TxRx-14: 00004000
p2p1-TxRx-15: 00008000
p2p1: 00008000
mlx4-comp-0@pci:0000:04:00.0: 00000100
mlx4-comp-1@pci:0000:04:00.0: 00000200
mlx4-comp-2@pci:0000:04:00.0: 00000004
mlx4-async@pci:0000:04:00.0: 00000008
: 0000ffff
: 0000ffff
em1-0: 00000040
em1-1: 00000080
: 0000ffff
em1-2: 00000020
em1-3: 00000002
em1-4: 00000400
em1-5: 00000800
em1-6: 00001000
em1-7: 00002000
uhci_hcd:usb3: 00000202
uhci_hcd:usb4: 00000404
ehci_hcd:usb1: 00000808
: ffffffff
uhci_hcd:usb8: 00004040
uhci_hcd:usb7: 00002020
ata_piix: 00000404
ata_piix: 00000202
IR-IO-APIC-edge: 00005555
ioc0: 00000101
serial: 00005555
: 00000808
dmar0: 00000001
: 0000ffff
pciehp: 00001010
: 0000ffff
rtc0: 00005555
acpi: 00005555

chkconfig irqbalance off
/etc/init.d/irqbalance stop

[root@dell-pet410-01 ~]# export IRQBALANCE_BANNED_CPUS=ffffffff,fffffffe

[root@dell-pet410-01 ~]# /usr/sbin/irqbalance -d -o

doesn't work

for i in `ls /proc/irq`; do echo -n $(grep " ${i}:" /proc/interrupts | awk '{print $NF}'); echo -n ": "; cat /proc/irq/${i}/smp_affinity; done
timer: ffffffff
i8042: 00005555
: 00000001
p2p1-TxRx-0: 00000001
: 0000ffff
p2p1-TxRx-1: 00000002
p2p1-TxRx-2: 00000004
p2p1-TxRx-3: 00000008
p2p1-TxRx-4: 00000010
p2p1-TxRx-5: 00000020
p2p1-TxRx-6: 00000040
p2p1-TxRx-7: 00000080
p2p1-TxRx-8: 00000100
p2p1-TxRx-9: 00000200
p2p1-TxRx-10: 00000400
i8042: 00005555
p2p1-TxRx-11: 00000800
p2p1-TxRx-12: 00001000
p2p1-TxRx-13: 00002000
p2p1-TxRx-14: 00004000
p2p1-TxRx-15: 00008000
p2p1: 00000001
mlx4-comp-0@pci:0000:04:00.0: 00000001
mlx4-comp-1@pci:0000:04:00.0: 00000001
mlx4-comp-2@pci:0000:04:00.0: 00000001
mlx4-async@pci:0000:04:00.0: 00000001
: 0000ffff
: 0000ffff
em1-0: 00000001
em1-1: 00000001
: 0000ffff
em1-2: 00000001
em1-3: 00000001
em1-4: 00000001
em1-5: 00000001
em1-6: 00000001
em1-7: 00000001
uhci_hcd:usb3: 00000001
uhci_hcd:usb4: 00000001
ehci_hcd:usb1: 00000001
: ffffffff
uhci_hcd:usb8: 00000001
uhci_hcd:usb7: 00000001
ata_piix: 00000001
ata_piix: 00000001
IR-IO-APIC-edge: 00005555
ioc0: 00000001
serial: 00005555
: 00000001
dmar0: 00000001
: 0000ffff
pciehp: 00000001
: 0000ffff
rtc0: 00005555
acpi: 00005555

[
notice all the p2p1 queues don't change


Additional info:

We recently had some past BZs on this see below

    Previously, the irqbalance daemon did not consider the NUMA node assignment for an IRQ (interrupt request) for the banned CPU set. Consequently, irqbalance set the affinity incorrectly when the IRQBALANCE_BANNED_IRQS variable was set to a single CPU. In addition, IRQs could not be assigned to a node that had no eligible CPUs. Node assignment has been restricted to nodes that have eligible CPUs as defined by the unbanned_cpus bitmask, thus fixing the bug. As a result, irqbalance now sets affinity properly, and IRQs are assigned to the respective nodes correctly. (BZ#1054590,BZ#1054591)

    Prior to this update, the dependency of the irqbalance daemon was set incorrectly referring to a wrong kernel version. As a consequence, irqbalance could not balance IRQs on NUMA systems. With this update, the dependency has been fixed, and IRQs are now balanced correctly on NUMA systems. Note that users of irqbalance packages have to update kernel to 2.6.32-358.2.1 or later in order to use the irqbalance daemon in correct manner. (BZ#1055572, BZ#1055574)

Comment 5 Don Howard 2014-04-30 22:21:39 UTC
*** Bug 1093105 has been marked as a duplicate of this bug. ***

Comment 10 errata-xmlrpc 2014-10-14 07:10:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1504.html


Note You need to log in before you can comment on or make changes to this bug.