Bug 712464 - bnx2 firmware fails to load
Summary: bnx2 firmware fails to load
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kernel
Version: 6.1
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: rc
: ---
Assignee: Neil Horman
QA Contact: Red Hat Kernel QE team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-06-10 17:39 UTC by chris.petty
Modified: 2011-06-15 18:17 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-06-15 18:07:00 UTC
Target Upstream Version:


Attachments (Terms of Use)
patch needed for compiling bnx2 source since recent kernel update (2.46 KB, application/octet-stream)
2011-06-10 17:39 UTC, chris.petty
no flags Details
udevadm monitor of unload/load bnx2 (11.17 KB, application/octet-stream)
2011-06-15 17:44 UTC, chris.petty
no flags Details
udevadm monitor of unload/load RH bnx2 (12.84 KB, application/octet-stream)
2011-06-15 17:45 UTC, chris.petty
no flags Details

Description chris.petty 2011-06-10 17:39:19 UTC
Created attachment 504164 [details]
patch needed for compiling bnx2 source since recent kernel update

Description of problem:
bnx2 firmware fails on bootup resulting in a NIC that can not function.

Version-Release number of selected component (if applicable):
kernel: 2.6.32-131.2.1.el6

How reproducible:
Every time.

Steps to Reproduce:
1. Reboot the machine into a kernel where i haven't compiled bnx2 from broadcom source
  
Actual results:
bnx2: Can't load firmware file "bnx2/bnx2-mips-06-6.2.1.fw"
bnx2 0000:05:00.0 : PCI INT A disabled
bnx2 probe of 0000:09:00.0 failed with error -2
bnx2 0000:09:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
bnx2 0000:09:00.0: firmware: requesting bnx2/bnx2-mips-06-6.2.1.fw


Additional info:
Prior to 2.6.32-131.2.1.el6 i could compile bnx2 from broadcom's src.rpm without issue.  As long as my compiled version was loaded NIC responds as expected.

With the most recent kernel update the bnx2x_compat.h has incompatible
definitions with the broadcom source.  We had to form a patch to compile the
drivers properly to get up and running.

Comment 2 Neil Horman 2011-06-13 11:52:17 UTC
I'm not sure I'm following you.  The bug description describes a problem loading firmware, but you're providing a patch that allows the driver to build - The driver builds just fine, or it wouldn't be in the release - I'm not sure what you mean by incompatibilities with the braodcom source - are you saying that you're trying to build the bnx2 driver from sources obtained from broadcom rather than the kernel.org tree?

As for the firmware failure - Can you tell me if the firmware file exists?  it should be in:
/lib/firmware/bnx2/bnx2-mips-06-6.2.1.fw


If its there can you use udevadm test to simulate a udev firmware load event and paste the output here?  Thanks!

Comment 3 chris.petty 2011-06-13 13:44:32 UTC
Sorry,

It's the actual kernel module that comes from the RHEL source that's having the issue.

The firmware itself works ( and does exist ).

[petty@node61 ~]$ ls -l /lib/firmware/bnx2/bnx2-mips-06-6.2.1.fw
-rw-r--r-- 1 root root 92792 Jun  2 11:51 /lib/firmware/bnx2/bnx2-mips-06-6.2.1.fw

However when the bnx2.ko from the source tree is used the firmware won't load.

I'm having to build the kernel modules directly from broadcom source rpms ( http://www.broadcom.com/support/ethernet_nic/netxtremeii.php ) against the kernel that i am running.

The results from this build are:
[petty@node61 ~]$ ls -l /lib/modules/2.6.32-131.2.1.el6.x86_64/updates/
-r--r--r-- 1 root root 1672816 Jun  9 15:15 bnx2i.ko
-r--r--r-- 1 root root 1418008 Jun  9 15:15 bnx2.ko
-rw-r--r-- 1 root root 6901423 Jun  9 15:15 bnx2x.ko
-r--r--r-- 1 root root 1208229 Jun  9 15:15 cnic.ko

Once this bnx2.ko is used instead of the default, the firmware loads fine.

The patch included was for building the ko's from source once the last kernel-firmware update was released.  This was needed because the released bnx2.ko was still not able to properly load the firmware, therefore i needed to rebuild from broadcom's source again.

Comment 4 Neil Horman 2011-06-13 18:32:11 UTC
Ok, thank you for the clarification.  however, I'm able to load firmware just fine with the in kernel driver that was released with RHEL6.1.  Can you run udevadm as I requested to isolate what the load issue is?

Comment 5 chris.petty 2011-06-13 19:51:50 UTC
Sorry, i'm unfamiliar with that ... is this what you're looking for:

[petty@node51 ~]$ sudo udevadm test /sys/class/net/eth0
[sudo] password for petty: 
run_command: calling: test
udevadm_test: version 147
This program is for debugging only, it does not run any program,
specified by a RUN key. It may show incorrect results, because
some values may be different, or not available at a simulation run.

parse_file: reading '/lib/udev/rules.d/10-console.rules' as rules file
parse_file: reading '/lib/udev/rules.d/10-dm.rules' as rules file
parse_file: reading '/lib/udev/rules.d/11-dm-lvm.rules' as rules file
parse_file: reading '/lib/udev/rules.d/13-dm-disk.rules' as rules file
parse_file: reading '/etc/udev/rules.d/40-hplip.rules' as rules file
parse_file: reading '/lib/udev/rules.d/40-isdn.rules' as rules file
parse_file: reading '/lib/udev/rules.d/40-libgphoto2.rules' as rules file
parse_file: reading '/lib/udev/rules.d/40-multipath.rules' as rules file
parse_file: reading '/lib/udev/rules.d/40-redhat.rules' as rules file
parse_file: reading '/lib/udev/rules.d/40-usb-media-players.rules' as rules file
parse_file: reading '/lib/udev/rules.d/50-firmware.rules' as rules file
parse_file: reading '/lib/udev/rules.d/50-udev-default.rules' as rules file
parse_file: reading '/etc/udev/rules.d/56-hpmud_support.rules' as rules file
parse_file: reading '/lib/udev/rules.d/60-cdrom_id.rules' as rules file
parse_file: reading '/etc/udev/rules.d/60-fprint-autosuspend.rules' as rules file
parse_file: reading '/lib/udev/rules.d/60-libmtp.rules' as rules file
parse_file: reading '/lib/udev/rules.d/60-net.rules' as rules file
parse_file: reading '/etc/udev/rules.d/60-openct.rules' as rules file
parse_file: reading '/etc/udev/rules.d/60-pcmcia.rules' as rules file
parse_file: reading '/lib/udev/rules.d/60-persistent-alsa.rules' as rules file
parse_file: reading '/lib/udev/rules.d/60-persistent-input.rules' as rules file
parse_file: reading '/lib/udev/rules.d/60-persistent-serial.rules' as rules file
parse_file: reading '/lib/udev/rules.d/60-persistent-storage-tape.rules' as rules file
parse_file: reading '/lib/udev/rules.d/60-persistent-storage.rules' as rules file
parse_file: reading '/lib/udev/rules.d/60-persistent-v4l.rules' as rules file
parse_file: reading '/etc/udev/rules.d/60-raw.rules' as rules file
parse_file: reading '/lib/udev/rules.d/61-gnome-bluetooth-rfkill.rules' as rules file
parse_file: reading '/lib/udev/rules.d/61-mobile-action.rules' as rules file
parse_file: reading '/lib/udev/rules.d/61-option-modem-modeswitch.rules' as rules file
parse_file: reading '/lib/udev/rules.d/61-persistent-storage-edd.rules' as rules file
parse_file: reading '/lib/udev/rules.d/64-device-mapper.rules' as rules file
parse_file: reading '/lib/udev/rules.d/64-md-raid.rules' as rules file
parse_file: reading '/lib/udev/rules.d/65-libsane.rules' as rules file
parse_file: reading '/lib/udev/rules.d/65-md-incremental.rules' as rules file
parse_file: reading '/lib/udev/rules.d/69-pilot-link.rules' as rules file
parse_file: reading '/lib/udev/rules.d/70-acl.rules' as rules file
parse_file: reading '/lib/udev/rules.d/70-anaconda.rules' as rules file
parse_file: reading '/etc/udev/rules.d/70-cups-libusb.rules' as rules file
parse_file: reading '/lib/udev/rules.d/70-hid2hci.rules' as rules file
parse_file: reading '/etc/udev/rules.d/70-persistent-cd.rules' as rules file
parse_file: reading '/etc/udev/rules.d/70-persistent-net.rules' as rules file
parse_file: reading '/lib/udev/rules.d/70-printers.rules' as rules file
parse_file: reading '/lib/udev/rules.d/75-cd-aliases-generator.rules' as rules file
parse_file: reading '/lib/udev/rules.d/75-net-description.rules' as rules file
parse_file: reading '/lib/udev/rules.d/75-persistent-net-generator.rules' as rules file
parse_file: reading '/lib/udev/rules.d/75-tty-description.rules' as rules file
parse_file: reading '/lib/udev/rules.d/77-mm-ericsson-mbm.rules' as rules file
parse_file: reading '/lib/udev/rules.d/77-mm-longcheer-port-types.rules' as rules file
parse_file: reading '/lib/udev/rules.d/77-mm-pcmcia-device-blacklist.rules' as rules file
parse_file: reading '/lib/udev/rules.d/77-mm-platform-serial-whitelist.rules' as rules file
parse_file: reading '/lib/udev/rules.d/77-mm-simtech-port-types.rules' as rules file
parse_file: reading '/lib/udev/rules.d/77-mm-usb-device-blacklist.rules' as rules file
parse_file: reading '/lib/udev/rules.d/77-mm-zte-port-types.rules' as rules file
parse_file: reading '/lib/udev/rules.d/77-nm-olpc-mesh.rules' as rules file
parse_file: reading '/lib/udev/rules.d/78-sound-card.rules' as rules file
parse_file: reading '/lib/udev/rules.d/79-fstab_import.rules' as rules file
parse_file: reading '/lib/udev/rules.d/80-drivers.rules' as rules file
parse_file: reading '/lib/udev/rules.d/80-udisks.rules' as rules file
parse_file: reading '/lib/udev/rules.d/85-regulatory.rules' as rules file
parse_file: reading '/lib/udev/rules.d/85-usbmuxd.rules' as rules file
parse_file: reading '/lib/udev/rules.d/88-clock.rules' as rules file
parse_file: reading '/etc/udev/rules.d/90-alsa.rules' as rules file
parse_file: reading '/etc/udev/rules.d/90-hal.rules' as rules file
parse_file: reading '/lib/udev/rules.d/90-pulseaudio.rules' as rules file
parse_file: reading '/etc/udev/rules.d/91-drm-modeset.rules' as rules file
parse_file: reading '/lib/udev/rules.d/95-devkit-power-battery-recall-dell.rules' as rules file
parse_file: reading '/lib/udev/rules.d/95-devkit-power-battery-recall-fujitsu.rules' as rules file
parse_file: reading '/lib/udev/rules.d/95-devkit-power-battery-recall-gateway.rules' as rules file
parse_file: reading '/lib/udev/rules.d/95-devkit-power-battery-recall-ibm.rules' as rules file
parse_file: reading '/lib/udev/rules.d/95-devkit-power-battery-recall-lenovo.rules' as rules file
parse_file: reading '/lib/udev/rules.d/95-devkit-power-battery-recall-toshiba.rules' as rules file
parse_file: reading '/lib/udev/rules.d/95-devkit-power-csr.rules' as rules file
parse_file: reading '/lib/udev/rules.d/95-devkit-power-hid.rules' as rules file
parse_file: reading '/lib/udev/rules.d/95-devkit-power-wup.rules' as rules file
parse_file: reading '/lib/udev/rules.d/95-dm-notify.rules' as rules file
parse_file: reading '/lib/udev/rules.d/95-keymap.rules' as rules file
parse_file: reading '/lib/udev/rules.d/95-udev-late.rules' as rules file
parse_file: reading '/etc/udev/rules.d/97-bluetooth-serial.rules' as rules file
parse_file: reading '/lib/udev/rules.d/97-bluetooth.rules' as rules file
parse_file: reading '/etc/udev/rules.d/98-kexec.rules' as rules file
parse_file: reading '/etc/udev/rules.d/99-fuse.rules' as rules file
udev_rules_new: rules use 183240 bytes tokens (15270 * 12 bytes), 34009 bytes buffer
udev_rules_new: temporary index used 57040 bytes (2852 * 20 bytes)
udev_device_new_from_syspath: device 0x18ee3a0 has devpath '/devices/pci0000:00/0000:00:01.0/0000:01:00.0/net/eth0'
udev_device_delete_db: remove db file for '/devices/pci0000:00/0000:00:01.0/0000:01:00.0/net/eth0' in '/dev/.udev/db/net:eth0 = No such file or directory'
udev_rules_apply_to_event: PROGRAM '/lib/udev/rename_device' /lib/udev/rules.d/60-net.rules:2
util_run_program: '/lib/udev/rename_device' started
util_run_program: '/lib/udev/rename_device' returned with exitcode 0
udev_rules_apply_to_event: RUN '/etc/sysconfig/network-scripts/net.hotplug' /lib/udev/rules.d/60-net.rules:3
udev_device_new_from_syspath: device 0x18d60a0 has devpath '/devices/pci0000:00/0000:00:01.0/0000:01:00.0'
udev_device_new_from_syspath: device 0x18edea0 has devpath '/devices/pci0000:00/0000:00:01.0'
udev_device_new_from_syspath: device 0x18ee160 has devpath '/devices/pci0000:00'
udev_rules_apply_to_event: IMPORT 'pci-db /devices/pci0000:00/0000:00:01.0/0000:01:00.0/net/eth0' /lib/udev/rules.d/75-net-description.rules:11
util_run_program: 'pci-db /devices/pci0000:00/0000:00:01.0/0000:01:00.0/net/eth0' started
util_run_program: '/lib/udev/pci-db' (stderr) 'libudev: udev_device_new_from_syspath: device 0xd5a130 has devpath '//devices/pci0000:00/0000:00:01.0/0000:01:00.0/net/eth0''
util_run_program: '/lib/udev/pci-db' (stderr) 'libudev: udev_device_new_from_syspath: device 0xd5a4a0 has devpath '//devices/pci0000:00/0000:00:01.0/0000:01:00.0/net''
util_run_program: '/lib/udev/pci-db' (stderr) 'libudev: udev_device_new_from_syspath: device 0xd5a710 has devpath '//devices/pci0000:00/0000:00:01.0/0000:01:00.0''
util_run_program: '/lib/udev/pci-db' (stdout) 'ID_VENDOR_FROM_DATABASE=Broadcom Corporation'
util_run_program: '/lib/udev/pci-db' (stdout) 'ID_MODEL_FROM_DATABASE=NetXtreme II BCM5709 Gigabit Ethernet'
util_run_program: 'pci-db /devices/pci0000:00/0000:00:01.0/0000:01:00.0/net/eth0' returned with exitcode 0
udev_rules_apply_to_event: IMPORT 'write_net_rules' /lib/udev/rules.d/75-persistent-net-generator.rules:93
util_run_program: 'write_net_rules' started
util_run_program: '/lib/udev/write_net_rules' (stdout) 'INTERFACE_NEW=eth5'
util_run_program: 'write_net_rules' returned with exitcode 0
udev_rules_apply_to_event: NAME 'eth5' /lib/udev/rules.d/75-persistent-net-generator.rules:96
udev_rules_apply_to_event: RUN 'socket:@/org/freedesktop/hal/udev_event' /etc/udev/rules.d/90-hal.rules:2
rename_netif: changing net interface name from 'eth0' to 'eth5'
rename_netif: error changing netif name 'eth0' to 'eth5': Device or resource busy
udevadm_test: UDEV_LOG=6
udevadm_test: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:01:00.0/net/eth0
udevadm_test: INTERFACE=eth0
udevadm_test: IFINDEX=2
udevadm_test: ACTION=add
udevadm_test: SUBSYSTEM=net
udevadm_test: ID_VENDOR_FROM_DATABASE=Broadcom Corporation
udevadm_test: ID_MODEL_FROM_DATABASE=NetXtreme II BCM5709 Gigabit Ethernet
udevadm_test: ID_BUS=pci
udevadm_test: ID_VENDOR_ID=0x14e4
udevadm_test: ID_MODEL_ID=0x1639
udevadm_test: NET_MATCHID=0000:01:00.0
udevadm_test: MATCHADDR=00:21:9b:95:94:02
udevadm_test: MATCHIFTYPE=1
udevadm_test: COMMENT=PCI device 0x14e4:0x1639 (bnx2)
udevadm_test: INTERFACE_NEW=eth5

Comment 6 Neil Horman 2011-06-14 13:51:33 UTC
What I was hoping for was this:
udevadm test --action="add" --subsystem="firmware" /sys/class/net/eth0

But don't bother, I just checked myself and it appears that udevadm test isn't able to generate firmware load requests for some reason.  

The thing I'm trying to get at is a trace log that shows the bnx2 driver issuing a kernel udev event requesting the load of the noted firmware, and udevs response to that request.  That should help us get closer to understanding why the firmware isn't loading (either (a) the request is never being issued or (b) it is being issued but udev isn't loading it for some reason.

An alternate approach here is (if you have direct access to the console), open up two terminals.  In one terminal, issue this command:
udevadm monitor

that will start the udevadm program listening for kernel events, which you can log to a file

in the other terminal issue these commands:
rmmod bnx2
modprobe bnx2

That should force the bnx2 card to re-init, and in the process request new firmware.  That will result in a replay of the firmware load request in the udevadm terminal.  If you could send that output to this bz, it will help us understand whats going on here.

Thanks!

Comment 7 chris.petty 2011-06-14 14:58:12 UTC
I'm not sure how helpful this is going to be, but on my local machine, which uses a different driver, when i monitor with udevadm

modprobe bnx2 creates:

KERNEL[1308062303.533476] add      /module/bnx2 (module)
KERNEL[1308062303.533496] add      /bus/pci/drivers/bnx2 (drivers)
UDEV  [1308062303.540513] add      /bus/pci/drivers/bnx2 (drivers)
UDEV  [1308062303.540527] add      /module/bnx2 (module)




All my machines which actually need bxn2 are remote and as well as nfs-root.  So if i unload the module i lose my connection and my OS.  I'm also only able to load one screen through kvm.  

I'll try to get an install on one of the machines which actually have a hard-drive if the above info isn't helpful.

Comment 8 Neil Horman 2011-06-14 16:03:46 UTC
Thank you for your effort, but no, that won't help.  The firmware addition events are only generated when the driver finds hardware that it is compatible with.  So if you don't have a bnx2 card in place on the system under test, you won't get any firmware requests from the driver.  

using kvm won't help either, as the kvm guests use virtual drivers and the kernel bridge (or routing code) to forward traffic to the physical device.  You have to modprobe the module on bare metal.

If you could get a local install and do the above, it would be a big help.  Thanks!

Comment 9 chris.petty 2011-06-15 17:43:12 UTC
Neil, thanks for the help.

Apparently its an issue with the diskless (oneSIS) nature of our OS.

On the local install you're correct, the updated kernel/kernel-firmware is working correctly.

With the rebuilt kernel module ( from broadcom source ), there isn't a request for the firmware. I'm guessing its built into the module.

With the module from the source tree, there is a request:
KERNEL[1308147008.697129] add      /devices/pci0000:00/0000:00:02.0/0000:06:00.0/0000:07:00.0/0000:08:00.0/0000:09:00.0/firmware/0000:09:00.0 (firmware)

This isn't present in my initrd, therefore its failing.

Sorry for the hassle, but thanks for the help.  I'll attach the two logs from udevadm .. one with the broadcom source, one with the RH source.

Comment 10 chris.petty 2011-06-15 17:44:13 UTC
Created attachment 504914 [details]
udevadm monitor of unload/load bnx2

Comment 11 chris.petty 2011-06-15 17:45:07 UTC
Created attachment 504915 [details]
udevadm monitor of unload/load RH bnx2

RH bnx2 from source tree

Comment 12 Neil Horman 2011-06-15 18:07:00 UTC
Its no hassle, thanks for letting me know.  It makes sense now, if you have a custom initrd and are loading bnx2 there you're going to want the corresponding firmware to be available as well.  I presume your initrd has udev running?  If so, you should just be able to pull in /lib/firmware/bnx2/* and it should all just work.  If you're doing something a bit more bare bones that doesn't use udev, you can still make the firmware load work.  Let me know if you need them and I can give you instructions.

Comment 13 chris.petty 2011-06-15 18:17:21 UTC
Looks like udev isn't running, so instructions would be useful, thanks.


Note You need to log in before you can comment on or make changes to this bug.