Hide Forgot
Created attachment 504164 [details] patch needed for compiling bnx2 source since recent kernel update Description of problem: bnx2 firmware fails on bootup resulting in a NIC that can not function. Version-Release number of selected component (if applicable): kernel: 2.6.32-131.2.1.el6 How reproducible: Every time. Steps to Reproduce: 1. Reboot the machine into a kernel where i haven't compiled bnx2 from broadcom source Actual results: bnx2: Can't load firmware file "bnx2/bnx2-mips-06-6.2.1.fw" bnx2 0000:05:00.0 : PCI INT A disabled bnx2 probe of 0000:09:00.0 failed with error -2 bnx2 0000:09:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 bnx2 0000:09:00.0: firmware: requesting bnx2/bnx2-mips-06-6.2.1.fw Additional info: Prior to 2.6.32-131.2.1.el6 i could compile bnx2 from broadcom's src.rpm without issue. As long as my compiled version was loaded NIC responds as expected. With the most recent kernel update the bnx2x_compat.h has incompatible definitions with the broadcom source. We had to form a patch to compile the drivers properly to get up and running.
I'm not sure I'm following you. The bug description describes a problem loading firmware, but you're providing a patch that allows the driver to build - The driver builds just fine, or it wouldn't be in the release - I'm not sure what you mean by incompatibilities with the braodcom source - are you saying that you're trying to build the bnx2 driver from sources obtained from broadcom rather than the kernel.org tree? As for the firmware failure - Can you tell me if the firmware file exists? it should be in: /lib/firmware/bnx2/bnx2-mips-06-6.2.1.fw If its there can you use udevadm test to simulate a udev firmware load event and paste the output here? Thanks!
Sorry, It's the actual kernel module that comes from the RHEL source that's having the issue. The firmware itself works ( and does exist ). [petty@node61 ~]$ ls -l /lib/firmware/bnx2/bnx2-mips-06-6.2.1.fw -rw-r--r-- 1 root root 92792 Jun 2 11:51 /lib/firmware/bnx2/bnx2-mips-06-6.2.1.fw However when the bnx2.ko from the source tree is used the firmware won't load. I'm having to build the kernel modules directly from broadcom source rpms ( http://www.broadcom.com/support/ethernet_nic/netxtremeii.php ) against the kernel that i am running. The results from this build are: [petty@node61 ~]$ ls -l /lib/modules/2.6.32-131.2.1.el6.x86_64/updates/ -r--r--r-- 1 root root 1672816 Jun 9 15:15 bnx2i.ko -r--r--r-- 1 root root 1418008 Jun 9 15:15 bnx2.ko -rw-r--r-- 1 root root 6901423 Jun 9 15:15 bnx2x.ko -r--r--r-- 1 root root 1208229 Jun 9 15:15 cnic.ko Once this bnx2.ko is used instead of the default, the firmware loads fine. The patch included was for building the ko's from source once the last kernel-firmware update was released. This was needed because the released bnx2.ko was still not able to properly load the firmware, therefore i needed to rebuild from broadcom's source again.
Ok, thank you for the clarification. however, I'm able to load firmware just fine with the in kernel driver that was released with RHEL6.1. Can you run udevadm as I requested to isolate what the load issue is?
Sorry, i'm unfamiliar with that ... is this what you're looking for: [petty@node51 ~]$ sudo udevadm test /sys/class/net/eth0 [sudo] password for petty: run_command: calling: test udevadm_test: version 147 This program is for debugging only, it does not run any program, specified by a RUN key. It may show incorrect results, because some values may be different, or not available at a simulation run. parse_file: reading '/lib/udev/rules.d/10-console.rules' as rules file parse_file: reading '/lib/udev/rules.d/10-dm.rules' as rules file parse_file: reading '/lib/udev/rules.d/11-dm-lvm.rules' as rules file parse_file: reading '/lib/udev/rules.d/13-dm-disk.rules' as rules file parse_file: reading '/etc/udev/rules.d/40-hplip.rules' as rules file parse_file: reading '/lib/udev/rules.d/40-isdn.rules' as rules file parse_file: reading '/lib/udev/rules.d/40-libgphoto2.rules' as rules file parse_file: reading '/lib/udev/rules.d/40-multipath.rules' as rules file parse_file: reading '/lib/udev/rules.d/40-redhat.rules' as rules file parse_file: reading '/lib/udev/rules.d/40-usb-media-players.rules' as rules file parse_file: reading '/lib/udev/rules.d/50-firmware.rules' as rules file parse_file: reading '/lib/udev/rules.d/50-udev-default.rules' as rules file parse_file: reading '/etc/udev/rules.d/56-hpmud_support.rules' as rules file parse_file: reading '/lib/udev/rules.d/60-cdrom_id.rules' as rules file parse_file: reading '/etc/udev/rules.d/60-fprint-autosuspend.rules' as rules file parse_file: reading '/lib/udev/rules.d/60-libmtp.rules' as rules file parse_file: reading '/lib/udev/rules.d/60-net.rules' as rules file parse_file: reading '/etc/udev/rules.d/60-openct.rules' as rules file parse_file: reading '/etc/udev/rules.d/60-pcmcia.rules' as rules file parse_file: reading '/lib/udev/rules.d/60-persistent-alsa.rules' as rules file parse_file: reading '/lib/udev/rules.d/60-persistent-input.rules' as rules file parse_file: reading '/lib/udev/rules.d/60-persistent-serial.rules' as rules file parse_file: reading '/lib/udev/rules.d/60-persistent-storage-tape.rules' as rules file parse_file: reading '/lib/udev/rules.d/60-persistent-storage.rules' as rules file parse_file: reading '/lib/udev/rules.d/60-persistent-v4l.rules' as rules file parse_file: reading '/etc/udev/rules.d/60-raw.rules' as rules file parse_file: reading '/lib/udev/rules.d/61-gnome-bluetooth-rfkill.rules' as rules file parse_file: reading '/lib/udev/rules.d/61-mobile-action.rules' as rules file parse_file: reading '/lib/udev/rules.d/61-option-modem-modeswitch.rules' as rules file parse_file: reading '/lib/udev/rules.d/61-persistent-storage-edd.rules' as rules file parse_file: reading '/lib/udev/rules.d/64-device-mapper.rules' as rules file parse_file: reading '/lib/udev/rules.d/64-md-raid.rules' as rules file parse_file: reading '/lib/udev/rules.d/65-libsane.rules' as rules file parse_file: reading '/lib/udev/rules.d/65-md-incremental.rules' as rules file parse_file: reading '/lib/udev/rules.d/69-pilot-link.rules' as rules file parse_file: reading '/lib/udev/rules.d/70-acl.rules' as rules file parse_file: reading '/lib/udev/rules.d/70-anaconda.rules' as rules file parse_file: reading '/etc/udev/rules.d/70-cups-libusb.rules' as rules file parse_file: reading '/lib/udev/rules.d/70-hid2hci.rules' as rules file parse_file: reading '/etc/udev/rules.d/70-persistent-cd.rules' as rules file parse_file: reading '/etc/udev/rules.d/70-persistent-net.rules' as rules file parse_file: reading '/lib/udev/rules.d/70-printers.rules' as rules file parse_file: reading '/lib/udev/rules.d/75-cd-aliases-generator.rules' as rules file parse_file: reading '/lib/udev/rules.d/75-net-description.rules' as rules file parse_file: reading '/lib/udev/rules.d/75-persistent-net-generator.rules' as rules file parse_file: reading '/lib/udev/rules.d/75-tty-description.rules' as rules file parse_file: reading '/lib/udev/rules.d/77-mm-ericsson-mbm.rules' as rules file parse_file: reading '/lib/udev/rules.d/77-mm-longcheer-port-types.rules' as rules file parse_file: reading '/lib/udev/rules.d/77-mm-pcmcia-device-blacklist.rules' as rules file parse_file: reading '/lib/udev/rules.d/77-mm-platform-serial-whitelist.rules' as rules file parse_file: reading '/lib/udev/rules.d/77-mm-simtech-port-types.rules' as rules file parse_file: reading '/lib/udev/rules.d/77-mm-usb-device-blacklist.rules' as rules file parse_file: reading '/lib/udev/rules.d/77-mm-zte-port-types.rules' as rules file parse_file: reading '/lib/udev/rules.d/77-nm-olpc-mesh.rules' as rules file parse_file: reading '/lib/udev/rules.d/78-sound-card.rules' as rules file parse_file: reading '/lib/udev/rules.d/79-fstab_import.rules' as rules file parse_file: reading '/lib/udev/rules.d/80-drivers.rules' as rules file parse_file: reading '/lib/udev/rules.d/80-udisks.rules' as rules file parse_file: reading '/lib/udev/rules.d/85-regulatory.rules' as rules file parse_file: reading '/lib/udev/rules.d/85-usbmuxd.rules' as rules file parse_file: reading '/lib/udev/rules.d/88-clock.rules' as rules file parse_file: reading '/etc/udev/rules.d/90-alsa.rules' as rules file parse_file: reading '/etc/udev/rules.d/90-hal.rules' as rules file parse_file: reading '/lib/udev/rules.d/90-pulseaudio.rules' as rules file parse_file: reading '/etc/udev/rules.d/91-drm-modeset.rules' as rules file parse_file: reading '/lib/udev/rules.d/95-devkit-power-battery-recall-dell.rules' as rules file parse_file: reading '/lib/udev/rules.d/95-devkit-power-battery-recall-fujitsu.rules' as rules file parse_file: reading '/lib/udev/rules.d/95-devkit-power-battery-recall-gateway.rules' as rules file parse_file: reading '/lib/udev/rules.d/95-devkit-power-battery-recall-ibm.rules' as rules file parse_file: reading '/lib/udev/rules.d/95-devkit-power-battery-recall-lenovo.rules' as rules file parse_file: reading '/lib/udev/rules.d/95-devkit-power-battery-recall-toshiba.rules' as rules file parse_file: reading '/lib/udev/rules.d/95-devkit-power-csr.rules' as rules file parse_file: reading '/lib/udev/rules.d/95-devkit-power-hid.rules' as rules file parse_file: reading '/lib/udev/rules.d/95-devkit-power-wup.rules' as rules file parse_file: reading '/lib/udev/rules.d/95-dm-notify.rules' as rules file parse_file: reading '/lib/udev/rules.d/95-keymap.rules' as rules file parse_file: reading '/lib/udev/rules.d/95-udev-late.rules' as rules file parse_file: reading '/etc/udev/rules.d/97-bluetooth-serial.rules' as rules file parse_file: reading '/lib/udev/rules.d/97-bluetooth.rules' as rules file parse_file: reading '/etc/udev/rules.d/98-kexec.rules' as rules file parse_file: reading '/etc/udev/rules.d/99-fuse.rules' as rules file udev_rules_new: rules use 183240 bytes tokens (15270 * 12 bytes), 34009 bytes buffer udev_rules_new: temporary index used 57040 bytes (2852 * 20 bytes) udev_device_new_from_syspath: device 0x18ee3a0 has devpath '/devices/pci0000:00/0000:00:01.0/0000:01:00.0/net/eth0' udev_device_delete_db: remove db file for '/devices/pci0000:00/0000:00:01.0/0000:01:00.0/net/eth0' in '/dev/.udev/db/net:eth0 = No such file or directory' udev_rules_apply_to_event: PROGRAM '/lib/udev/rename_device' /lib/udev/rules.d/60-net.rules:2 util_run_program: '/lib/udev/rename_device' started util_run_program: '/lib/udev/rename_device' returned with exitcode 0 udev_rules_apply_to_event: RUN '/etc/sysconfig/network-scripts/net.hotplug' /lib/udev/rules.d/60-net.rules:3 udev_device_new_from_syspath: device 0x18d60a0 has devpath '/devices/pci0000:00/0000:00:01.0/0000:01:00.0' udev_device_new_from_syspath: device 0x18edea0 has devpath '/devices/pci0000:00/0000:00:01.0' udev_device_new_from_syspath: device 0x18ee160 has devpath '/devices/pci0000:00' udev_rules_apply_to_event: IMPORT 'pci-db /devices/pci0000:00/0000:00:01.0/0000:01:00.0/net/eth0' /lib/udev/rules.d/75-net-description.rules:11 util_run_program: 'pci-db /devices/pci0000:00/0000:00:01.0/0000:01:00.0/net/eth0' started util_run_program: '/lib/udev/pci-db' (stderr) 'libudev: udev_device_new_from_syspath: device 0xd5a130 has devpath '//devices/pci0000:00/0000:00:01.0/0000:01:00.0/net/eth0'' util_run_program: '/lib/udev/pci-db' (stderr) 'libudev: udev_device_new_from_syspath: device 0xd5a4a0 has devpath '//devices/pci0000:00/0000:00:01.0/0000:01:00.0/net'' util_run_program: '/lib/udev/pci-db' (stderr) 'libudev: udev_device_new_from_syspath: device 0xd5a710 has devpath '//devices/pci0000:00/0000:00:01.0/0000:01:00.0'' util_run_program: '/lib/udev/pci-db' (stdout) 'ID_VENDOR_FROM_DATABASE=Broadcom Corporation' util_run_program: '/lib/udev/pci-db' (stdout) 'ID_MODEL_FROM_DATABASE=NetXtreme II BCM5709 Gigabit Ethernet' util_run_program: 'pci-db /devices/pci0000:00/0000:00:01.0/0000:01:00.0/net/eth0' returned with exitcode 0 udev_rules_apply_to_event: IMPORT 'write_net_rules' /lib/udev/rules.d/75-persistent-net-generator.rules:93 util_run_program: 'write_net_rules' started util_run_program: '/lib/udev/write_net_rules' (stdout) 'INTERFACE_NEW=eth5' util_run_program: 'write_net_rules' returned with exitcode 0 udev_rules_apply_to_event: NAME 'eth5' /lib/udev/rules.d/75-persistent-net-generator.rules:96 udev_rules_apply_to_event: RUN 'socket:@/org/freedesktop/hal/udev_event' /etc/udev/rules.d/90-hal.rules:2 rename_netif: changing net interface name from 'eth0' to 'eth5' rename_netif: error changing netif name 'eth0' to 'eth5': Device or resource busy udevadm_test: UDEV_LOG=6 udevadm_test: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:01:00.0/net/eth0 udevadm_test: INTERFACE=eth0 udevadm_test: IFINDEX=2 udevadm_test: ACTION=add udevadm_test: SUBSYSTEM=net udevadm_test: ID_VENDOR_FROM_DATABASE=Broadcom Corporation udevadm_test: ID_MODEL_FROM_DATABASE=NetXtreme II BCM5709 Gigabit Ethernet udevadm_test: ID_BUS=pci udevadm_test: ID_VENDOR_ID=0x14e4 udevadm_test: ID_MODEL_ID=0x1639 udevadm_test: NET_MATCHID=0000:01:00.0 udevadm_test: MATCHADDR=00:21:9b:95:94:02 udevadm_test: MATCHIFTYPE=1 udevadm_test: COMMENT=PCI device 0x14e4:0x1639 (bnx2) udevadm_test: INTERFACE_NEW=eth5
What I was hoping for was this: udevadm test --action="add" --subsystem="firmware" /sys/class/net/eth0 But don't bother, I just checked myself and it appears that udevadm test isn't able to generate firmware load requests for some reason. The thing I'm trying to get at is a trace log that shows the bnx2 driver issuing a kernel udev event requesting the load of the noted firmware, and udevs response to that request. That should help us get closer to understanding why the firmware isn't loading (either (a) the request is never being issued or (b) it is being issued but udev isn't loading it for some reason. An alternate approach here is (if you have direct access to the console), open up two terminals. In one terminal, issue this command: udevadm monitor that will start the udevadm program listening for kernel events, which you can log to a file in the other terminal issue these commands: rmmod bnx2 modprobe bnx2 That should force the bnx2 card to re-init, and in the process request new firmware. That will result in a replay of the firmware load request in the udevadm terminal. If you could send that output to this bz, it will help us understand whats going on here. Thanks!
I'm not sure how helpful this is going to be, but on my local machine, which uses a different driver, when i monitor with udevadm modprobe bnx2 creates: KERNEL[1308062303.533476] add /module/bnx2 (module) KERNEL[1308062303.533496] add /bus/pci/drivers/bnx2 (drivers) UDEV [1308062303.540513] add /bus/pci/drivers/bnx2 (drivers) UDEV [1308062303.540527] add /module/bnx2 (module) All my machines which actually need bxn2 are remote and as well as nfs-root. So if i unload the module i lose my connection and my OS. I'm also only able to load one screen through kvm. I'll try to get an install on one of the machines which actually have a hard-drive if the above info isn't helpful.
Thank you for your effort, but no, that won't help. The firmware addition events are only generated when the driver finds hardware that it is compatible with. So if you don't have a bnx2 card in place on the system under test, you won't get any firmware requests from the driver. using kvm won't help either, as the kvm guests use virtual drivers and the kernel bridge (or routing code) to forward traffic to the physical device. You have to modprobe the module on bare metal. If you could get a local install and do the above, it would be a big help. Thanks!
Neil, thanks for the help. Apparently its an issue with the diskless (oneSIS) nature of our OS. On the local install you're correct, the updated kernel/kernel-firmware is working correctly. With the rebuilt kernel module ( from broadcom source ), there isn't a request for the firmware. I'm guessing its built into the module. With the module from the source tree, there is a request: KERNEL[1308147008.697129] add /devices/pci0000:00/0000:00:02.0/0000:06:00.0/0000:07:00.0/0000:08:00.0/0000:09:00.0/firmware/0000:09:00.0 (firmware) This isn't present in my initrd, therefore its failing. Sorry for the hassle, but thanks for the help. I'll attach the two logs from udevadm .. one with the broadcom source, one with the RH source.
Created attachment 504914 [details] udevadm monitor of unload/load bnx2
Created attachment 504915 [details] udevadm monitor of unload/load RH bnx2 RH bnx2 from source tree
Its no hassle, thanks for letting me know. It makes sense now, if you have a custom initrd and are loading bnx2 there you're going to want the corresponding firmware to be available as well. I presume your initrd has udev running? If so, you should just be able to pull in /lib/firmware/bnx2/* and it should all just work. If you're doing something a bit more bare bones that doesn't use udev, you can still make the firmware load work. Let me know if you need them and I can give you instructions.
Looks like udev isn't running, so instructions would be useful, thanks.