Description of problem: Using the bnx2x driver on a Cisco UCS Blade C200-M1 to bring up an Broadcom BCM57711 fails with: bnx2x: eth2: using MSI-X ADDRCONF(NETDEV_UP): eth2: link is not ready Also, kernel-2.6.18-169.el5.gtest.76.x86_64.rpm doesn't work with the NIC, see the dmesg below: bnx2x_set_int_mode:6821(eth2)]Multi requested but failed to enable MSI-X set number of queues to 1 bnx2x: eth2: using MSI IRQ 123 ADDRCONF(NETDEV_UP): eth2: link is not ready [bnx2x_set_int_mode:6821(eth3)]Multi requested but failed to enable MSI-X set number of queues to 1 bnx2x: eth3: using MSI IRQ 131 ADDRCONF(NETDEV_UP): eth3: link is not ready This is the console output when passing debug=0x20004: [bnx2x_set_gpio:1914(eth3)]Set GPIO 1 (shift 1) -> output low [bnx2x_set_gpio:1914(eth3)]Set GPIO 2 (shift 2) -> output low [bnx2x_link_reset:4920(eth3)]reset external PHY [bnx2x_stats_init:3123(eth3)]port_stx 0xa7304 [bnx2x_phy_init:4657(eth3)]Phy Initialization started [bnx2x_phy_init:4659(eth3)]req_speed = 0, req_flowctrl=0 [bnx2x_emac_init:244(eth3)]EMAC reset reg is 1024 [bnx2x_phy_deassert:523(eth3)]bnx2x_phy_deassert:XGXS [bnx2x_phy_init:4827(eth3)]Phy address = 0x1 [bnx2x_ext_phy_reset:1860(eth3)]Port 1: bnx2x_ext_phy_reset [bnx2x_ext_phy_reset:1976(eth3)]BAD XGXS ext_phy_config 0xff000910 [bnx2x_ext_phy_init:3002(eth3)]control reg 0xffff (after 1000 ms) [bnx2x_ext_phy_init:3538(eth3)]BAD XGXS ext_phy_config 0xff000910 [bnx2x_link_int_enable:4036(eth3)]enabled XGXS interrupt [bnx2x_link_int_enable:4043(eth3)]enabled external phy int [bnx2x_link_int_enable:4063(eth3)]port 1, is_xgxs=1, int_status 0x0 [bnx2x_link_int_enable:4068(eth3)] int_mask 0x3c8001, MI_INT 0, SERDES_LINK 0 [bnx2x_link_int_enable:4071(eth3)] 10G 0, XGXS_LINK 0 ADDRCONF(NETDEV_UP): eth3: link is not ready [bnx2x_stats_handle:4167(eth3)]state 0 -> event 3 -> state 0 [bnx2x_set_led:4503(eth3)]bnx2x_set_led: port 1, mode 0 [bnx2x_set_led:4505(eth3)]speed 0x0, hw_led_mode 0x1 [bnx2x_set_gpio:1914(eth3)]Set GPIO 2 (shift 2) -> output low [bnx2x_link_reset:4920(eth3)]reset external PHY ACPI: PCI interrupt for device 0000:03:00.1 disabled [bnx2x_stats_handle:4167(eth2)]state 0 -> event 3 -> state 0 [bnx2x_set_led:4503(eth2)]bnx2x_set_led: port 0, mode 0 [bnx2x_set_led:4505(eth2)]speed 0x0, hw_led_mode 0x1 [bnx2x_set_gpio:1914(eth2)]Set GPIO 1 (shift 5) -> output low [bnx2x_set_gpio:1914(eth2)]Set GPIO 2 (shift 6) -> output low [bnx2x_link_reset:4920(eth2)]reset external PHY ACPI: PCI interrupt for device 0000:03:00.0 disabled Broadcom NetXtreme II 5771x 10Gigabit Ethernet Driver bnx2x 1.48.105 (2009/03/02) PCI: Enabling device 0000:03:00.0 (0000 -> 0002) ACPI: PCI Interrupt 0000:03:00.0[A] -> GSI 24 (level, low) -> IRQ 185 PCI: Setting latency timer of device 0000:03:00.0 to 64 bnx2x: part number 394D4342-31373735-31314131-473331 [bnx2x_link_settings_supported:7803(eth%d)]NVRAM config error. BAD XGXS ext_phy_config 0xff000911 [bnx2x_link_settings_requested:7989(eth%d)]NVRAM config error. Invalid link_config 0x1070000 speed_cap_mask 0x500050 eth2: Broadcom NetXtreme II BCM57711 XGb (A0) PCI-E x8 5GHz (Gen2) found at mem b2000000, IRQ 185, node addr 001018570924 PCI: Enabling device 0000:03:00.1 (0000 -> 0002) ACPI: PCI Interrupt 0000:03:00.1[B] -> GSI 34 (level, low) -> IRQ 107 PCI: Setting latency timer of device 0000:03:00.1 to 64 bnx2x: part number 394D4342-31373735-31314131-473331 [bnx2x_link_settings_supported:7803(eth%d)]NVRAM config error. BAD XGXS ext_phy_config 0xff000910 [bnx2x_link_settings_requested:7989(eth%d)]NVRAM config error. Invalid link_config 0x1070000 speed_cap_mask 0x500050 [bnx2x_set_int_mode:6821(eth2)]Multi requested but failed to enable MSI-X set number of queues to 1 bnx2x: eth2: using MSI IRQ 139 ADDRCONF(NETDEV_UP): eth2: link is not ready eth3: Broadcom NetXtreme II BCM57711 XGb (A0) PCI-E x8 5GHz (Gen2) found at mem b3000000, IRQ 107, node addr 001018570926 [bnx2x_set_int_mode:6821(eth3)]Multi requested but failed to enable MSI-X set number of queues to 1 bnx2x: eth3: using MSI IRQ 147 ADDRCONF(NETDEV_UP): eth3: link is not ready
Event posted on 10-29-2009 03:52pm BRT by fbl These errors seems to be the problem: [bnx2x_link_settings_supported:7803(eth%d)]NVRAM config error. BAD XGXS ext_phy_config 0xff000911 [bnx2x_link_settings_requested:7989(eth%d)]NVRAM config error. Invalid link_config 0x1070000 speed_cap_mask 0x500050 The first one comes from: 7628 static void __devinit bnx2x_link_settings_supported(struct bnx2x *bp, 7629 u32 switch_cfg) 7630 { <snipped> 7634 switch (switch_cfg) { <snipped> 7686 case SWITCH_CFG_10G: 7687 BNX2X_DEV_INFO("switch_cfg 0x%x (10G)\n", switch_cfg); 7688 7689 ext_phy_type = 7690 XGXS_EXT_PHY_TYPE(bp->link_params.ext_phy_config); 7691 switch (ext_phy_type) { <snipped> 7800 default: 7801 BNX2X_ERR("NVRAM config error. " 7802 "BAD XGXS ext_phy_config 0x%x\n", 7803 bp->link_params.ext_phy_config); 7804 return; 7805 } so, switch_cfg is 0x01000000 and ext_phy_type is unknown (0xff000911) so the default is used failing the device. The ext_phy_type = bp->link_params.ext_phy_config | 0x0000ff00 which is 0x900. but that isn't defined yet on gospo's kernel: 227 #define PORT_HW_CFG_XGXS_EXT_PHY_TYPE_DIRECT 0x00000000 228 #define PORT_HW_CFG_XGXS_EXT_PHY_TYPE_BCM8071 0x00000100 229 #define PORT_HW_CFG_XGXS_EXT_PHY_TYPE_BCM8072 0x00000200 230 #define PORT_HW_CFG_XGXS_EXT_PHY_TYPE_BCM8073 0x00000300 231 #define PORT_HW_CFG_XGXS_EXT_PHY_TYPE_BCM8705 0x00000400 232 #define PORT_HW_CFG_XGXS_EXT_PHY_TYPE_BCM8706 0x00000500 233 #define PORT_HW_CFG_XGXS_EXT_PHY_TYPE_BCM8726 0x00000600 234 #define PORT_HW_CFG_XGXS_EXT_PHY_TYPE_BCM8481 0x00000700 235 #define PORT_HW_CFG_XGXS_EXT_PHY_TYPE_SFX7101 0x00000800 236 #define PORT_HW_CFG_XGXS_EXT_PHY_TYPE_FAILURE 0x0000fd00 237 #define PORT_HW_CFG_XGXS_EXT_PHY_TYPE_NOT_CONN 0x0000ff00 but upstream seems to have that fixed: #define PORT_HW_CFG_XGXS_EXT_PHY_TYPE_SFX7101 0x00000800 #define PORT_HW_CFG_XGXS_EXT_PHY_TYPE_BCM8727 0x00000900 <---- #define PORT_HW_CFG_XGXS_EXT_PHY_TYPE_BCM8727_NOC 0x00000a00 I'll check what patch introduced it. Flavio This event sent from IssueTracker by fbl issue 356912
Event posted on 10-29-2009 03:53pm BRT by fbl This upstream commit might fix this problem: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=4d295db0efd2ccf06edb7a45ad885b40c56b7161 This event sent from IssueTracker by fbl issue 356912
Event posted on 10-29-2009 04:49pm BRT by fbl I've backported that upstream patch to gospo's kernel. Brew build URL: https://brewweb.devel.redhat.com/taskinfo?taskID=2053999 Can you give a try and report back your results? I'd need the console output passing 'debug=0x20006' to the driver. thanks, Flavio This event sent from IssueTracker by fbl issue 356912 it_file 269225
Below is the feedback regarding to the last test kernel at comment#4: ---8<---- That works! [bnx2x_set_int_mode:6838(eth2)]Multi requested but failed to enable MSI-X set number of queues to 1 bnx2x: eth2: using MSI IRQ 123 ADDRCONF(NETDEV_UP): eth2: link is not ready bnx2x: eth2 NIC Link is Down bnx2x: eth2 NIC Link is Down bnx2x: eth2 NIC Link is Up, 10000 Mbps full duplex, receive & transmit flow control ON ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready [bnx2x_set_int_mode:6838(eth3)]Multi requested but failed to enable MSI-X set number of queues to 1 bnx2x: eth3: using MSI IRQ 131 ADDRCONF(NETDEV_UP): eth3: link is not ready bnx2x: eth3 NIC Link is Down bnx2x: eth3 NIC Link is Down bnx2x: eth3 NIC Link is Up, 10000 Mbps full duplex, receive & transmit flow control ON ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready ---8<---- Flavio
Flavio, thanks for solving this. Upstream commit 4d295db0efd2ccf06edb7a45ad885b40c56b7161 will be included in RHEL5.5 driver update. Can I close this bug as duplicate of 515716 ?
Stanislaw, I think so. Do you know if there is a similar ticket for RHEL4? I guess it will be requested for RHEL4 too. thanks!
We have no RHEL4 request for adding support for BCM8727 SFP+ PHY nor request to update driver to never version (that would kill me :). If customers will need this PHY support please open new bug report for RHEL4.
*** This bug has been marked as a duplicate of bug 515716 ***
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release.
@Cisco We need to confirm that there is third-party commitment to test for the resolution of this request during the RHEL 5.5 Beta Test Phase AND 5.3.z builds before we can approve it for acceptance into the release. RHEL 5.5 Beta Test Phase is expected to begin around February 2010. RHEL 5.4.z Test Phase begins immediately once a test build is made available. In order to avoid any unnecessary delays, please post a confirmation as soon as possible, including the contact information for testing engineers. Any additional information about alternative testing variations we could use to reproduce this issue in-house would be appreciated.
Here are test packages with bnx2x update (version 1.52.1-5) for RHEL5.5. http://people.redhat.com/sgruszka/rhel5.5-broadcom/ Cisco, please test update works on your hardware.
Cisco IT reports back that they've successfully tested the bnx2x update provided in comment #24
Has this patch mentioned above been backported to RHEL5.4?
Have you tried kernel 2.6.18-164.8.1?