Bug 531882 - Broadcom Everest Dual port 10Gb with SFP+ (57711) NIC fails with no link
Summary: Broadcom Everest Dual port 10Gb with SFP+ (57711) NIC fails with no link
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kernel
Version: 5.4
Hardware: All
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Stanislaw Gruszka
QA Contact: Red Hat Kernel QE team
URL:
Whiteboard:
Depends On:
Blocks: 540381
TreeView+ depends on / blocked
 
Reported: 2009-10-29 18:16 UTC by Flavio Leitner
Modified: 2018-10-20 04:04 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2010-02-15 21:48:17 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Flavio Leitner 2009-10-29 18:16:17 UTC
Description of problem:
Using the bnx2x driver on a Cisco UCS Blade C200-M1 to bring up an 
Broadcom BCM57711 fails with:

  bnx2x: eth2: using MSI-X
  ADDRCONF(NETDEV_UP): eth2: link is not ready


Also, kernel-2.6.18-169.el5.gtest.76.x86_64.rpm doesn't work with the NIC,
see the dmesg below:
bnx2x_set_int_mode:6821(eth2)]Multi requested but failed to enable MSI-X  set number of queues to 1
bnx2x: eth2: using MSI  IRQ 123
ADDRCONF(NETDEV_UP): eth2: link is not ready
[bnx2x_set_int_mode:6821(eth3)]Multi requested but failed to enable MSI-X  set number of queues to 1
bnx2x: eth3: using MSI  IRQ 131
ADDRCONF(NETDEV_UP): eth3: link is not ready


This is the console output when passing debug=0x20004:
[bnx2x_set_gpio:1914(eth3)]Set GPIO 1 (shift 1) -> output low
[bnx2x_set_gpio:1914(eth3)]Set GPIO 2 (shift 2) -> output low
[bnx2x_link_reset:4920(eth3)]reset external PHY
[bnx2x_stats_init:3123(eth3)]port_stx 0xa7304
[bnx2x_phy_init:4657(eth3)]Phy Initialization started
[bnx2x_phy_init:4659(eth3)]req_speed = 0, req_flowctrl=0
[bnx2x_emac_init:244(eth3)]EMAC reset reg is 1024
[bnx2x_phy_deassert:523(eth3)]bnx2x_phy_deassert:XGXS
[bnx2x_phy_init:4827(eth3)]Phy address = 0x1
[bnx2x_ext_phy_reset:1860(eth3)]Port 1: bnx2x_ext_phy_reset
[bnx2x_ext_phy_reset:1976(eth3)]BAD XGXS ext_phy_config 0xff000910
[bnx2x_ext_phy_init:3002(eth3)]control reg 0xffff (after 1000 ms)
[bnx2x_ext_phy_init:3538(eth3)]BAD XGXS ext_phy_config 0xff000910
[bnx2x_link_int_enable:4036(eth3)]enabled XGXS interrupt
[bnx2x_link_int_enable:4043(eth3)]enabled external phy int
[bnx2x_link_int_enable:4063(eth3)]port 1, is_xgxs=1, int_status 0x0
[bnx2x_link_int_enable:4068(eth3)] int_mask 0x3c8001, MI_INT 0, SERDES_LINK 0
[bnx2x_link_int_enable:4071(eth3)] 10G 0, XGXS_LINK 0
ADDRCONF(NETDEV_UP): eth3: link is not ready
[bnx2x_stats_handle:4167(eth3)]state 0 -> event 3 -> state 0
[bnx2x_set_led:4503(eth3)]bnx2x_set_led: port 1, mode 0
[bnx2x_set_led:4505(eth3)]speed 0x0, hw_led_mode 0x1
[bnx2x_set_gpio:1914(eth3)]Set GPIO 2 (shift 2) -> output low
[bnx2x_link_reset:4920(eth3)]reset external PHY
ACPI: PCI interrupt for device 0000:03:00.1 disabled
[bnx2x_stats_handle:4167(eth2)]state 0 -> event 3 -> state 0
[bnx2x_set_led:4503(eth2)]bnx2x_set_led: port 0, mode 0
[bnx2x_set_led:4505(eth2)]speed 0x0, hw_led_mode 0x1
[bnx2x_set_gpio:1914(eth2)]Set GPIO 1 (shift 5) -> output low
[bnx2x_set_gpio:1914(eth2)]Set GPIO 2 (shift 6) -> output low
[bnx2x_link_reset:4920(eth2)]reset external PHY
ACPI: PCI interrupt for device 0000:03:00.0 disabled
Broadcom NetXtreme II 5771x 10Gigabit Ethernet Driver bnx2x 1.48.105 (2009/03/02)
PCI: Enabling device 0000:03:00.0 (0000 -> 0002)
ACPI: PCI Interrupt 0000:03:00.0[A] -> GSI 24 (level, low) -> IRQ 185
PCI: Setting latency timer of device 0000:03:00.0 to 64
bnx2x: part number 394D4342-31373735-31314131-473331
[bnx2x_link_settings_supported:7803(eth%d)]NVRAM config error. BAD XGXS ext_phy_config 0xff000911
[bnx2x_link_settings_requested:7989(eth%d)]NVRAM config error. Invalid link_config 0x1070000  speed_cap_mask 0x500050
eth2: Broadcom NetXtreme II BCM57711 XGb (A0) PCI-E x8 5GHz (Gen2) found at mem b2000000, IRQ 185, node addr 001018570924
PCI: Enabling device 0000:03:00.1 (0000 -> 0002)
ACPI: PCI Interrupt 0000:03:00.1[B] -> GSI 34 (level, low) -> IRQ 107
PCI: Setting latency timer of device 0000:03:00.1 to 64
bnx2x: part number 394D4342-31373735-31314131-473331
[bnx2x_link_settings_supported:7803(eth%d)]NVRAM config error. BAD XGXS ext_phy_config 0xff000910
[bnx2x_link_settings_requested:7989(eth%d)]NVRAM config error. Invalid link_config 0x1070000  speed_cap_mask 0x500050
[bnx2x_set_int_mode:6821(eth2)]Multi requested but failed to enable MSI-X  set number of queues to 1
bnx2x: eth2: using MSI  IRQ 139
ADDRCONF(NETDEV_UP): eth2: link is not ready
eth3: Broadcom NetXtreme II BCM57711 XGb (A0) PCI-E x8 5GHz (Gen2) found at mem b3000000, IRQ 107, node addr 001018570926
[bnx2x_set_int_mode:6821(eth3)]Multi requested but failed to enable MSI-X  set number of queues to 1
bnx2x: eth3: using MSI  IRQ 147
ADDRCONF(NETDEV_UP): eth3: link is not ready

Comment 2 Issue Tracker 2009-10-29 18:52:28 UTC
Event posted on 10-29-2009 03:52pm BRT by fbl


These errors seems to be the problem:
[bnx2x_link_settings_supported:7803(eth%d)]NVRAM config error. BAD XGXS
ext_phy_config 0xff000911
[bnx2x_link_settings_requested:7989(eth%d)]NVRAM config error. Invalid
link_config 0x1070000  speed_cap_mask 0x500050


The first one comes from:
 7628 static void __devinit bnx2x_link_settings_supported(struct bnx2x
*bp, 
 7629                                                     u32 switch_cfg)
 7630 {
<snipped>
 7634         switch (switch_cfg) {
<snipped>

 7686         case SWITCH_CFG_10G:
 7687                 BNX2X_DEV_INFO("switch_cfg 0x%x (10G)\n",
switch_cfg);
 7688 
 7689                 ext_phy_type =
 7690                        
XGXS_EXT_PHY_TYPE(bp->link_params.ext_phy_config);
 7691                 switch (ext_phy_type) {
<snipped>

 7800                 default:
 7801                         BNX2X_ERR("NVRAM config error. "
 7802                                   "BAD XGXS ext_phy_config
0x%x\n",
 7803                                   bp->link_params.ext_phy_config);
 7804                         return;
 7805                 }

so, switch_cfg is 0x01000000 and ext_phy_type is unknown (0xff000911)
so the default is used failing the device.

The ext_phy_type = bp->link_params.ext_phy_config | 0x0000ff00
which is 0x900.

but that isn't defined yet on gospo's kernel:
 227 #define PORT_HW_CFG_XGXS_EXT_PHY_TYPE_DIRECT        0x00000000
 228 #define PORT_HW_CFG_XGXS_EXT_PHY_TYPE_BCM8071       0x00000100
 229 #define PORT_HW_CFG_XGXS_EXT_PHY_TYPE_BCM8072       0x00000200
 230 #define PORT_HW_CFG_XGXS_EXT_PHY_TYPE_BCM8073       0x00000300
 231 #define PORT_HW_CFG_XGXS_EXT_PHY_TYPE_BCM8705       0x00000400
 232 #define PORT_HW_CFG_XGXS_EXT_PHY_TYPE_BCM8706       0x00000500
 233 #define PORT_HW_CFG_XGXS_EXT_PHY_TYPE_BCM8726       0x00000600
 234 #define PORT_HW_CFG_XGXS_EXT_PHY_TYPE_BCM8481       0x00000700
 235 #define PORT_HW_CFG_XGXS_EXT_PHY_TYPE_SFX7101       0x00000800
 236 #define PORT_HW_CFG_XGXS_EXT_PHY_TYPE_FAILURE       0x0000fd00
 237 #define PORT_HW_CFG_XGXS_EXT_PHY_TYPE_NOT_CONN      0x0000ff00

but upstream seems to have that fixed:
#define PORT_HW_CFG_XGXS_EXT_PHY_TYPE_SFX7101       0x00000800
#define PORT_HW_CFG_XGXS_EXT_PHY_TYPE_BCM8727       0x00000900 <----
#define PORT_HW_CFG_XGXS_EXT_PHY_TYPE_BCM8727_NOC   0x00000a00

I'll check what patch introduced it.

Flavio


This event sent from IssueTracker by fbl 
 issue 356912

Comment 3 Issue Tracker 2009-10-29 18:53:52 UTC
Event posted on 10-29-2009 03:53pm BRT by fbl

This upstream commit might fix this problem:
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=4d295db0efd2ccf06edb7a45ad885b40c56b7161


This event sent from IssueTracker by fbl 
 issue 356912

Comment 4 Issue Tracker 2009-10-29 19:49:06 UTC
Event posted on 10-29-2009 04:49pm BRT by fbl

I've backported that upstream patch to gospo's kernel.
Brew build URL: https://brewweb.devel.redhat.com/taskinfo?taskID=2053999

Can you give a try and report back your results? 
I'd need the console output passing 'debug=0x20006' to the driver.

thanks,
Flavio


This event sent from IssueTracker by fbl 
 issue 356912
it_file 269225

Comment 6 Flavio Leitner 2009-11-03 19:17:16 UTC
Below is the feedback regarding to the last test kernel at comment#4:
---8<----
That works!

[bnx2x_set_int_mode:6838(eth2)]Multi requested but failed to enable MSI-X  set number of queues to 1
bnx2x: eth2: using MSI  IRQ 123
ADDRCONF(NETDEV_UP): eth2: link is not ready
bnx2x: eth2 NIC Link is Down
bnx2x: eth2 NIC Link is Down
bnx2x: eth2 NIC Link is Up, 10000 Mbps full duplex, receive & transmit flow control ON
ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready

[bnx2x_set_int_mode:6838(eth3)]Multi requested but failed to enable MSI-X  set number of queues to 1
bnx2x: eth3: using MSI  IRQ 131
ADDRCONF(NETDEV_UP): eth3: link is not ready
bnx2x: eth3 NIC Link is Down
bnx2x: eth3 NIC Link is Down
bnx2x: eth3 NIC Link is Up, 10000 Mbps full duplex, receive & transmit flow control ON
ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
---8<----

Flavio

Comment 7 Stanislaw Gruszka 2009-11-04 06:54:37 UTC
Flavio, thanks for solving this.

Upstream commit 4d295db0efd2ccf06edb7a45ad885b40c56b7161 will be included in RHEL5.5 driver update. Can I close this bug as duplicate of 515716 ?

Comment 8 Flavio Leitner 2009-11-04 13:31:40 UTC
Stanislaw, I think so. Do you know if there is a similar ticket for RHEL4?
I guess it will be requested for RHEL4 too.
thanks!

Comment 9 Stanislaw Gruszka 2009-11-04 14:15:10 UTC
We have no RHEL4 request for adding support for BCM8727 SFP+ PHY nor request to update driver to never version (that would kill me :). If customers will need this PHY support please open new bug report for RHEL4.

Comment 10 Stanislaw Gruszka 2009-11-04 14:16:56 UTC

*** This bug has been marked as a duplicate of bug 515716 ***

Comment 13 RHEL Program Management 2009-11-16 09:50:59 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.

Comment 17 Chris Ward 2009-11-19 13:52:11 UTC
@Cisco

We need to confirm that there is third-party commitment to 
test for the resolution of this request during the RHEL 5.5 
Beta Test Phase AND 5.3.z builds before we can approve it 
for acceptance into the release.

RHEL 5.5 Beta Test Phase is expected to begin around February
2010.

RHEL 5.4.z Test Phase begins immediately once a test build
is made available.

In order to avoid any unnecessary delays, please post a 
confirmation as soon as possible, including the contact 
information for testing engineers.

Any additional information about alternative testing variations we 
could use to reproduce this issue in-house would be appreciated.

Comment 24 Stanislaw Gruszka 2009-11-23 12:11:40 UTC
Here are test packages with bnx2x update (version 1.52.1-5) for RHEL5.5.

http://people.redhat.com/sgruszka/rhel5.5-broadcom/

Cisco, please test update works on your hardware.

Comment 25 Chris Ward 2009-12-04 09:32:09 UTC
Cisco IT reports back that they've successfully tested the bnx2x update provided in comment #24

Comment 29 Michael Hoffman 2010-03-15 17:28:37 UTC
Has this patch mentioned above been backported to RHEL5.4?

Comment 30 Andrius Benokraitis 2010-03-15 18:14:35 UTC
Have you tried kernel 2.6.18-164.8.1?


Note You need to log in before you can comment on or make changes to this bug.