Red Hat Bugzilla – Bug 624386
IBM BladeCenter HS22 stops responding, messes up network switch
Last modified: 2010-12-03 07:50:06 EST
Created attachment 438913 [details]
Ping log from the crash event displaying the network latency problem and disconnect.
Description of problem:
We have a number of HS22 blades in an IBM BladeCenter H, which also has a Nortel layer 2/3 gigabit switch (IBM product code 32R1860), to which all the blades are connected via chassis internal links. All the blades are running CentOS 5.5, except for one which is running Fedora 12.
Several times now the Fedora 12 blade has locked up. It doesn't respond to network requests, the screen is blank when viewed via the Advanced Management Module (a separate HW component in the chassis for managing the blades, connected via a separate network connection). In about a minute or so the AMM will report that management network connectivity has been lost to the Fedora 12 blade. After this happens, network traffic to other blades in the same chassis will first show increased latency and then stop entirely. Network functionality is restored immediately when the Fedora 12 blade is powered off.
Version-Release number of selected component (if applicable):
This happens relatively infrequently, every few weeks. Today's crash occurred during moderate network activity (running yum update).
All that log shows is that something is wrong.
What kind of network adapter is in that machine? Is there anything in the system log when that happens?
Created attachment 439123 [details]
lspci -v output on the affected machine
Dual Broadcom Corporation NetXtreme II BCM5709S Gigabit Ethernet (rev 20). lspci -v attached.
At least /var/log/messages shows just normal looking activity, and then the messages after reboot, any other files I should check?
Also, the problem is not related to broken hardware, since the crashes moved to a different machine after swapping the Fedora 12 RAID-1 system drives to it. Under CentOS 5.5 all the blades seem to work fine.
Can you try booting with the kernel option "pcie_aspm=off"? You could also try "pci=nomsi" in addition to that but it's doubtful that MSI is causing any problems.
Sorry, I didn't notice your comment before now. I thought Bugzilla would automatically send updates to the bug reporter, but apparently I need to add myself manually to the CC list.
The crash happened again this week with the same kernel. I'll try rebooting with that option and we'll see if it helps. It may take another month though before it crashes again.
This message is a reminder that Fedora 12 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 12. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora
'version' of '12'.
Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version prior to Fedora 12's end of life.
Bug Reporter: Thank you for reporting this issue and we are sorry that
we may not be able to fix it before Fedora 12 is end of life. If you
would still like to see this bug fixed and are able to reproduce it
against a later version of Fedora please change the 'version' of this
bug to the applicable version. If you are unable to change the version,
please add a comment here and someone will do it for you.
Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.
The process we are following is described here:
Fedora 12 changed to end-of-life (EOL) status on 2010-12-02. Fedora 12 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.
If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version.
Thank you for reporting this bug and we are sorry it could not be fixed.