Bug 598946

Created attachment 419001 [details]
QLogic firmware dump

Created attachment 419002 [details]
/var/log/messages for the above issue

QLogic has released OOT (out-of-train) drivers with 5.03.02 firmware
for RHEL5.x (8.03.01.06):

http://driverdownloads.qlogic.com/QLogicDriverDownloads_UI/SearchByProduct.aspx?ProductCategory=39&Product=1044&Os=65

Patches are also being pushed for RHEL5.6 that include the updated
firmware.

(In reply to comment #3)
> Patches are also being pushed for RHEL5.6 that include the updated
> firmware.    

Any plans of including this updated driver/firmware in 5.5.z itself? If so, that would be really helpful.

(In reply to comment #4)
> (In reply to comment #3)
> > Patches are also being pushed for RHEL5.6 that include the updated
> > firmware.    
> 
> Any plans of including this updated driver/firmware in 5.5.z itself? If so,
> that would be really helpful.    

No, firmware is not distributed via z-stream. Any updates to the firmware should be made through Qlogic - Red Hat has no visibility/support statements regarding certain firmware versions. Red Hat defers to Qlogic on stuff like this.

(In reply to comment #5)
> No, firmware is not distributed via z-stream. Any updates to the firmware
> should be made through Qlogic - Red Hat has no visibility/support statements
> regarding certain firmware versions. Red Hat defers to Qlogic on stuff like
> this.    

But seems QLogic bundles the firmware along with the native inbox driver now. The current 4.04.xx firmware has issues causing errors on the RHEL 5.5 host as reported above. 

Since RHEL 5.6 is a long way off, we are requesting for the upgraded native inbox FC driver v8.03.01.06.05.06-k (which packages the fixed firmware v5.03.02 in it) for 5.5.z itself - since the current native RHEL 5.5 QLogic FC solution is buggy.

(In reply to comment #6)
> (In reply to comment #5)
> > No, firmware is not distributed via z-stream. Any updates to the firmware
> > should be made through Qlogic - Red Hat has no visibility/support statements
> > regarding certain firmware versions. Red Hat defers to Qlogic on stuff like
> > this.    
> 
> But seems QLogic bundles the firmware along with the native inbox driver now.
> The current 4.04.xx firmware has issues causing errors on the RHEL 5.5 host as
> reported above. 
> 
> Since RHEL 5.6 is a long way off, we are requesting for the upgraded native
> inbox FC driver v8.03.01.06.05.06-k (which packages the fixed firmware v5.03.02
> in it) for 5.5.z itself - since the current native RHEL 5.5 QLogic FC solution
> is buggy.    

Not quite - upgrading of the firmware inbox or out-of-box doesn't present the same issues as using an inbox or out-of-box driver. Firmware is a blob that RH doesn't have source code for, so it doesn't matter where it comes from. Red Hat defers to QLogic anyway for a firmware issue, even if it is distributed by Red Hat. Customers can (and should) grab the latest firmware from QLogic since it is released much more often.

Closing - QLogic will update firmware as part of the normal process in RHEL 5.6.

At the request of QLogic - will open up additional dialogue with NetApp.

NetApp - how frequently are you hitting this?

(In reply to comment #9)
> At the request of QLogic - will open up additional dialogue with NetApp.
> 
> NetApp - how frequently are you hitting this?    

Seen consistently during heavy IO.

*** Bug 604134 has been marked as a duplicate of this bug. ***

Re-opening -- QLogic to look into this.

QLogic, any update here?

Tanvi,

QLogic is investigating if the reward outweighs the risk. Updating a full-blown firmware via z-stream will be largely *untested* by Red Hat and will introduce additional risk. Stay tuned.

Andrius.

Qlogic to provide testing information of the firmware update under the rhel5.5 environment.

This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.

Based on discussions with Tom Coughlan and Rob Evers, and weighing the risk/reward and seriousness of this defect, RH Engineering is recommendeding this be proposed for RHEL 5.5.z.

You can get a test 5.5.z kernel with the new firmware here:

http://people.redhat.com/coughlan/bz598946/

The test kernel looks good - we don't see the QLogic firmware errors now.

in kernel-2.6.18-209.el5
You can download this test kernel from http://people.redhat.com/jwilson/el5

Detailed testing feedback is always welcomed.

Reminder! There should be a fix present for this BZ in snapshot 3 -- unless otherwise noted in a previous comment.

Please test and update this BZ with test results as soon as possible.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0017.html

Andrew,
Does this symptom happen on ISP2532 (HPAK344A) with fw=4.04.05?
One customer saw this "ISP System Error" only once.

I found the firmware release not, but I have no idea which error
report correspond to this issue.
http://filedownloads.qlogic.com/files/software/75742/release.pdf

Thanks

(In reply to comment #34)
> Andrew,
> Does this symptom happen on ISP2532 (HPAK344A) with fw=4.04.05?
> One customer saw this "ISP System Error" only once.
> 
> I found the firmware release not, but I have no idea which error
> report correspond to this issue.
> http://filedownloads.qlogic.com/files/software/75742/release.pdf
> 
> Thanks

It's possible.  The problem in this BZ was originally with the 4.04.09 firmware.  I'd recommend using the latest RHEL 5 release (RHEL 5.7 as of this date) as it has the updated firmware.