Co-dependent on openipmi package update https://bugzilla.redhat.com/show_bug.cgi?id=475542

Created attachment 326650 [details]
Fix for bug that causes SOL sessions to randomly drop



The attached patch fixes a bug that's been lurking for a long time,
first in the lan plugin and then propagated to the lanplus plugin.  If
there are multiple entries on the ipmi_req_entries linked list, and
ipmi_req_remove_entry() is called to remove the first entry, all
additional entries are also removed from the list, causing problems when
those requests complete.  This condition is rather rare but seems to
occur more frequently under high latency conditions on the network.

IBM is signed up to test and provide feedback.

A full scale driver update is too large for RHEL 5.4.  Please be more specific with your request and identify specific bug fixes.  Comment #2 is not a kernel patch, if that is what you want fixed then we should change the component on this BZ.

(In reply to comment #7)
> A full scale driver update is too large for RHEL 5.4.  Please be more specific
> with your request and identify specific bug fixes.  Comment #2 is not a kernel
> patch, if that is what you want fixed then we should change the component on
> this BZ.
>

Hi Peter,

The patch/fix in Comment #2 has been applied to the ipmitool project and will be included in the imminent roll of ipmitool to 1.8.11.  We have another  BZ/Request for 5.4 upgrades for that utility.

The most critical OpenIPMI driver fixes we would want to have pulled into RHEL5.4 are:

-- It may sound trivial, but it's very difficult to find all the real code diffs with all the comment and white space diffs without the style patches applied.  Pulling in the style patches will go a long way toward making the drivers more maintainable/debuggable:

ipmi-style-fixes-in-the-base-code.patch
ipmi-style-fixes-in-the-system-interface-code.patch
ipmi-style-fixes-in-the-system-interface-code-checkpatch-fixes.patch
ipmi-style-fixes-in-the-misc-code.patch

-- A fix for a suspend/resume crash.  Note that it affects two ipmi driver files.  The part of the patch for drivers/hwmon/ibmaem.c doesn't seem to be in the RHEL5.3 code can be ignored (according to the patch author DJ Wong).
http://fixunix.com/kernel/555930-patch-fix-platform-drivers-crash-suspend-resume.html

-- lock changes to prevent a race condition.
http://lkml.org/lkml/2008/6/25/393

-- some signedness issues
http://linux.derkeiler.com/Mailing-Lists/Kernel/2007-10/msg04439.html

-- fix to hold ATTN until upper layer is ready
ipmi-hold-attn-until-upper-layer-ready.patch
http://osdir.com/ml/linux.kernel.openipmi/2008-02/msg00013.html

-- add ipmi_poll_interface to make sure watchdog timeout message on a panic goes out
http://lkml.org/lkml/2007/10/12/289

-- fix broken run-to-completion code:
ipmi-run-to-completion-fixes.patch
ipmi-run-to-completion-fixes-fix.patch
ipmi-run-to-completion-fixes-checkpatch-fixes.patch
ipmi-dont-grab-locks-in-run-to-completion-mode.patch
ipmi-dont-grab-locks-in-run-to-completion-mode-fix.patch

-- support for enabling/disabling interrupts upon system startup/shutdown
http://lkml.org/lkml/2007/2/14/228

Thank you.

Updating PM score.

I will review the patch sets posted in comment #5 for inclusion in RHEL 5.4.

(In reply to comment #10)
> I will review the patch sets posted in comment #5 for inclusion in RHEL 5.4.

Thank you very much.  :-)  For 5.4, could you also please consider swapping out the ipmi_dev_order=2 workaround added to RHEL5.3 for the one-line dmi_scan.c fix which is in the upstream kernel to fix the ipmi device nodes out-of-order issue?    (Reference RH430157/LTC41679)

Thanks again.

What is the value add in providing the polled interface to the watchdog driver?
Are there specific customer requests for this to be added?

-- add ipmi_poll_interface to make sure watchdog timeout message on a panic
goes out
http://lkml.org/lkml/2007/10/12/289

I cannot swap out the ipmi_dev_order=2 workaround, as stated in the original BZ it would break kAPI for existing RHEL customers.

We do not need the fasync() BKL pushdown patch as fasync() still runs with BKL protection in the dev RHEL 5.4 tree.

------- Comment From hebertc.com 2009-03-18 18:27 EDT-------
(In reply to comment #12)
> What is the value add in providing the polled interface to the watchdog driver?
> Are there specific customer requests for this to be added?
> -- add ipmi_poll_interface to make sure watchdog timeout message on a panic
> goes out
> http://lkml.org/lkml/2007/10/12/289

I don't know of a specific customer request for this to be added at this time.  However, the number of our customers using the IPMI hardware watchdog is expected to continue increasing;  inclusion of  watchdog driver fixes, or functionality and usability enhancements now may decrease the likelihood that we will be faced with a customer issue to resolve in the future..

> I cannot swap out the ipmi_dev_order=2 workaround, as stated in the original BZ
> it would break kAPI for existing RHEL customers.

Although I think only IBM customers have this multinode/multiBMC issue, I understand your concern and can appreciate using caution..  The workaround is not ideal but it is a workaround.

Thanks much.

The deadline for kernel code submissions is quickly approaching. Is this on track for the 4/23 kernel freeze? If you need help with testing, access to hardware, etc., just let me know. Thanks!

Jesse

Test kernels are available, email sent out to the partners.

------- Comment From hebertc.com 2009-04-17 22:23 EDT-------
(In reply to comment #16)
> Test kernels are available, email sent out to the partners.

Hi,

We've tested the i686, x86_64, src, and the i686 and x86_64 xen rpms with a focus on ipmi wellness and functionality on a dual-node x3950 and on an dx340 Server.  Additionally, Suspend/Resume functionality was tested and also Pounder/LTP was run concurrently with a loop which performed a ipmi driver load/unload and various ipmitool info/fru/sdr/sel print routines and some testing of the ipmi_watchdog driver.

General ipmi  wellness tests:    passed on all 4 tested kernels.

Resume/Suspend:  required the ipmi_si driver to be unloaded before Suspend would work (the kipmi<N> process(es) prevented suspend).  If your suspend scripts have a list of known "can't sleep/resume with these modules loaded" drivers, maybe we could put ipmi_si onto that list and have it automatically unloaded?  Otherwise,  maybe we could just release note this extra procedural step. Note that since the ipmi_watchdog and ipmi_poweroff drivers don't require the ipmi_si driver, requiring an unload of that one driver should be fine.   The resume/suspend patch itself is fine.

Please let me know if you need anything else done with these kernels.

Thank you very much,

This enhancement request was evaluated by the full Red Hat Enterprise Linux 
team for inclusion in a Red Hat Enterprise Linux minor release.   As a 
result of this evaluation, Red Hat has tentatively approved inclusion of 
this feature in the next Red Hat Enterprise Linux Update minor release.   
While it is a goal to include this enhancement in the next minor release 
of Red Hat Enterprise Linux, the enhancement is not yet committed for 
inclusion in the next minor release pending the next phase of actual 
code integration and successful Red Hat and partner testing.

in kernel-2.6.18-141.el5
You can download this test kernel from http://people.redhat.com/dzickus/el5

Please do NOT transition this bugzilla state to VERIFIED until our QE team
has sent specific instructions indicating when to do so.  However feel free
to provide a comment indicating that this fix has been verified.

*** Bug 225432 has been marked as a duplicate of this bug. ***

~~ Attention Partners RHEL 5.4 Partner Alpha Released! ~~

RHEL 5.4 Partner Alpha has been released on partners.redhat.com. There should
be a fix present that addresses this particular request. Please test and report back your results here, at your earliest convenience. Our Public Beta release is just around the corner!

If you encounter any issues, please set the bug back to the ASSIGNED state and
describe the issues you encountered. If you have verified the request functions as expected, please set your Partner ID in the Partner field above to indicate successful test results. Do not flip the bug status to VERIFIED. Further questions can be directed to your Red Hat Partner Manager. Thanks!

------- Comment From hebertc.com 2009-06-24 18:32 EDT-------
(In reply to comment #22)

Hi,

I've run an ipmi in-band test suite to verify general driver wellness on a 2-node x3950 running the 32-bit,  32bitXEN, 32-bitPAE, 64-bit, and 64-bitXEN kernels.  All tests passed.  The driver version is verified to be 39.1:

ipmi message handler version 39.1
IPMI System Interface driver.

The tests were repeated on an HS20 running the 32-bit, 32-bitPAE, and 32-bitXEN kernels and an issue was found with the ipmi_watchdog driver and the ipmi startup script.  The HS20 system has on-chip watchdog hardware whose driver (i6300esb) "grabs" the /dev/watchdog (misc) device at boot time.  When the ipmi_watchdog driver is subsequently loaded, the load should fail, however it appears to "half fail" leaving the driver in an unusable state but appearing to be loaded and available:

1)  When running "service ipmi start" with the ipmi_watchdog driver configured to be loaded (via the /etc/sysconfig/ipmi file), the ipmi_watchdog driver appears to have been loaded ok:

# service ipmi start
Starting ipmi drivers:                                     [  OK  ]
Starting ipmi_watchdog driver:                             [  OK  ]

However, it was not loaded/initialized properly:

# dmesg |tail
SELinux: initialized (dev autofs, type autofs), uses genfs_contexts
eth0: no IPv6 routers present
ipmi message handler version 39.1
IPMI System Interface driver.
ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca8, slave address 0x20, irq 0
ipmi: Found new BMC (man_id: 0x000002,  prod_id: 0x0043, dev_id: 0x20)
IPMI kcs interface initialized
ipmi device interface
IPMI Watchdog: Unable to register misc device
IPMI Watchdog: driver initialized

Subsequent testing proves that although the ipmi_watchdog driver appears to be loaded, "ipmitool mc watchdog get" shows that the config settings in the /etc/sysconfig/ipmi file have not been set:

# grep WATCHDOG /etc/sysconfig/ipmi
## Description: Enable IPMI_WATCHDOG if you want the IPMI watchdog
# Enable IPMI_WATCHDOG if you want the IPMI watchdog
IPMI_WATCHDOG=yes
IPMI_WATCHDOG_OPTIONS="timeout=60 action=power_cycle start_now=1"

# ipmitool mc watchdog get
Watchdog Timer Use:     BIOS FRB2 (0x01)
Watchdog Timer Is:      Stopped
Watchdog Timer Actions: No action (0x00)
Pre-timeout interval:   0 seconds
Timer Expiration Flags: 0x02
Initial Countdown:      0 sec
Present Countdown:      0 sec
[root@elm3a27 PAE]# lsmod |grep ipmi
ipmi_watchdog          21129  0
ipmi_devintf           13129  0
ipmi_si                42829  0
ipmi_msghandler        39153  3 ipmi_watchdog,ipmi_devintf,ipmi_si

Subsequent watchdog testing continues confirmation that it is not working.

The problem is that the i6300esb driver has the same major/minor device numbers and is already using /dev/watchdog as of boot-time:
# lsmod |grep i6300
i6300esb               10841  0

# modinfo i6300esb
filename:       /lib/modules/2.6.18-152.el5xen/kernel/drivers/char/watchdog/i6300esb.ko
alias:          char-major-10-130
license:        GPL
description:    Watchdog driver for Intel 6300ESB chipsets
author:         Ross Biro and David H????rdeman
srcversion:     2A37792AAD84EC032278ECA
alias:          pci:v00008086d000025ABsv*sd*bc*sc*i*
depends:
vermagic:       2.6.18-152.el5xen SMP mod_unload 686 REGPARM 4KSTACKS gcc-4.1
parm:           heartbeat:Watchdog heartbeat in seconds. (1<heartbeat<2046, default=30) (int)
parm:           nowayout:Watchdog cannot be stopped once started (default=CONFIG_WATCHDOG_NOWAYOUT) (int)
module_sig:     883f3504a27af664a752b13a67179611257f509b6e87ea7e62e95c15ef1c26a12e3c75ff23aca409e33dcbcb4b2b5a9bbc6880e99fd6082d91b292c

Note that if the ipmi drivers are unloaded (must use "service ipmi stop-all" or an equivalent to unload the ipmi_watchdog driver) and then if the i6300esb driver is unloaded (the /dev/watchdog device node is removed at that point) and then if the ipmi drivers are reloaded ("service ipmi start"), all appears to be working well with the ipmi_watchdog driver.

So, it seems there are two problems with ipmi on systems that have a second on-board watchdog chip:

1)  the init.d/ipmi script does not properly return a failure when the ipmi_watchdog driver is improperly loaded.

2) the ipmi driver does not return a failure to the startup script for reporting to the user when the ipmi_watchdog driver can not be fully loaded.  Additionally, the ipmi_watchdog driver remains "partially" loaded so a user might think it was operational but it is not.

2.5) A release note or Tech Tip is needed to help customers recognize when they will need to move any on-board (very spartan) watchdog driver out of the way to allow use of the ipmi_watchdog driver and how to accomplish this move/switch.

Please note that this issue is not caused by the driver updates added to the ipmi drivers for v5.4 and discussed earlier in this BZ. I can reproduce it on an x3550 M2  2.6.30 kernel where there is driver support for an Intel TCO Watchdog Timer device (driver is iTCO_wdt; which supports the ICH10 TCO device):
00:1f.0 ISA bridge: Intel Corporation 82801JIB (ICH10) LPC Interface Controller

As per the instructions on the previous comment for issues found, I believe we should move this BZ back to  "Assigned".

Thank you.

Please open a new Bugzilla to report any issues found during LTC testing, and move this bug back to ON_QA.  Please include test results when running RHEL 5.3 in the new Bugzilla.  Regardless of this being an upstream issue or a regression I cannot submit a second patch set against the same Bugzilla number.

------- Comment From hebertc.com 2009-06-25 15:45 EDT-------
(In reply to comment #24)
> Please open a new Bugzilla to report any issues found during LTC testing, and
> move this bug back to ON_QA.  Please include test results when running RHEL 5.3
> in the new Bugzilla.  Regardless of this being an upstream issue or a
> regression I cannot submit a second patch set against the same Bugzilla number.

OK -- will do.  Thanks.

Note: additionally, make sure to link to the new bug from here.

Moving back to ON_QA.

~~ Attention - RHEL 5.4 Beta Released! ~~

RHEL 5.4 Beta has been released! There should be a fix present in the Beta release that addresses this particular request. Please test and report back results here, at your earliest convenience. RHEL 5.4 General Availability release is just around the corner!

If you encounter any issues while testing Beta, please describe the issues you have encountered and set the bug into NEED_INFO. If you encounter new issues, please clone this bug to open a new issue and request it be reviewed for inclusion in RHEL 5.4 or a later update, if it is not of urgent severity.

Please do not flip the bug status to VERIFIED. Only post your verification results, and if available, update Verified field with the appropriate value.

Questions can be posted to this bug or your customer or partner representative.

~~ Attention Partners - RHEL 5.4 Snapshot 1 Released! ~~

RHEL 5.4 Snapshot 1 has been released on partners.redhat.com. If you have already reported your test results, you can safely ignore this request. Otherwise, please notice that there should be a fix available now that addresses this particular request. Please test and report back your results here, at your earliest convenience. The RHEL 5.4 exception freeze is quickly approaching.

If you encounter any issues while testing Beta, please describe the issues you have encountered and set the bug into NEED_INFO. If you encounter new issues, please clone this bug to open a new issue and request it be reviewed for inclusion in RHEL 5.4 or a later update, if it is not of urgent severity.

Do not flip the bug status to VERIFIED. Instead, please set your Partner ID in the Verified field above if you have successfully verified the resolution of this issue. 

Further questions can be directed to your Red Hat Partner Manager or other appropriate customer representative.

Could someone point us to the new bugs filed, as described in comment #21. Thanks

As I understand it, the patch for this bug is the one attached to this BZ.
If anyone has opened a bz for the issues in comment #21 I am not aware of it.

Peter, is this correct?

------- Comment From bigmach.com 2009-07-29 16:14 EDT-------
(In reply to comment #31)
> As I understand it, the patch for this bug is the one attached to this BZ.
> If anyone has opened a bz for the issues in comment #21 I am not aware of it.
>
> Peter, is this correct?
>

I filed a bug today for this issue, it is in the process of being mirrored.

------- Comment From kamaleshb.com 2009-07-30 02:11 EDT-------
(In reply to comment #32)
> (In reply to comment #31)
> > As I understand it, the patch for this bug is the one attached to this BZ.
> > If anyone has opened a bz for the issues in comment #21 I am not aware of it.
> >
> > Peter, is this correct?
> >
>
> I filed a bug today for this issue, it is in the process of being mirrored.
>
Redhat BugNo#514678 and LTC bugNo#55129 has been raised.

Guy - This bug was used to submit kernel changes based on comments #4 and #5.  I only used comment #5 to identify the areas of interest being reported by the submitter, the patches that I submitted were based on the upstream kernel.  Bug 514678 will cover the service ipmi start issue reported in comment #21 as I requested in comment #23.

The original ipmitool patch in comment #2, if still required, needs to be addressed in a separate bugzilla.

------- Comment From bigmach.com 2009-07-31 17:14 EDT-------
(In reply to comment #34)
> ...
> The original ipmitool patch in comment #2, if still required, needs to be
> addressed in a separate bugzilla.

The patch from comment #2 is in the upstream ipmitool-1.8.11 source and is therefore no longer needed.

~~ Attention Partners - RHEL 5.4 Snapshot 5 Released! ~~

RHEL 5.4 Snapshot 5 is the FINAL snapshot to be release before RC. It has been 
released on partners.redhat.com. If you have already reported your test results, 
you can safely ignore this request. Otherwise, please notice that there should be 
a fix available now that addresses this particular issue. Please test and report 
back your results here, at your earliest convenience.

If you encounter any issues while testing Beta, please describe the 
issues you have encountered and set the bug into NEED_INFO. If you 
encounter new issues, please clone this bug to open a new issue and 
request it be reviewed for inclusion in RHEL 5.4 or a later update, if it 
is not of urgent severity. If it is urgent, escalate the issue to your partner manager as soon as possible. There is /very/ little time left to get additional code into 5.4 before GA.

Partners, after you have verified, do not flip the bug status to VERIFIED. Instead, please set your Partner ID in the Verified field above if you have successfully verified the resolution of this issue. 

Further questions can be directed to your Red Hat Partner Manager or other 
appropriate customer representative.

IBM, can you please verify this feature is functioning correctly in 5.4?

IBM, please update us with the latest test results for confirming the
resolution of this request. Thanks.

------- Comment From coschult.com 2009-08-10 12:17 EDT-------
Verified under 64-bit RHEL 5.4 snap 5 on an x3850 M2.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2009-1243.html

------- Comment From prem.karat.ibm.com 2011-02-22 05:01 EDT-------
(In reply to comment #40)
> An advisory has been issued which should help the problem
> described in this bug report. This report is therefore being
> closed with a resolution of ERRATA. For more information
> on therefore solution and/or where to find the updated files,
> please follow the link below. You may reopen this bug report
> if the solution does not work for you.
>
> http://rhn.redhat.com/errata/RHSA-2009-1243.html

***Reviewed as a part of clean up activity********

Closing this one out as per the last comment

Cheers,
Prem