Hello, I suspect the error could be rather in the firmware, as indicated by "Contact your hardware vendor for assistance.", or in the kernel driver. I have found only one occurrence of the error message on the Web: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1006641

I agree that this error seems to be specific to the HP hardware, I have EL9 running on Dell platforms that do not exhibit this issue.

I will add that I had/have this exact server running on RHEL 7.9 and do not see the issue on that release, so the problem is specific to RHEL9 and the iLO 5.

(In reply to Larry Mills from comment #2)

> I will add that I had/have this exact server running on RHEL 7.9 and do not
> see the issue on that release, so the problem is specific to RHEL9 and the
> iLO 5.

Thank you for the information. If the RHEL 7.9 ipmitool command works properly, I suggest installing it (or copying the binary from the RHEL 7.9 machine) on RHEL 9 and running it there (you will also need the old libraries /usr/lib64/libcrypto.so.10 /usr/lib64/libncurses.so.5 /usr/lib64/libreadline.so.6 /usr/lib64/libtinfo.so.5). That should determine whether the problem lies in the new ipmitool or in some other part of the RHEL 9 system (most likely the kernel).

Hello Pavel,


As you suggested, I copied the ipmitool binary from RHEL 7.9 along with the needed libraries over to the HP EL9 system.  The syslog error message(s) still occurred with the old binary, so it looks like the problem is not with ipmitool, but the kernel.   Likely module "ipmi_si"?

Thank you for your help with determining the root cause of this, yes, I believe it must be in the kernel. Tony Camuso, can you please have a look?

Hello Larry,

can you please provide the output of
ipmitool fru print

and 

ipmitool mc info

to show what firmware version is being used? (Just in case the error is specific to some firmware versions.)

Here is the requested output (S/N redacted):

FRU Device Description : Builtin FRU Device (ID 0)
 Chassis Type          : Rack Mount Chassis
 Chassis Serial        : XXX
 Board Mfg Date        : Mon Jan 10 18:00:00 2022
 Board Mfg             : HPE
 Board Product         : ProLiant DL380 Gen10
 Board Serial          : XXX
 Board Part Number     : 868705-B21
 Product Manufacturer  : HPE
 Product Name          : ProLiant DL380 Gen10
 Product Part Number   : 868705-B21
 Product Serial        : XXX

FRU Device Description : BMC CONTROLLER (ID 238)
 Product Manufacturer  : HPE
 Product Name          : BMC CONTROLLER
 Product Part Number   : iLO 5

FRU Device Description : MB BIOS (ID 239)
 Product Manufacturer  : HPE
 Product Name          : SYSTEM BIOS
 Product Part Number   : U30
 Product Version       : 03/08/2022

FRU Device Description : CPU 1 (ID 16)
 Product Manufacturer  : Intel(R) Corporation
 Product Name          : Intel(R) Xeon(R) Gold 6242R CPU @ 3.10GHz

FRU Device Description : CPU 2 (ID 17)
 Product Manufacturer  : Intel(R) Corporation
 Product Name          : Intel(R) Xeon(R) Gold 6242R CPU @ 3.10GHz

FRU Device Description : PSU 1 (ID 144)
 Product Manufacturer  : LTEON
 Product Name          : HpeServerPowerSupply
 Product Part Number   : P38995-B21
 Product Version       : 2.00
 Product Serial        : XXX
 Product Extra         : 800

FRU Device Description : PSU 2 (ID 145)
 Product Manufacturer  : LTEON
 Product Name          : HpeServerPowerSupply
 Product Part Number   : P38995-B21
 Product Version       : 2.00
 Product Serial        : XXX
 Product Extra         : 800

FRU Device Description : Embedded 0 (ID 2)
 Board Mfg Date        : Wed Sep  8 19:00:00 2021
 Board Part Number     : P07473-001
 Product Manufacturer  : STL
 Product Name          : HPE Smart Stor Hybridcap
 Product Part Number   : P02377-B21
 Product Version       : 01
 Product Serial        : XXX

FRU Device Description : Ethernet Adptr (ID 3)
 Product Name          : Empty

FRU Device Description : Ethernet Adptr (ID 4)
 Product Name          : HPE Ethernet 1Gb 4-port 331i Adapter - NIC

FRU Device Description : SAS Ctlr (ID 5)
 Product Name          : HPE Smart Array P816i-a SR Gen10
 Product Part Number   : 804341-002
 Product Version       : A
 Product Serial        : XXX

FRU Device Description : PCI-E Slot 1 (ID 6)
 Device not present (Requested sensor, data, or record not found)

FRU Device Description : PCI-E Slot 2 (ID 7)
 Device not present (Requested sensor, data, or record not found)

FRU Device Description : PCI-E Slot 3 (ID 8)
 Device not present (Requested sensor, data, or record not found)

FRU Device Description : Video (ID 9)
 Product Name          : Embedded Video Controller


[root@el9 tmp]# ipmitool mc info
Device ID                 : 19
Device Revision           : 3
Firmware Revision         : 2.65
IPMI Version              : 2.0
Manufacturer ID           : 47196
Manufacturer Name         : Unknown (0xB85C)
Product ID                : 8192 (0x2000)
Product Name              : Unknown (0x2000)
Device Available          : yes
Provides Device SDRs      : no
Additional Device Support :
    Sensor Device
    SDR Repository Device
    SEL Device
    FRU Inventory Device
    IPMB Event Receiver
    Chassis Device
Aux Firmware Rev Info     :
    0x00
    0x00
    0x00
    0x00

Thank you for the information, Larry.

(In reply to Larry Mills from comment #0)
> Description of problem:
> 
> When executing the command "ipmitool sensor list", the following error
> message is
> generated to syslog:
> 
> localhost kernel: ipmi_si IPI0001:00: IPMI message handler: IPMI message
> received with no owner. This could be because of a malformed message, or
> because of a hardware error. Contact your hardware vendor for assistance.
> 
> The sensor list command returns the expected sensor output, but an a syslog
> error is generated for each sensor value returned.
> 
> The problem occurs on a HP DL380 Gen10 server, which contains a iLO 5 BMC.
> 
> Version-Release number of selected component (if applicable):
> 
> Kernel:    5.14.0-119.el9.x86_64
> ipmitool:  1.8.18-25.el9
> HP iLO 5:  v2.65
> 
> 
> How reproducible:
> 
> 
> Steps to Reproduce:
> 1. On a HP system with an iLO 5, execute the command: "ipmitool sensor list"
>  
> 2. Observe the following output in the syslog, one output line per sensor
>    value:
> 
>  ipmi_si IPI0001:00: IPMI message handler: IPMI message received with no
> owner. This could be because of a malformed message, or because of a
> hardware error. Contact your hardware vendor for assistance.
> 
> Actual results:
> 
> 
> Expected results:
> 
> No IPMI-related errors in syslog
> 
> Additional info:

Indeed, I was able to reproduce this on a DL380 G10, but NOT on a DL380 G9, so there is something platform-specific going on there. I will try to find the problem with the G10.

Glad it could be reproduced.  I suspect it's more an iLO 5 issue, but I don't have another (i.e. non DL380) HP Gen10 platform available to test that theory.

(In reply to Larry Mills from comment #17)
> Glad it could be reproduced.  I suspect it's more an iLO 5 issue, but I
> don't have another (i.e. non DL380) HP Gen10 platform available to test that
> theory.

I have been able to isolate the problem to the ipmi updates I checked-in for RHEL-9.1.

I'm going to bisect to see if I can find the offending commit.

Found the commit that's causing the problem. 

Now I need to determine why.

commit e9af21e48eadc586cfb9f802cabb8aabf467e9a0
Author: Tony Camuso <tcamuso>
Date:   Tue Mar 22 16:11:34 2022 -0400

    ipmi: Export ipmb_checksum()
    
    Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2067267
    Brew: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=44059002
    Test: brew test.cki.ipmi
    Upstream Status: v5.17
    
    Cherry-picked from the following upstream commit.
    
    commit 1e4071f6282b3323435b02b1719bcfbfe1b57150
    Author: Corey Minyard <minyard>
    Date:   Fri Sep 24 07:12:42 2021 -0500
    
        ipmi: Export ipmb_checksum()
    
        It will be needed by the upcoming ipmb direct addressing.
    
        Signed-off-by: Corey Minyard <minyard>
        Tested-by: Andrew Manley <andrew.manley>
        Reviewed-by: Andrew Manley <andrew.manley>
    
    Signed-off-by: Tony Camuso <tcamuso>

 drivers/char/ipmi/ipmi_msghandler.c | 3 ++-
 include/linux/ipmi.h                | 3 +++
 2 files changed, 5 insertions(+), 1 deletion(-)

(In reply to Larry Mills from comment #17)
> Glad it could be reproduced.  I suspect it's more an iLO 5 issue, but I
> don't have another (i.e. non DL380) HP Gen10 platform available to test that
> theory.

Sorry, that was the wrong commit. THIS is the offending commit.

commit 24aebbc9389cfbc29043579359cd4a8668b0254a (HEAD)
Author: Tony Camuso <tcamuso>
Date:   Tue Mar 22 16:11:35 2022 -0400

    ipmi: Add support for IPMB direct messages
    
    Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2067267
    Brew: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=44059002
    Test: brew test.cki.ipmi
    Upstream Status: v5.17
    
    Cherry-picked from the following upstream commit.
    
    commit 059747c245f0e9af5e109eece7d3414dbe08d513
    Author: Corey Minyard <minyard>
    Date:   Fri Sep 24 11:42:56 2021 -0500
    
        ipmi: Add support for IPMB direct messages
    
        An application has come up that has a device sitting right on the IPMB
        that would like to communicate with the BMC on the IPMB using normal
        IPMI commands.
    
        Sending these commands and handling the responses is easy enough, no
        modifications are needed to the IPMI infrastructure.  But if this is an
        application that also needs to receive IPMB commands and respond, some
        way is needed to handle these incoming commands and send the responses.
    
        Currently, the IPMI message handler only sends commands to the interface
        and only receives responses from interface.  This change extends the
        interface to receive commands/responses and send commands/responses.
        These are formatted differently in support of receiving/sending IPMB
        messages directly.
    
        Signed-off-by: Corey Minyard <minyard>
        Tested-by: Andrew Manley <andrew.manley>
        Reviewed-by: Andrew Manley <andrew.manley>
    
    Signed-off-by: Tony Camuso <tcamuso>

(In reply to Larry Mills from comment #0)

I think I've found the problem. 

I will submit a patch upstream to see if I've correctly isolated and fixed the problem.

(In reply to Larry Mills from comment #0)

I was right. The bug was what I thought it was, but they already fixed it upstream.
Unfortunately, it was fixed after I had checked in my changes.

I will endeavor to get this into the RHEL-9.1.

commit 3d092ef09303e615707dc5755cf0e29b4df7555f
Author: Corey Minyard <cminyard>
Date:   Tue Apr 19 12:08:09 2022 -0500

    ipmi: When handling send message responses, don't process the message
    
    A chunk was dropped when the code handling send messages was rewritten.
    Those messages shouldn't be processed normally, they are just an
    indication that the message was successfully sent and the timers should
    be started for the real response that should be coming later.
    
    Add back in the missing chunk to just discard the message and go on.
    
    Fixes: 059747c245f0 ("ipmi: Add support for IPMB direct messages")
    Reported-by: Joe Wiese <jwiese>
    Cc: stable.org # v5.16+
    Signed-off-by: Corey Minyard <cminyard>
    Tested-by: Joe Wiese <jwiese>

diff --git a/drivers/char/ipmi/ipmi_msghandler.c b/drivers/char/ipmi/ipmi_msghandler.c
index c59265146e9c..5f7abf456a17 100644
--- a/drivers/char/ipmi/ipmi_msghandler.c
+++ b/drivers/char/ipmi/ipmi_msghandler.c
@@ -4518,6 +4518,8 @@ static int handle_one_recv_msg(struct ipmi_smi *intf,
                } else
                        /* The message was sent, start the timer. */
                        intf_start_seq_timer(intf, msg->msgid);
+               requeue = 0;
+               goto out;
        } else if (((msg->rsp[0] >> 2) != ((msg->data[0] >> 2) | 1))
                   || (msg->rsp[1] != msg->data[1])) {
                /*

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: kernel security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8267