RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1951480 - "ipmitool sol activate" crashes constantly
Summary: "ipmitool sol activate" crashes constantly
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: ipmitool
Version: 8.3
Hardware: All
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: Pavel Cahyna
QA Contact: Jeff Bastian
Prerana Sharma
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-04-20 08:56 UTC by Renaud Métrich
Modified: 2021-11-26 15:00 UTC (History)
4 users (show)

Fixed In Version: ipmitool-1.8.18-18.el8
Doc Type: Bug Fix
Doc Text:
.`ipmitool sol activate` command no longer crashes Previously, after upgrading from RHEL 7 to RHEL 8 the `ipmitool sol activate` command would crash while trying to access the remote console on an IBM DataPower appliance. With this update, the bug has been fixed and one can use `ipmitool` to access the remote console again.
Clone Of:
Environment:
Last Closed: 2021-11-09 18:50:34 UTC
Type: Bug
Target Upstream Version:
Embargoed:
pm-rhel: mirror+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 6542401 0 None None None 2021-11-26 15:00:20 UTC
Red Hat Product Errata RHBA-2021:4307 0 None None None 2021-11-09 18:50:35 UTC

Description Renaud Métrich 2021-04-20 08:56:54 UTC
Description of problem:

A customer reported that since upgrading his systems from RHEL7 to RHEL8.3, he could see "ipmitool sol activate" command crash constantly when trying to access the remote console.

Coredump analysis below:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
Core was generated by `ipmitool -I lanplus -L OPERATOR -H <IP ADDR> -U user-id1 -P XXXXXXXXX sol a'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __memmove_ssse3 () at ../sysdeps/x86_64/multiarch/memcpy-ssse3.S:2831
2831        movdqu    0x40(%rsi), %xmm4
(gdb) up
#1  0x000055c9c74c1d15 in memmove (__len=<optimized out>, __src=0x55c9c772b816 <rsp+22>, 
    __dest=0x55c9c772b801 <rsp+1>) at /usr/include/bits/string_fortified.h:40
warning: Source file is more recent than executable.
40      return __builtin___memmove_chk (__dest, __src, __len, __bos0 (__dest));
(gdb) up
#2  ipmi_lan_poll_single (intf=intf@entry=0x55c9c7728720 <ipmi_lanplus_intf>) at lanplus.c:819
819                    memmove(rsp->data, rsp->data + offset, extra_data_length);
(gdb) list
814                 * rsp->data_len becomes the length of that data
815                 */
816                extra_data_length = payload_size - (offset - payload_start) - 1;
817                if (extra_data_length) {
818                    rsp->data_len = extra_data_length;
819                    memmove(rsp->data, rsp->data + offset, extra_data_length);
820                } else {
821                    rsp->data_len = 0;
822                }
823            }
(gdb) p rsp->data_len 
$1 = 117440512
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

From above, the extra_data_length is just huge, it cannot be correct. This causes unallocated memory to be reached.
Below is the source code from RHEL 8.3.0 codebase (src/plugins/lanplus/lanplus.c file):
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
 811                         /*
 812                          * Good packet.  Shift response data to start of array.
 813                          * rsp->data becomes the variable length IPMI response data
 814                          * rsp->data_len becomes the length of that data
 815                          */
 816                         extra_data_length = payload_size - (offset - payload_start) - 1;
 817                         if (extra_data_length) {
 818                                 rsp->data_len = extra_data_length;
 819                                 memmove(rsp->data, rsp->data + offset, extra_data_length);
 820                         } else {
 821                                 rsp->data_len = 0;
 822                         }
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

On line 817 we see that the condition is entered (because extra_data_length != 0) and the memmove() happens.
Similar code is seen on the RHEL 7.9 codebase.

However, checking the Upstream code, it's different: the condition is more robust:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
 781                         /*
 782                          * Good packet.  Shift response data to start of array.
 783                          * rsp->data becomes the variable length IPMI response data
 784                          * rsp->data_len becomes the length of that data
 785                          */
 786                         extra_data_length = payload_size - (offset - payload_start) - 1;
 787                         if (extra_data_length > 0) {
 788                                 rsp->data_len = extra_data_length;
 789                                 memmove(rsp->data, rsp->data + offset, extra_data_length);
 790                                 offset = 0;
 791                                 payload_start = 0;
 792                                 payload_size = extra_data_length;
 793                         } else {
 794                                 rsp->data_len = 0;
 795                         }
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

The latter was brought by commit below:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
commit 9ec2232321a7bca7e1fb8f939d071f12c8dfa7fd
Author: pjdhpe <44778156+pjdhpe.github.com>
Date:   Wed Nov 28 07:27:29 2018 -0600

    lanplus: Fix segfault for truncated dcmi response

    On occasion a dcmi power reading will return error C6, and a
    truncated response payload. As the decrypted payload is shorter
    than the expected length, lanplus_decrypt_aes_cbc_128() adjusts
    the payload_size downward by one byte. In ipmi_lan_poll_single()
    the calculation to determine if the payload size has increased
    erroniously sets extra_data_length to -1, with a subsequent
    segv when calling a memmove to shift response data.
    The fix is to check for a positive value in the extra_data_length.

    Resolves ipmitool/ipmitool#72
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

We need this commit to be backported, I gave a test package to the customer and he confirmed it fixes the issue.

Version-Release number of selected component (if applicable):

ipmitool-1.8.18-17.el8_3


How reproducible:

Always on customer systems

Comment 1 Pavel Cahyna 2021-04-20 09:10:03 UTC
Thank you for the detailed investigation. What system model/manufacturer have the problem? I suppose QE will like to know this so that they may try to reproduce it on our test systems.

Comment 2 Renaud Métrich 2021-04-20 09:50:31 UTC
Apparently it's a IBM DataPower appliance.

The weird thing is this didn't crash on RHEL7 even though code was similar.

Comment 3 Pavel Cahyna 2021-04-20 10:01:18 UTC
This? https://en.wikipedia.org/wiki/IBM_WebSphere_DataPower_SOA_Appliances

Sounds like something that will be needed to be tested by the customer, unless it is possible to reproduce the problem on more usual hardware.

Comment 8 Jeff Bastian 2021-07-21 18:25:54 UTC
Regression testing passsed with ipmitool-1.8.18-18.el8:
https://beaker.engineering.redhat.com/jobs/5604675
https://beaker.engineering.redhat.com/jobs/5604717

I also manually tested 'ipmitool sol activate' on the various RHEL architectures, although, I don't have an IBM DataPower appliance to use for testing, so I picked a random Beaker system with a BMC.

:::::::::::::
:: aarch64 ::
:::::::::::::

[root@hpe-apollo-cn99xx-15-vm-26 ~]# rpm -q ipmitool
ipmitool-1.8.18-18.el8.aarch64

[root@hpe-apollo-cn99xx-15-vm-26 ~]# ipmitool -I lanplus -H ampere-mtsnow-altra-04-bmc.mgmt.lab.eng.rdu2.redhat.com -U admin -E sol deactivate
Info: SOL payload already de-activated

[root@hpe-apollo-cn99xx-15-vm-26 ~]# ipmitool -I lanplus -H ampere-mtsnow-altra-04-bmc.mgmt.lab.eng.rdu2.redhat.com -U admin -E sol activate
[SOL Session operational.  Use ~? for help]
Red Hat Enterprise Linux 8.4 (Ootpa)
Kernel 4.18.0-305.el8.aarch64 on an aarch64

ampere-mtsnow-altra-04 login: 


:::::::::::::
:: ppc64le ::
:::::::::::::

[root@ibm-p9z-20-lp16 ~]# rpm -q ipmitool
ipmitool-1.8.18-18.el8.ppc64le

[root@ibm-p9z-20-lp16 ~]# ipmitool -I lanplus -H ampere-mtsnow-altra-04-bmc.mgmt.lab.eng.rdu2.redhat.com -U admin -E sol deactivate
Info: SOL payload already de-activated

[root@ibm-p9z-20-lp16 ~]# ipmitool -I lanplus -H ampere-mtsnow-altra-04-bmc.mgmt.lab.eng.rdu2.redhat.com -U admin -E sol activate
[SOL Session operational.  Use ~? for help]
Red Hat Enterprise Linux 8.4 (Ootpa)
Kernel 4.18.0-305.el8.aarch64 on an aarch64

ampere-mtsnow-altra-04 login: 


:::::::::::
:: s390x ::
:::::::::::

[root@ibm-z-132 ~]# rpm -q ipmitool
ipmitool-1.8.18-18.el8.s390x

[root@ibm-z-132 ~]# ipmitool -I lanplus -H ampere-mtsnow-altra-04-bmc.mgmt.lab.eng.rdu2.redhat.com -U admin -E sol deactivate
Info: SOL payload already de-activated

[root@ibm-z-132 ~]# ipmitool -I lanplus -H ampere-mtsnow-altra-04-bmc.mgmt.lab.eng.rdu2.redhat.com -U admin -E sol activate
[SOL Session operational.  Use ~? for help]
Red Hat Enterprise Linux 8.4 (Ootpa)
Kernel 4.18.0-305.el8.aarch64 on an aarch64

ampere-mtsnow-altra-04 login: 


::::::::::::
:: x86_64 ::
::::::::::::

[root@kvm-06-guest16 ~]# rpm -q ipmitool
ipmitool-1.8.18-18.el8.x86_64

[root@kvm-06-guest16 ~]# ipmitool -I lanplus -H ampere-mtsnow-altra-04-bmc.mgmt.lab.eng.rdu2.redhat.com -U admin -E sol deactivate
Info: SOL payload already de-activated

[root@kvm-06-guest16 ~]# ipmitool -I lanplus -H ampere-mtsnow-altra-04-bmc.mgmt.lab.eng.rdu2.redhat.com -U admin -E sol activate
[SOL Session operational.  Use ~? for help]
Red Hat Enterprise Linux 8.4 (Ootpa)
Kernel 4.18.0-305.el8.aarch64 on an aarch64

ampere-mtsnow-altra-04 login:

Comment 19 errata-xmlrpc 2021-11-09 18:50:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ipmitool bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4307


Note You need to log in before you can comment on or make changes to this bug.