Bug 546386

Summary: ipmitool tsol dies with backtrace
Product: Red Hat Enterprise Linux 5 Reporter: Shad L. Lords <slords>
Component: OpenIPMIAssignee: Jan Safranek <jsafrane>
Status: CLOSED ERRATA QA Contact: qe-baseos-daemons
Severity: medium Docs Contact:
Priority: low    
Version: 5.4CC: azelinka, kvolny, rvokal
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
URL:
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
If the user called 'ipmitool tsol' on a serial console over LAN on Tyan hardware, the application terminated unexpectedly. This happened due to a mistake in serial-over-lan on Tyan hardware. Now this issue is fixed
 Story Points: ---
Clone Of: Environment:
Last Closed: 2011-01-13 23:30:33 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Easier to read backtrace
none
Packet capture from a el4 box that was successful
none
Packet capture from a el5 box that failed
none
full backtrace after crash
none
reproducer none

Description Shad L. Lords 2009-12-10 19:12:22 UTC 
Description of problem:

When trying to connect to Tyan IPMI v1.5 sol the application crashes with a buffer overflow.

Version-Release number of selected component (if applicable):

OpenIPMI-2.0.16-5.el5_4.1

How reproducible:

Always

Steps to Reproduce:
1. ipmitool -I lan -H <ip> -P <pass> tsol
2. receive any data from remote console (press <ENTER> in my case)
  
Actual results:

[Starting SOL with receiving address <ip>:6230]
[SOL Session operational.  Use ~? for help]
*** buffer overflow detected ***: ipmitool terminated
======= Backtrace: =========
/lib64/libc.so.6(__chk_fail+0x2f)[0x38a08e77af]
/lib64/libc.so.6[0x38a08e7d18]
ipmitool[0x43475d]
ipmitool[0x433a20]
ipmitool[0x404500]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x38a081d994]
ipmitool[0x404439]
======= Memory map: ========
00400000-00476000 r-xp 00000000 fd:00 272957                             /usr/bin/ipmitool
00676000-0068f000 rw-p 00076000 fd:00 272957                             /usr/bin/ipmitool
0068f000-00712000 rw-p 0068f000 00:00 0
0088e000-00893000 rw-p 0008e000 fd:00 272957                             /usr/bin/ipmitool
19a7c000-19a9d000 rw-p 19a7c000 00:00 0
38a0400000-38a041c000 r-xp 00000000 fd:00 360465                         /lib64/ld-2.5.so
38a061b000-38a061c000 r--p 0001b000 fd:00 360465                         /lib64/ld-2.5.so
38a061c000-38a061d000 rw-p 0001c000 fd:00 360465                         /lib64/ld-2.5.so
38a0800000-38a094d000 r-xp 00000000 fd:00 360466                         /lib64/libc-2.5.so
38a094d000-38a0b4d000 ---p 0014d000 fd:00 360466                         /lib64/libc-2.5.so
38a0b4d000-38a0b51000 r--p 0014d000 fd:00 360466                         /lib64/libc-2.5.so
38a0b51000-38a0b52000 rw-p 00151000 fd:00 360466                         /lib64/libc-2.5.so
38a0b52000-38a0b57000 rw-p 38a0b52000 00:00 0
38a0c00000-38a0c02000 r-xp 00000000 fd:00 363436                         /lib64/libdl-2.5.so
38a0c02000-38a0e02000 ---p 00002000 fd:00 363436                         /lib64/libdl-2.5.so
38a0e02000-38a0e03000 r--p 00002000 fd:00 363436                         /lib64/libdl-2.5.so
38a0e03000-38a0e04000 rw-p 00003000 fd:00 363436                         /lib64/libdl-2.5.so
38a2000000-38a2082000 r-xp 00000000 fd:00 363461                         /lib64/libm-2.5.so
38a2082000-38a2281000 ---p 00082000 fd:00 363461                         /lib64/libm-2.5.so
38a2281000-38a2282000 r--p 00081000 fd:00 363461                         /lib64/libm-2.5.so
38a2282000-38a2283000 rw-p 00082000 fd:00 363461                         /lib64/libm-2.5.so
38a2400000-38a2414000 r-xp 00000000 fd:00 267717                         /usr/lib64/libz.so.1.2.3
38a2414000-38a2613000 ---p 00014000 fd:00 267717                         /usr/lib64/libz.so.1.2.3
38a2613000-38a2614000 rw-p 00013000 fd:00 267717                         /usr/lib64/libz.so.1.2.3
38a2800000-38a280d000 r-xp 00000000 fd:00 363462                         /lib64/libgcc_s-4.1.2-20080825.so.1
38a280d000-38a2a0d000 ---p 0000d000 fd:00 363462                         /lib64/libgcc_s-4.1.2-20080825.so.1
38a2a0d000-38a2a0e000 rw-p 0000d000 fd:00 363462                         /lib64/libgcc_s-4.1.2-20080825.so.1
38a2c00000-38a2c4e000 r-xp 00000000 fd:00 271011                         /usr/lib64/libncurses.so.5.5
38a2c4e000-38a2e4e000 ---p 0004e000 fd:00 271011                         /usr/lib64/libncurses.so.5.5
38a2e4e000-38a2e5c000 rw-p 0004e000 fd:00 271011                         /usr/lib64/libncurses.so.5.5
38a2e5c000-38a2e5d000 rw-p 38a2e5c000 00:00 0
38a3000000-38a3035000 r-xp 00000000 fd:00 267752                         /usr/lib64/libreadline.so.5.1
38a3035000-38a3234000 ---p 00035000 fd:00 267752                         /usr/lib64/libreadline.so.5.1
38a3234000-38a323c000 rw-p 00034000 fd:00 267752                         /usr/lib64/libreadline.so.5.1
38a323c000-38a323d000 rw-p 38a323c000 00:00 0
38a3800000-38a392d000 r-xp 00000000 fd:00 363467                         /lib64/libcrypto.so.0.9.8e
38a392d000-38a3b2c000 ---p 0012d000 fd:00 363467                         /lib64/libcrypto.so.0.9.8e
38a3b2c000-38a3b4d000 rw-p 0012c000 fd:00 363467                         /lib64/libcrypto.so.0.9.8e
38a3b4d000-38a3b51000 rw-p 38a3b4d000 00:00 0
2b7f01ff3000-2b7f01ff5000 rw-p 2b7f01ff3000 00:00 0
2b7f01ff9000-2b7f01ffd000 rw-p 2b7f01ff9000 00:00 0
7fffbc6b7000-7fffbc6cc000 rw-p 7ffffffea000 00:00 0                      [stack]
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0                  [vdsAborted

Expected results:

remote console data is received and displayed.

Additional info:

This appears to be similar to Bug 316181 which was closed due to Fedora 9 EOL.
Comment 1 Shad L. Lords 2009-12-10 19:35:50 UTC 
Created attachment 377571 [details]
Easier to read backtrace
Comment 2 Shad L. Lords 2009-12-10 20:04:40 UTC 
On a side note this works fine on a EL4 box with OpenIPMI-1.4.14-1.4E.25 installed.
Comment 3 Jan Safranek 2010-01-04 15:37:24 UTC 
The stack trace is not much useful without debuginfos installed... please install them! See http://kbase.redhat.com/faq/docs/DOC-9908 and install following:

$ yum --enablerepo rhel-debuginfo install glibc-debuginfo OpenIPMI-debuginfo

Now try to reproduce the bug with full core dump:
$ ulimit -c unlimited
$ ipmitool -I lan -H <ip> -P <pass> tsol
... it should dump a core file name core.<PID>

Finally, analyze the core file and send me output of following:
$ gdb /usr/bin/ipmitool <the core file>
(gdb) bt full

Thanks in advance!

And out of curiosity, what HW do you use? Maybe I can find some internally...
Comment 4 Shad L. Lords 2010-01-04 17:08:21 UTC 
I've tried what you suggested but there is no core file created.  The output with the debug packages also doesn't look any different (other than memory addresses).

The hardware that I'm running is a Tyan S2881 Thunder K8SR motherboard with 2 x Opteron 280 processors and 16GB of ram.  The motherboard has a Tyan M3289 IPMI card attached.

The crash occurs when trying to go from one of these boxes to another one.  I've also confirmed that the crash also occurs coming from a Compaq ML115 G1 so it doesn't appear to be dependent on where it is coming from.

Data on the wire appears to be identical up to the point that it crashes.  I'll attach packet captures in case it helps diagnose it at all.
Comment 5 Shad L. Lords 2010-01-04 17:09:15 UTC 
Created attachment 381591 [details]
Packet capture from a el4 box that was successful
Comment 6 Shad L. Lords 2010-01-04 17:09:34 UTC 
Created attachment 381592 [details]
Packet capture from a el5 box that failed
Comment 7 Jan Safranek 2010-01-20 09:18:22 UTC 
I am still not able to reproduce the bug - I've created dummy IPMI server which sends exact packets from your traces, but ipmitool refuses to crash. Without useful stack trace (i.e. with debug symbols) I can't locate the error.

So, as last resort, try to start ipmitool under debugger:
$ gdb /usr/bin/ipmitool

(gdb) r -I lan -H localhost -P pass tsol

After it crashes, send me output of 'bt full' gdb command - it shows complete stack trace, assuming you have correct debuginfo packages installed.

If you have troubles installing debuginfo, please contact your Red Hat support at redhat.com/support - that's the service you pay for.
Comment 8 Shad L. Lords 2010-01-20 15:10:03 UTC 
Created attachment 385698 [details]
full backtrace after crash

Here is the output of the crash.  This time it gives what appears to be much more useful information.
Comment 9 Jan Safranek 2010-01-21 10:30:08 UTC 
Created attachment 385896 [details]
reproducer

Reproducer usage:

$ perl tsol.pl &
$ ipmitool -I lan -H localhost -P pass tsol
-> ipmtiool crashes spectacularly
Comment 10 Jan Safranek 2010-01-21 10:36:25 UTC 
Aehm, the wireshark logs contain "CentOS release 5.4 (Final)" console output... are you sure you do have valid RHEL subscription?

Anyway, there really is a bug in our ipmitool so why not to fix it. Please remember to go through redhat.com/support next time, thanks!
Comment 11 Shad L. Lords 2010-01-21 13:51:20 UTC 
No I do not have a valid RHEL subscription.  However I do believe that reporting bugs that exist in your release helps everyone out and makes for a better product.
Comment 12 Jan Safranek 2010-01-21 14:55:12 UTC 
> No I do not have a valid RHEL subscription.  However I do believe that
> reporting bugs that exist in your release helps everyone out and makes for a
> better product.    
If you believe so, please report the bugs upstream directly next time...

I've just sent patches upstream:
http://sourceforge.net/mailarchive/forum.php?thread_name=20100121121244.28842.26348.stgit%40dhcp-lab-211.englab.brq.redhat.com&forum_name=ipmitool-devel
Comment 13 Shad L. Lords 2010-01-21 15:01:31 UTC 
Thanks for the information.  If it is fixed upstream will it still be pulled/incorporated into redhat?
Comment 14 Jan Safranek 2010-01-21 15:13:41 UTC 
(In reply to comment #13)
> Thanks for the information.  If it is fixed upstream will it still be
> pulled/incorporated into redhat?    

Dunno, not all packages are updated in every release. It depends if there is a request from support, i.e. from paying customers ;-).
Comment 15 Shad L. Lords 2010-01-26 16:37:23 UTC 
I can confirm that the patch fixes the issue for me.
Comment 19 Eva Kopalova 2010-12-20 12:03:54 UTC 
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
If the user called 'ipmitool tsol' on a serial console over LAN on Tyan hardware, the application terminated unexpectedly. This happened due to a mistake in serial-over-lan on Tyan hardware. Now this issue is fixed
Comment 21 errata-xmlrpc 2011-01-13 23:30:33 UTC 
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0096.html