Bug 1259523 - "*** buffer overflow detected ***: multipath terminated" when logged to many iSCSI LUNs
"*** buffer overflow detected ***: multipath terminated" when logged to many ...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: device-mapper-multipath (Show other bugs)
7.2
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Ben Marzinski
Martin Hoyer
: Regression
: 1261561 (view as bug list)
Depends On:
Blocks: 1187263
  Show dependency treegraph
 
Reported: 2015-09-02 17:30 EDT by Martin Hoyer
Modified: 2015-11-19 07:57 EST (History)
8 users (show)

See Also:
Fixed In Version: device-mapper-multipath-0.4.9-83.el7
Doc Type: Bug Fix
Doc Text:
Cause: Multipath wasn't allocating enough space to store the string for the /sys/class/scsi_host/host<X> pathname if X was larger than 3 digits Consequence: Multipath would crash with memory corruption if there were more than 1000 scsi hosts. This can happen if there are too many iscsi devices. Fix: Multipath now allocates enough space to handle the pathname at any possible size. Result: Multipath will no longer crash with memory corruption if there are two many iscsi devices that it is trying to use.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-11-19 07:57:54 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
dmesg log (732.12 KB, text/plain)
2015-09-02 17:30 EDT, Martin Hoyer
no flags Details

  None (edit)
Description Martin Hoyer 2015-09-02 17:30:52 EDT
Created attachment 1069598 [details]
dmesg log

Description of problem:
During our test for checking send targets discovery many iSCSI LUNs, we hit a issue, when multipath gets terminated after logging in to many iSCSI devices.

There is 2048 iSCSI devices

After running multipath:

*** buffer overflow detected ***: multipath terminated
======= Backtrace: =========
/lib64/libc.so.6(__fortify_fail+0x37)[0x7f1befc46a77]
/lib64/libc.so.6(+0x10bc30)[0x7f1befc44c30]
/lib64/libc.so.6(+0x10b139)[0x7f1befc44139]
/lib64/libc.so.6(_IO_default_xsputn+0xbc)[0x7f1befbb1a1c]
/lib64/libc.so.6(_IO_vfprintf+0xb0d)[0x7f1befb8105d]
/lib64/libc.so.6(__vsprintf_chk+0x88)[0x7f1befc441c8]
/lib64/libc.so.6(__sprintf_chk+0x7d)[0x7f1befc4411d]
/lib64/libmultipath.so.0(sysfs_get_iscsi_ip_address+0x40)[0x7f1beff137f0]
/lib64/libmultipath.so.0(+0x2dc69)[0x7f1beff27c69]
/lib64/libmultipath.so.0(get_path_layout+0x8c)[0x7f1beff27ebc]
multipath[0x402804]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f1befb5ab15]
multipath[0x402cd1]
======= Memory map: ========
00400000-00405000 r-xp 00000000 fd:00 203854686                          /usr/sbin/multipath
00604000-00605000 r--p 00004000 fd:00 203854686                          /usr/sbin/multipath
00605000-00606000 rw-p 00005000 fd:00 203854686                          /usr/sbin/multipath
00627000-025cf000 rw-p 00000000 00:00 0                                  [heap]
7f1bed48e000-7f1bed490000 r-xp 00000000 fd:00 817763                     /usr/lib64/multipath/libprioontap.so
7f1bed490000-7f1bed68f000 ---p 00002000 fd:00 817763                     /usr/lib64/multipath/libprioontap.so
7f1bed68f000-7f1bed690000 r--p 00001000 fd:00 817763                     /usr/lib64/multipath/libprioontap.so
7f1bed690000-7f1bed691000 rw-p 00002000 fd:00 817763                     /usr/lib64/multipath/libprioontap.so
7f1bed691000-7f1bed694000 r-xp 00000000 fd:00 817147                     /usr/lib64/multipath/libchecktur.so
7f1bed694000-7f1bed893000 ---p 00003000 fd:00 817147                     /usr/lib64/multipath/libchecktur.so
7f1bed893000-7f1bed894000 r--p 00002000 fd:00 817147                     /usr/lib64/multipath/libchecktur.so
7f1bed894000-7f1bed895000 rw-p 00003000 fd:00 817147                     /usr/lib64/multipath/libchecktur.so
7f1bed895000-7f1bed896000 r-xp 00000000 fd:00 817149                     /usr/lib64/multipath/libprioconst.so
7f1bed896000-7f1beda95000 ---p 00001000 fd:00 817149                     /usr/lib64/multipath/libprioconst.so
7f1beda95000-7f1beda96000 r--p 00000000 fd:00 817149                     /usr/lib64/multipath/libprioconst.so
7f1beda96000-7f1beda97000 rw-p 00001000 fd:00 817149                     /usr/lib64/multipath/libprioconst.so
7f1beda97000-7f1beda98000 r-xp 00000000 fd:00 202270762                  /usr/lib64/libaio.so.1.0.1
7f1beda98000-7f1bedc97000 ---p 00001000 fd:00 202270762                  /usr/lib64/libaio.so.1.0.1
7f1bedc97000-7f1bedc98000 r--p 00000000 fd:00 202270762                  /usr/lib64/libaio.so.1.0.1
7f1bedc98000-7f1bedc99000 rw-p 00001000 fd:00 202270762                  /usr/lib64/libaio.so.1.0.1
7f1bedc99000-7f1bedc9b000 r-xp 00000000 fd:00 817141                     /usr/lib64/multipath/libcheckdirectio.so
7f1bedc9b000-7f1bede9a000 ---p 00002000 fd:00 817141                     /usr/lib64/multipath/libcheckdirectio.so
7f1bede9a000-7f1bede9b000 r--p 00001000 fd:00 817141                     /usr/lib64/multipath/libcheckdirectio.so
7f1bede9b000-7f1bede9c000 rw-p 00002000 fd:00 817141                     /usr/lib64/multipath/libcheckdirectio.so
7f1bede9c000-7f1bedeb1000 r-xp 00000000 fd:00 201329780                  /usr/lib64/libz.so.1.2.7
7f1bedeb1000-7f1bee0b0000 ---p 00015000 fd:00 201329780                  /usr/lib64/libz.so.1.2.7
7f1bee0b0000-7f1bee0b1000 r--p 00014000 fd:00 201329780                  /usr/lib64/libz.so.1.2.7
7f1bee0b1000-7f1bee0b2000 rw-p 00015000 fd:00 201329780                  /usr/lib64/libz.so.1.2.7
7f1bee0b2000-7f1bee0c1000 r-xp 00000000 fd:00 201329821                  /usr/lib64/libbz2.so.1.0.6
7f1bee0c1000-7f1bee2c0000 ---p 0000f000 fd:00 201329821                  /usr/lib64/libbz2.so.1.0.6
7f1bee2c0000-7f1bee2c1000 r--p 0000e000 fd:00 201329821                  /usr/lib64/libbz2.so.1.0.6
7f1bee2c1000-7f1bee2c2000 rw-p 0000f000 fd:00 201329821                  /usr/lib64/libbz2.so.1.0.6
7f1bee2c2000-7f1bee2d7000 r-xp 00000000 fd:00 201329809                  /usr/lib64/libelf-0.163.so
7f1bee2d7000-7f1bee4d6000 ---p 00015000 fd:00 201329809                  /usr/lib64/libelf-0.163.so
7f1bee4d6000-7f1bee4d7000 r--p 00014000 fd:00 201329809                  /usr/lib64/libelf-0.163.so
7f1bee4d7000-7f1bee4d8000 rw-p 00015000 fd:00 201329809                  /usr/lib64/libelf-0.163.so
7f1bee4d8000-7f1bee4dc000 r-xp 00000000 fd:00 201329974                  /usr/lib64/libattr.so.1.1.0
7f1bee4dc000-7f1bee6db000 ---p 00004000 fd:00 201329974                  /usr/lib64/libattr.so.1.1.0
7f1bee6db000-7f1bee6dc000 r--p 00003000 fd:00 201329974                  /usr/lib64/libattr.so.1.1.0
7f1bee6dc000-7f1bee6dd000 rw-p 00004000 fd:00 201329974                  /usr/lib64/libattr.so.1.1.0
7f1bee6dd000-7f1bee701000 r-xp 00000000 fd:00 201329694                  /usr/lib64/liblzma.so.5.0.99
7f1bee701000-7f1bee900000 ---p 00024000 fd:00 201329694                  /usr/lib64/liblzma.so.5.0.99
7f1bee900000-7f1bee901000 r--p 00023000 fd:00 201329694                  /usr/lib64/liblzma.so.5.0.99
7f1bee901000-7f1bee902000 rw-p 00024000 fd:00 201329694                  /usr/lib64/liblzma.so.5.0.99
7f1bee902000-7f1bee962000 r-xp 00000000 fd:00 201329761                  /usr/lib64/libpcre.so.1.2.0
7f1bee962000-7f1beeb61000 ---p 00060000 fd:00 201329761                  /usr/lib64/libpcre.so.1.2.0
7f1beeb61000-7f1beeb62000 r--p 0005f000 fd:00 201329761                  /usr/lib64/libpcre.so.1.2.0
7f1beeb62000-7f1beeb63000 rw-p 00060000 fd:00 201329761                  /usr/lib64/libpcre.so.1.2.0
7f1beeb63000-7f1beeba8000 r-xp 00000000 fd:00 201330052                  /usr/lib64/libdw-0.163.so
7f1beeba8000-7f1beeda7000 ---p 00045000 fd:00 201330052                  /usr/lib64/libdw-0.163.so
7f1beeda7000-7f1beeda9000 r--p 00044000 fd:00 201330052                  /usr/lib64/libdw-0.163.so
7f1beeda9000-7f1beedaa000 rw-p 00046000 fd:00 201330052                  /usr/lib64/libdw-0.163.so
7f1beedaa000-7f1beeeab000 r-xp 00000000 fd:00 201329596                  /usr/lib64/libm-2.17.so
7f1beeeab000-7f1bef0aa000 ---p 00101000 fd:00 201329596                  /usr/lib64/libm-2.17.so
7f1bef0aa000-7f1bef0ab000 r--p 00100000 fd:00 201329596                  /usr/lib64/libm-2.17.so
7f1bef0ab000-7f1bef0ac000 rw-p 00101000 fd:00 201329596                  /usr/lib64/libm-2.17.so
7f1bef0ac000-7f1bef0b0000 r-xp 00000000 fd:00 201329976                  /usr/lib64/libcap.so.2.22
7f1bef0b0000-7f1bef2af000 ---p 00004000 fd:00 201329976                  /usr/lib64/libcap.so.2.22
7f1bef2af000-7f1bef2b0000 r--p 00003000 fd:00 201329976                  /usr/lib64/libcap.so.2.22
7f1bef2b0000-7f1bef2b1000 rw-p 00004000 fd:00 201329976                  /usr/lib64/libcap.so.2.22
7f1bef2b1000-7f1bef2c6000 r-xp 00000000 fd:00 201326732                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f1bef2c6000-7f1bef4c5000 ---p 00015000 fd:00 201326732                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f1bef4c5000-7f1bef4c6000 r--p 00014000 fd:00 201326732                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f1bef4c6000-7f1bef4c7000 rw-p 00015000 fd:00 201326732                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f1bef4c7000-7f1bef50b000 r-xp 00000000 fd:00 201329695                  /usr/lib64/libsepol.so.1
7f1bef50b000-7f1bef70a000 ---p 00044000 fd:00 201329695                  /usr/lib64/libsepol.so.1
7f1bef70a000-7f1bef70b000 r--p 00043000 fd:00 201329695                  /usr/lib64/libsepol.so.1
7f1bef70b000-7f1bef70c000 rw-p 00044000 fd:00 201329695                  /usr/lib64/libsepol.so.1
7f1bef70c000-7f1bef72d000 r-xp 00000000 fd:00 201329777                  /usr/lib64/libselinux.so.1
7f1bef72d000-7f1bef92d000 ---p 00021000 fd:00 201329777                  /usr/lib64/libselinux.so.1
7f1bef92d000-7f1bef92e000 r--p 00021000 fd:00 201329777                  /usr/lib64/libselinux.so.1
7f1bef92e000-7f1bef92f000 rw-p 00022000 fd:00 201329777                  /usr/lib64/libselinux.so.1
7f1bef92f000-7f1bef931000 rw-p 00000000 00:00 0 
7f1bef931000-7f1bef938000 r-xp 00000000 fd:00 201329618                  /usr/lib64/librt-2.17.so
7f1bef938000-7f1befb37000 ---p 00007000 fd:00 201329618                  /usr/lib64/librt-2.17.so
7f1befb37000-7f1befb38000 r--p 00006000 fd:00 201329618                  /usr/lib64/librt-2.17.so
7f1befb38000-7f1befb39000 rw-p 00007000 fd:00 201329618                  /usr/lib64/librt-2.17.so
7f1befb39000-7f1befcef000 r-xp 00000000 fd:00 201329588                  /usr/lib64/libc-2.17.so
7f1befcef000-7f1befeef000 ---p 001b6000 fd:00 201329588                  /usr/lib64/libc-2.17.so
7f1befeef000-7f1befef3000 r--p 001b6000 fd:00 201329588                  /usr/lib64/libc-2.17.so
7f1befef3000-7f1befef5000 rw-p 001ba000 fd:00 201329588                  /usr/lib64/libc-2.17.so
7f1befef5000-7f1befefa000 rw-p 00000000 00:00 0 
7f1befefa000-7f1beff42000 r-xp 00000000 fd:00 203854683                  /usr/lib64/libmultipath.so.0
7f1beff42000-7f1bf0142000 ---p 00048000 fd:00 203854683                  /usr/lib64/libmultipath.so.0
7f1bf0142000-7f1bf0143000 r--p 00048000 fd:00 203854683                  /usr/lib64/libmultipath.so.0
7f1bf0143000-7f1bf0148000 rw-p 00049000 fd:00 203854683                  /usr/lib64/libmultipath.so.0
7f1bf0148000-7f1bf0149000 rw-p 00000000 00:00 0 
7f1bf0149000-7f1bf014c000 r-xp 00000000 fd:00 201329594                  /usr/lib64/libdl-2.17.so
7f1bf014c000-7f1bf034b000 ---p 00003000 fd:00 201329594                  /usr/lib64/libdl-2.17.so
7f1bf034b000-7f1bf034c000 r--p 00002000 fd:00 201329594                  /usr/lib64/libdl-2.17.so
7f1bf034c000-7f1bf034d000 rw-p 00003000 fd:00 201329594                  /usr/lib64/libdl-2.17.so
7f1bf034d000-7f1bf039e000 r-xp 00000000 fd:00 202296243                  /usr/lib64/libdevmapper.so.1.02
7f1bf039e000-7f1bf059e000 ---p 00051000 fd:00 202296243                  /usr/lib64/libdevmapper.so.1.02
7f1bf059e000-7f1bf059f000 r--p 00051000 fd:00 202296243                  /usr/lib64/libdevmapper.so.1.02
7f1bf059f000-7f1bf05a2000 rw-p 00052000 fd:00 202296243                  /usr/lib64/libdevmapper.so.1.02
7f1bf05a2000-7f1bf05a4000 rw-p 00000000 00:00 0 
7f1bf05a4000-7f1bf05ba000 r-xp 00000000 fd:00 201329614                  /usr/lib64/libpthread-2.17.so
7f1bf05ba000-7f1bf07ba000 ---p 00016000 fd:00 201329614                  /usr/lib64/libpthread-2.17.so
7f1bf07ba000-7f1bf07bb000 r--p 00016000 fd:00 201329614                  /usr/lib64/libpthread-2.17.so
7f1bf07bb000-7f1bf07bc000 rw-p 00017000 fd:00 201329614                  /usr/lib64/libpthread-2.17.so
7f1bf07bc000-7f1bf07c0000 rw-p 00000000 00:00 0 
7f1bf07c0000-7f1bf07e1000 r-xp 00000000 fd:00 201329581                  /usr/lib64/ld-2.17.so
7f1bf09b7000-7f1bf09c0000 rw-p 00000000 00:00 0 
7f1bf09c0000-7f1bf09d2000 r-xp 00000000 fd:00 202181518                  /usr/lib64/libudev.so.1.6.2
7f1bf09d2000-7f1bf09d3000 ---p 00012000 fd:00 202181518                  /usr/lib64/libudev.so.1.6.2
7f1bf09d3000-7f1bf09d4000 r--p 00012000 fd:00 202181518                  /usr/lib64/libudev.so.1.6.2
7f1bf09d4000-7f1bf09d5000 rw-p 00013000 fd:00 202181518                  /usr/lib64/libudev.so.1.6.2
7f1bf09d5000-7f1bf09d6000 rw-p 00000000 00:00 0 
7f1bf09df000-7f1bf09e1000 rw-p 00000000 00:00 0 
7f1bf09e1000-7f1bf09e2000 r--p 00021000 fd:00 201329581                  /usr/lib64/ld-2.17.so
7f1bf09e2000-7f1bf09e3000 rw-p 00022000 fd:00 201329581                  /usr/lib64/ld-2.17.so
7f1bf09e3000-7f1bf09e4000 rw-p 00000000 00:00 0 
7ffe22bf6000-7ffe22c17000 rw-p 00000000 00:00 0                          [stack]
7ffe22d3f000-7ffe22d41000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Return:0


This issue was not seen on RHEL-7.1 or RHEL-7.2 alpha(device-mapper-multipath-0.4.9-77.el7.x86_64)


Version-Release number of selected component (if applicable):
device-mapper-multipath-0.4.9-82.el7.x86_64
kernel-3.10.0-306.0.1.el7
iscsi-initiator-utils-6.2.0.873-32.el7

How reproducible:
needs further testing

Steps to Reproduce:
1.Establish iSCSI session with many LUNs
2.Run multipath on them
3.see '*** buffer overflow detected ***: multipath terminated' message

Actual results:
multipath terminated

Expected results:
created multipath devices

Additional info:
see call trace in dmesg log
Comment 7 David Bulkow 2015-09-10 10:08:31 EDT
*** Bug 1261561 has been marked as a duplicate of this bug. ***
Comment 8 Martin Hoyer 2015-09-15 04:52:41 EDT
Working well with device-mapper-multipath-0.4.9-83.el7
Comment 9 errata-xmlrpc 2015-11-19 07:57:54 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2132.html

Note You need to log in before you can comment on or make changes to this bug.