1259523 – "*** buffer overflow detected ***: multipath terminated" when logged to many iSCSI LUNs

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1259523 - "*** buffer overflow detected ***: multipath terminated" when logged to many iSCSI LUNs

Summary: "*** buffer overflow detected ***: multipath terminated" when logged to many ...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	device-mapper-multipath
Sub Component:
Version:	7.2
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	---
Assignee:	Ben Marzinski
QA Contact:	Martin Hoyer
Docs Contact:
URL:
Whiteboard:
Duplicates (1):	1261561 (view as bug list)
Depends On:
Blocks:	1187263
TreeView+	depends on / blocked

Reported:	2015-09-02 21:30 UTC by Martin Hoyer
Modified:	2021-09-03 11:57 UTC (History)
CC List:	8 users (show)
Fixed In Version:	device-mapper-multipath-0.4.9-83.el7
Doc Type:	Bug Fix
Doc Text:	Cause: Multipath wasn't allocating enough space to store the string for the /sys/class/scsi_host/host<X> pathname if X was larger than 3 digits Consequence: Multipath would crash with memory corruption if there were more than 1000 scsi hosts. This can happen if there are too many iscsi devices. Fix: Multipath now allocates enough space to handle the pathname at any possible size. Result: Multipath will no longer crash with memory corruption if there are two many iscsi devices that it is trying to use.
Clone Of:
Environment:
Last Closed:	2015-11-19 12:57:54 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
dmesg log (732.12 KB, text/plain) 2015-09-02 21:30 UTC, Martin Hoyer	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2015:2132	0	normal	SHIPPED_LIVE	device-mapper-multipath bug fix and enhancement update	2015-11-19 11:21:43 UTC

Description Martin Hoyer 2015-09-02 21:30:52 UTC

Created attachment 1069598 [details]
dmesg log

Description of problem:
During our test for checking send targets discovery many iSCSI LUNs, we hit a issue, when multipath gets terminated after logging in to many iSCSI devices.

There is 2048 iSCSI devices

After running multipath:

*** buffer overflow detected ***: multipath terminated
======= Backtrace: =========
/lib64/libc.so.6(__fortify_fail+0x37)[0x7f1befc46a77]
/lib64/libc.so.6(+0x10bc30)[0x7f1befc44c30]
/lib64/libc.so.6(+0x10b139)[0x7f1befc44139]
/lib64/libc.so.6(_IO_default_xsputn+0xbc)[0x7f1befbb1a1c]
/lib64/libc.so.6(_IO_vfprintf+0xb0d)[0x7f1befb8105d]
/lib64/libc.so.6(__vsprintf_chk+0x88)[0x7f1befc441c8]
/lib64/libc.so.6(__sprintf_chk+0x7d)[0x7f1befc4411d]
/lib64/libmultipath.so.0(sysfs_get_iscsi_ip_address+0x40)[0x7f1beff137f0]
/lib64/libmultipath.so.0(+0x2dc69)[0x7f1beff27c69]
/lib64/libmultipath.so.0(get_path_layout+0x8c)[0x7f1beff27ebc]
multipath[0x402804]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f1befb5ab15]
multipath[0x402cd1]
======= Memory map: ========
00400000-00405000 r-xp 00000000 fd:00 203854686                          /usr/sbin/multipath
00604000-00605000 r--p 00004000 fd:00 203854686                          /usr/sbin/multipath
00605000-00606000 rw-p 00005000 fd:00 203854686                          /usr/sbin/multipath
00627000-025cf000 rw-p 00000000 00:00 0                                  [heap]
7f1bed48e000-7f1bed490000 r-xp 00000000 fd:00 817763                     /usr/lib64/multipath/libprioontap.so
7f1bed490000-7f1bed68f000 ---p 00002000 fd:00 817763                     /usr/lib64/multipath/libprioontap.so
7f1bed68f000-7f1bed690000 r--p 00001000 fd:00 817763                     /usr/lib64/multipath/libprioontap.so
7f1bed690000-7f1bed691000 rw-p 00002000 fd:00 817763                     /usr/lib64/multipath/libprioontap.so
7f1bed691000-7f1bed694000 r-xp 00000000 fd:00 817147                     /usr/lib64/multipath/libchecktur.so
7f1bed694000-7f1bed893000 ---p 00003000 fd:00 817147                     /usr/lib64/multipath/libchecktur.so
7f1bed893000-7f1bed894000 r--p 00002000 fd:00 817147                     /usr/lib64/multipath/libchecktur.so
7f1bed894000-7f1bed895000 rw-p 00003000 fd:00 817147                     /usr/lib64/multipath/libchecktur.so
7f1bed895000-7f1bed896000 r-xp 00000000 fd:00 817149                     /usr/lib64/multipath/libprioconst.so
7f1bed896000-7f1beda95000 ---p 00001000 fd:00 817149                     /usr/lib64/multipath/libprioconst.so
7f1beda95000-7f1beda96000 r--p 00000000 fd:00 817149                     /usr/lib64/multipath/libprioconst.so
7f1beda96000-7f1beda97000 rw-p 00001000 fd:00 817149                     /usr/lib64/multipath/libprioconst.so
7f1beda97000-7f1beda98000 r-xp 00000000 fd:00 202270762                  /usr/lib64/libaio.so.1.0.1
7f1beda98000-7f1bedc97000 ---p 00001000 fd:00 202270762                  /usr/lib64/libaio.so.1.0.1
7f1bedc97000-7f1bedc98000 r--p 00000000 fd:00 202270762                  /usr/lib64/libaio.so.1.0.1
7f1bedc98000-7f1bedc99000 rw-p 00001000 fd:00 202270762                  /usr/lib64/libaio.so.1.0.1
7f1bedc99000-7f1bedc9b000 r-xp 00000000 fd:00 817141                     /usr/lib64/multipath/libcheckdirectio.so
7f1bedc9b000-7f1bede9a000 ---p 00002000 fd:00 817141                     /usr/lib64/multipath/libcheckdirectio.so
7f1bede9a000-7f1bede9b000 r--p 00001000 fd:00 817141                     /usr/lib64/multipath/libcheckdirectio.so
7f1bede9b000-7f1bede9c000 rw-p 00002000 fd:00 817141                     /usr/lib64/multipath/libcheckdirectio.so
7f1bede9c000-7f1bedeb1000 r-xp 00000000 fd:00 201329780                  /usr/lib64/libz.so.1.2.7
7f1bedeb1000-7f1bee0b0000 ---p 00015000 fd:00 201329780                  /usr/lib64/libz.so.1.2.7
7f1bee0b0000-7f1bee0b1000 r--p 00014000 fd:00 201329780                  /usr/lib64/libz.so.1.2.7
7f1bee0b1000-7f1bee0b2000 rw-p 00015000 fd:00 201329780                  /usr/lib64/libz.so.1.2.7
7f1bee0b2000-7f1bee0c1000 r-xp 00000000 fd:00 201329821                  /usr/lib64/libbz2.so.1.0.6
7f1bee0c1000-7f1bee2c0000 ---p 0000f000 fd:00 201329821                  /usr/lib64/libbz2.so.1.0.6
7f1bee2c0000-7f1bee2c1000 r--p 0000e000 fd:00 201329821                  /usr/lib64/libbz2.so.1.0.6
7f1bee2c1000-7f1bee2c2000 rw-p 0000f000 fd:00 201329821                  /usr/lib64/libbz2.so.1.0.6
7f1bee2c2000-7f1bee2d7000 r-xp 00000000 fd:00 201329809                  /usr/lib64/libelf-0.163.so
7f1bee2d7000-7f1bee4d6000 ---p 00015000 fd:00 201329809                  /usr/lib64/libelf-0.163.so
7f1bee4d6000-7f1bee4d7000 r--p 00014000 fd:00 201329809                  /usr/lib64/libelf-0.163.so
7f1bee4d7000-7f1bee4d8000 rw-p 00015000 fd:00 201329809                  /usr/lib64/libelf-0.163.so
7f1bee4d8000-7f1bee4dc000 r-xp 00000000 fd:00 201329974                  /usr/lib64/libattr.so.1.1.0
7f1bee4dc000-7f1bee6db000 ---p 00004000 fd:00 201329974                  /usr/lib64/libattr.so.1.1.0
7f1bee6db000-7f1bee6dc000 r--p 00003000 fd:00 201329974                  /usr/lib64/libattr.so.1.1.0
7f1bee6dc000-7f1bee6dd000 rw-p 00004000 fd:00 201329974                  /usr/lib64/libattr.so.1.1.0
7f1bee6dd000-7f1bee701000 r-xp 00000000 fd:00 201329694                  /usr/lib64/liblzma.so.5.0.99
7f1bee701000-7f1bee900000 ---p 00024000 fd:00 201329694                  /usr/lib64/liblzma.so.5.0.99
7f1bee900000-7f1bee901000 r--p 00023000 fd:00 201329694                  /usr/lib64/liblzma.so.5.0.99
7f1bee901000-7f1bee902000 rw-p 00024000 fd:00 201329694                  /usr/lib64/liblzma.so.5.0.99
7f1bee902000-7f1bee962000 r-xp 00000000 fd:00 201329761                  /usr/lib64/libpcre.so.1.2.0
7f1bee962000-7f1beeb61000 ---p 00060000 fd:00 201329761                  /usr/lib64/libpcre.so.1.2.0
7f1beeb61000-7f1beeb62000 r--p 0005f000 fd:00 201329761                  /usr/lib64/libpcre.so.1.2.0
7f1beeb62000-7f1beeb63000 rw-p 00060000 fd:00 201329761                  /usr/lib64/libpcre.so.1.2.0
7f1beeb63000-7f1beeba8000 r-xp 00000000 fd:00 201330052                  /usr/lib64/libdw-0.163.so
7f1beeba8000-7f1beeda7000 ---p 00045000 fd:00 201330052                  /usr/lib64/libdw-0.163.so
7f1beeda7000-7f1beeda9000 r--p 00044000 fd:00 201330052                  /usr/lib64/libdw-0.163.so
7f1beeda9000-7f1beedaa000 rw-p 00046000 fd:00 201330052                  /usr/lib64/libdw-0.163.so
7f1beedaa000-7f1beeeab000 r-xp 00000000 fd:00 201329596                  /usr/lib64/libm-2.17.so
7f1beeeab000-7f1bef0aa000 ---p 00101000 fd:00 201329596                  /usr/lib64/libm-2.17.so
7f1bef0aa000-7f1bef0ab000 r--p 00100000 fd:00 201329596                  /usr/lib64/libm-2.17.so
7f1bef0ab000-7f1bef0ac000 rw-p 00101000 fd:00 201329596                  /usr/lib64/libm-2.17.so
7f1bef0ac000-7f1bef0b0000 r-xp 00000000 fd:00 201329976                  /usr/lib64/libcap.so.2.22
7f1bef0b0000-7f1bef2af000 ---p 00004000 fd:00 201329976                  /usr/lib64/libcap.so.2.22
7f1bef2af000-7f1bef2b0000 r--p 00003000 fd:00 201329976                  /usr/lib64/libcap.so.2.22
7f1bef2b0000-7f1bef2b1000 rw-p 00004000 fd:00 201329976                  /usr/lib64/libcap.so.2.22
7f1bef2b1000-7f1bef2c6000 r-xp 00000000 fd:00 201326732                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f1bef2c6000-7f1bef4c5000 ---p 00015000 fd:00 201326732                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f1bef4c5000-7f1bef4c6000 r--p 00014000 fd:00 201326732                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f1bef4c6000-7f1bef4c7000 rw-p 00015000 fd:00 201326732                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f1bef4c7000-7f1bef50b000 r-xp 00000000 fd:00 201329695                  /usr/lib64/libsepol.so.1
7f1bef50b000-7f1bef70a000 ---p 00044000 fd:00 201329695                  /usr/lib64/libsepol.so.1
7f1bef70a000-7f1bef70b000 r--p 00043000 fd:00 201329695                  /usr/lib64/libsepol.so.1
7f1bef70b000-7f1bef70c000 rw-p 00044000 fd:00 201329695                  /usr/lib64/libsepol.so.1
7f1bef70c000-7f1bef72d000 r-xp 00000000 fd:00 201329777                  /usr/lib64/libselinux.so.1
7f1bef72d000-7f1bef92d000 ---p 00021000 fd:00 201329777                  /usr/lib64/libselinux.so.1
7f1bef92d000-7f1bef92e000 r--p 00021000 fd:00 201329777                  /usr/lib64/libselinux.so.1
7f1bef92e000-7f1bef92f000 rw-p 00022000 fd:00 201329777                  /usr/lib64/libselinux.so.1
7f1bef92f000-7f1bef931000 rw-p 00000000 00:00 0 
7f1bef931000-7f1bef938000 r-xp 00000000 fd:00 201329618                  /usr/lib64/librt-2.17.so
7f1bef938000-7f1befb37000 ---p 00007000 fd:00 201329618                  /usr/lib64/librt-2.17.so
7f1befb37000-7f1befb38000 r--p 00006000 fd:00 201329618                  /usr/lib64/librt-2.17.so
7f1befb38000-7f1befb39000 rw-p 00007000 fd:00 201329618                  /usr/lib64/librt-2.17.so
7f1befb39000-7f1befcef000 r-xp 00000000 fd:00 201329588                  /usr/lib64/libc-2.17.so
7f1befcef000-7f1befeef000 ---p 001b6000 fd:00 201329588                  /usr/lib64/libc-2.17.so
7f1befeef000-7f1befef3000 r--p 001b6000 fd:00 201329588                  /usr/lib64/libc-2.17.so
7f1befef3000-7f1befef5000 rw-p 001ba000 fd:00 201329588                  /usr/lib64/libc-2.17.so
7f1befef5000-7f1befefa000 rw-p 00000000 00:00 0 
7f1befefa000-7f1beff42000 r-xp 00000000 fd:00 203854683                  /usr/lib64/libmultipath.so.0
7f1beff42000-7f1bf0142000 ---p 00048000 fd:00 203854683                  /usr/lib64/libmultipath.so.0
7f1bf0142000-7f1bf0143000 r--p 00048000 fd:00 203854683                  /usr/lib64/libmultipath.so.0
7f1bf0143000-7f1bf0148000 rw-p 00049000 fd:00 203854683                  /usr/lib64/libmultipath.so.0
7f1bf0148000-7f1bf0149000 rw-p 00000000 00:00 0 
7f1bf0149000-7f1bf014c000 r-xp 00000000 fd:00 201329594                  /usr/lib64/libdl-2.17.so
7f1bf014c000-7f1bf034b000 ---p 00003000 fd:00 201329594                  /usr/lib64/libdl-2.17.so
7f1bf034b000-7f1bf034c000 r--p 00002000 fd:00 201329594                  /usr/lib64/libdl-2.17.so
7f1bf034c000-7f1bf034d000 rw-p 00003000 fd:00 201329594                  /usr/lib64/libdl-2.17.so
7f1bf034d000-7f1bf039e000 r-xp 00000000 fd:00 202296243                  /usr/lib64/libdevmapper.so.1.02
7f1bf039e000-7f1bf059e000 ---p 00051000 fd:00 202296243                  /usr/lib64/libdevmapper.so.1.02
7f1bf059e000-7f1bf059f000 r--p 00051000 fd:00 202296243                  /usr/lib64/libdevmapper.so.1.02
7f1bf059f000-7f1bf05a2000 rw-p 00052000 fd:00 202296243                  /usr/lib64/libdevmapper.so.1.02
7f1bf05a2000-7f1bf05a4000 rw-p 00000000 00:00 0 
7f1bf05a4000-7f1bf05ba000 r-xp 00000000 fd:00 201329614                  /usr/lib64/libpthread-2.17.so
7f1bf05ba000-7f1bf07ba000 ---p 00016000 fd:00 201329614                  /usr/lib64/libpthread-2.17.so
7f1bf07ba000-7f1bf07bb000 r--p 00016000 fd:00 201329614                  /usr/lib64/libpthread-2.17.so
7f1bf07bb000-7f1bf07bc000 rw-p 00017000 fd:00 201329614                  /usr/lib64/libpthread-2.17.so
7f1bf07bc000-7f1bf07c0000 rw-p 00000000 00:00 0 
7f1bf07c0000-7f1bf07e1000 r-xp 00000000 fd:00 201329581                  /usr/lib64/ld-2.17.so
7f1bf09b7000-7f1bf09c0000 rw-p 00000000 00:00 0 
7f1bf09c0000-7f1bf09d2000 r-xp 00000000 fd:00 202181518                  /usr/lib64/libudev.so.1.6.2
7f1bf09d2000-7f1bf09d3000 ---p 00012000 fd:00 202181518                  /usr/lib64/libudev.so.1.6.2
7f1bf09d3000-7f1bf09d4000 r--p 00012000 fd:00 202181518                  /usr/lib64/libudev.so.1.6.2
7f1bf09d4000-7f1bf09d5000 rw-p 00013000 fd:00 202181518                  /usr/lib64/libudev.so.1.6.2
7f1bf09d5000-7f1bf09d6000 rw-p 00000000 00:00 0 
7f1bf09df000-7f1bf09e1000 rw-p 00000000 00:00 0 
7f1bf09e1000-7f1bf09e2000 r--p 00021000 fd:00 201329581                  /usr/lib64/ld-2.17.so
7f1bf09e2000-7f1bf09e3000 rw-p 00022000 fd:00 201329581                  /usr/lib64/ld-2.17.so
7f1bf09e3000-7f1bf09e4000 rw-p 00000000 00:00 0 
7ffe22bf6000-7ffe22c17000 rw-p 00000000 00:00 0                          [stack]
7ffe22d3f000-7ffe22d41000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Return:0


This issue was not seen on RHEL-7.1 or RHEL-7.2 alpha(device-mapper-multipath-0.4.9-77.el7.x86_64)


Version-Release number of selected component (if applicable):
device-mapper-multipath-0.4.9-82.el7.x86_64
kernel-3.10.0-306.0.1.el7
iscsi-initiator-utils-6.2.0.873-32.el7

How reproducible:
needs further testing

Steps to Reproduce:
1.Establish iSCSI session with many LUNs
2.Run multipath on them
3.see '*** buffer overflow detected ***: multipath terminated' message

Actual results:
multipath terminated

Expected results:
created multipath devices

Additional info:
see call trace in dmesg log

Comment 7 David Bulkow 2015-09-10 14:08:31 UTC

*** Bug 1261561 has been marked as a duplicate of this bug. ***

Comment 8 Martin Hoyer 2015-09-15 08:52:41 UTC

Working well with device-mapper-multipath-0.4.9-83.el7

Comment 9 errata-xmlrpc 2015-11-19 12:57:54 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2132.html

Note You need to log in before you can comment on or make changes to this bug.