Bug 1283181 - multipathd crash when admin try to remove non-multipath device
multipathd crash when admin try to remove non-multipath device
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: device-mapper-multipath (Show other bugs)
6.7
All Linux
unspecified Severity medium
: rc
: ---
Assigned To: Ben Marzinski
Lin Li
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-11-18 07:19 EST by masanari iida
Modified: 2016-05-10 16:13 EDT (History)
10 users (show)

See Also:
Fixed In Version: device-mapper-multipath-0.4.9.89.el6
Doc Type: Bug Fix
Doc Text:
Cause: Multipathd wasn't checking if a pointer was NULL before dereferencing it on a failed command line remove for a missing path device Consequence: multipathd would crash if users tried to remove a path that it wasn't monitoring with the "multipathd del path" command Fix: multipathd now checks if the pointer is NULL before dereferencing it. Result: multipathd no longer crashed when users try to remove paths that are not being monitored.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-05-10 16:13:30 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
corefile of multipathd collected by abrt (1.76 MB, application/octet-stream)
2015-11-18 07:28 EST, masanari iida
no flags Details

  None (edit)
Description masanari iida 2015-11-18 07:19:20 EST
Description of problem:
multipathd crash when admin try to remove non-multipath device from CLI.

Version-Release number of selected component (if applicable):
device-mapper-multipath-0.4.9-87.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Configure multipath 
/dev/sda and /dev/sdb are not a member of multipath.
/dev/sdc and /dev/sdd are member of multipath. (iscsi device).


# multipath -ll
mpatha (1IET     00010001) dm-0 IET,VIRTUAL-DISK
size=204M features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 33:0:0:1 sdc 8:32 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 34:0:0:1 sdd 8:48 active ready running

2.
# multipathd -k
multipathd> list paths
hcil     dev dev_t pri dm_st  chk_st dev_st  next_check
33:0:0:1 sdc 8:32  1   active ready  running XXXXXXXXX. 18/20
34:0:0:1 sdd 8:48  1   active ready  running XXXXXXXXX. 19/20
multipathd> del path sda
#  <=  multipathd segv and automatically back to shell prompt.

3.
kernel: multipathd[6043]: segfault at 0 ip 0000000000408aff sp 00007f7614dc1c00 error 6 in multipathd[400000+10000]
abrtd: Directory 'ccpp-2015-11-18-21:08:48-6041' creation detected
abrt[6081]: Saved core dump of pid 6041 (/sbin/multipathd) to /var/spool/abrt/ccpp-2015-11-18-21:08:48-6041 (1957888 bytes)


Actual results:
multipathd segv

Expected results:
multipathd CLI interface return warning about non-multipath device remove.

Additional info:

# grep -v "#" /etc/multipath.conf

defaults {
        user_friendly_names yes
}
blacklist {
        devnode "^(sda|sdb)[0-9]*"
}
blacklist_exceptions {
        wwid 00010001
}

Business impact: none
Comment 2 masanari iida 2015-11-18 07:28 EST
Created attachment 1096021 [details]
corefile of multipathd collected by abrt
Comment 3 masanari iida 2015-11-18 08:01:08 EST
Additional information
Following commands also cause the multipathd to crash
remove path sda  (sda is not managed by multipathd.)
remove path 8:48 (8:48 = /dev/sdc. It _is_ managed by multipathd.)
del path 8:48 (8:48 = /dev/sdc. It _is_ managed by multipathd.)

"del/remove path" with correct device (/dev/sdc or sdc) works fine.
But "del/remove path" with any other word crash the multipathd.  :(
Comment 4 masanari iida 2015-11-18 21:51:45 EST
Program terminated with signal SIGSEGV, Segmentation fault.
#0  ev_remove_path (devname=0x7fcb88001fa0 "sda", vecs=0x1914c10,
    del_sysdev=0x0) at main.c:595
595     main.c: No such file or directory.
[Current thread is 1 (LWP 1723)]
(gdb)
(gdb) info shared
From                To                  Syms Read   Shared Object Library
0x000000349e4077a0  0x000000349e436cf8  Yes (*)     ./lib64/libdevmapper.so.1.02
0x000000349d814550  0x000000349d832af8  Yes         ./lib64/libreadline.so.6
0x00000034a8006a30  0x00000034a801cec8  Yes         ./lib64/libncurses.so.5
0x000000349c000de0  0x000000349c001998  Yes         ./lib64/libdl.so.2
0x000000307680ec90  0x00000030768359b8  Yes         ./lib64/libmultipath.so
0x0000003076c021b0  0x0000003076c05f58  Yes         ./lib64/libmpathpersist.so.0
0x0000003076402910  0x0000003076412f78  Yes         ./lib64/libgcc_s.so.1
0x000000349c805760  0x000000349c811148  Yes         ./lib64/libpthread.so.0
0x000000349c41eaa0  0x000000349c5400cc  Yes         ./lib64/libc.so.6
0x000000349dc058d0  0x000000349dc15cb8  Yes         ./lib64/libselinux.so.1
0x000000349e003e80  0x000000349e02e618  Yes         ./lib64/libsepol.so.1
0x00000034a7002930  0x00000034a7009a78  Yes         ./lib64/libudev.so.0
0x00000034ae40c840  0x00000034ae415c08  Yes         ./lib64/libtinfo.so.5
0x000000349bc00b00  0x000000349bc19feb  Yes         ./lib64/ld-linux-x86-64.so.2
0x00007fcb8ebcfb50  0x00007fcb8ebd0478  Yes         ./lib64/multipath/libcheckdirectio.so
0x00007fcb8e9ce570  0x00007fcb8e9ce721  Yes         ./lib64/libaio.so.1
0x00007fcb8e7cd460  0x00007fcb8e7cd578  Yes         ./lib64/multipath/libprioconst.so
(*): Shared library is missing debugging information.

(gdb) bt
#0  ev_remove_path (devname=0x7fcb88001fa0 "sda", vecs=0x1914c10,
    del_sysdev=0x0) at main.c:595
#1  0x000000000040ac9d in parse_cmd (cmd=<optimized out>,
    reply=0x7fcb8e7cbd80, len=0x7fcb8e7cbd9c, data=0x1914c10) at cli.c:407
#2  0x0000000000407fc3 in uxsock_trigger (
    str=0x7fcb88001d10 "remove path sda", reply=0x7fcb8e7cbd80,
    len=0x7fcb8e7cbd9c, trigger_data=<optimized out>) at main.c:718
#3  0x00000000004095d8 in uxsock_listen (
    uxsock_trigger=0x407f70 <uxsock_trigger>, trigger_data=0x1914c10)
    at uxlsnr.c:148
#4  0x0000000000405ddb in uxlsnrloop (ap=0x1914c10) at main.c:895
#5  0x000000349c807a51 in start_thread (arg=0x7fcb8e7cc700)
    at pthread_create.c:301
#6  0x000000349c4e896d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
Comment 6 Lin Li 2016-02-23 22:22:30 EST
Reproduced on device-mapper-multipath-0.4.9-87.el6
1). Configure multipath 
/dev/sda and /dev/sdb are a member of multipath. (iscsi device).
/dev/sdc and /dev/sdd are not member of multipath. 

# multipath -ll
mpathb (360fff19abdd9552f8a36e5355226ba27) dm-0 EQLOGIC,100E-00
size=50G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 3:0:0:0 sdb 8:16 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 0:0:0:0 sda 8:0  active ready running

2). # multipathd -k
multipathd> list paths
hcil    dev dev_t pri dm_st  chk_st dev_st  next_check     
3:0:0:0 sdb 8:16  1   active ready  running XX........ 4/20
0:0:0:0 sda 8:0   1   active ready  running X......... 3/20
multipathd> del path sdc
#  <=  multipathd segv and automatically back to shell prompt.

3). # dmesg
multipathd[2031]: segfault at 0 ip 0000000000408aff sp 00007f73e6759c00 error 6 in multipathd (deleted)[400000+10000]

4). check /var/log/messages
Feb 24 04:03:43 storageqe-84 kernel: multipathd[2031]: segfault at 0 ip 0000000000408aff sp 00007f73e6759c00 error 6 in multipathd (deleted)[400000+10000]
Feb 24 04:03:43 storageqe-84 abrt[53057]: File '/sbin/multipathd' seems to be deleted
Feb 24 04:03:43 storageqe-84 abrt[53057]: Saved core dump of pid 2024 (/sbin/multipathd) to /var/spool/abrt/ccpp-2016-02-24-04:03:43-2024 (1945600 bytes)



Verified on device-mapper-multipath-0.4.9-92.el6
1).1). Configure multipath 
/dev/sda and /dev/sdb are a member of multipath. (iscsi device).
/dev/sdc and /dev/sdd are not member of multipath. 

# multipath -ll
mpathb (360fff19abdd9552f8a36e5355226ba27) dm-0 EQLOGIC,100E-00
size=50G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 3:0:0:0 sdb 8:16 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 0:0:0:0 sda 8:0  active ready running

2).multipathd -k
multipathd> list paths
hcil    dev dev_t pri dm_st  chk_st dev_st  next_check      
3:0:0:0 sdb 8:16  1   active ready  running XXXXXXXXX. 19/20
0:0:0:0 sda 8:0   1   active ready  running XXXXXXXXX. 19/20
multipathd> del path sdc
ok
multipathd> 

multipathd no longer crashed
Comment 8 errata-xmlrpc 2016-05-10 16:13:30 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0777.html

Note You need to log in before you can comment on or make changes to this bug.