Bug 895110

Created attachment 678263 [details]
multipath_malloc error

This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.

How many LUNs are on the server?  I'm not able to recreate this with the identical packages, using 100 interfaces to 10 LUNs

I was able to reproduce it using tgtd with 1 target with 1 LUN.

cat /etc/tgt/targets.conf
default-driver iscsi
<target iqn.2009-10.com.redhat:storage-1>
    write-cache off
    allow-in-use yes
    <backing-store /var/lib/tgtd/loop-disk-1-1>
        scsi_sn 1
        scsi_id 1
        lun 1
    </backing-store>
</target>

-------------
for i in {1..100}; do 
iscsiadm -m iface -o new -I multi_iqn_$i; 
iscsiadm -m iface -I multi_iqn_$i -o update -n iface.initiatorname -v iqn.1994-05.com.redhat:multi-iqn-1; 
iscsiadm -m discovery -t st -p 127.0.0.1 -I multi_iqn_$i -o new;
done

# multipath -l
#

# iscsiadm -m node -l

# multipath -l | grep sd | wc
     38     303    1824
#


NOTE: I could not see the error being logged in any file, only on the console. It also showed:

mp->params too small
mp->params too small
mp->params too small

Hrm.. That was the first thing I tried.  I didn't have the targets on the same node, however.  I'll try that.

Could you also post your /etc/multipath.conf.

Created attachment 691422 [details]
multipath.conf

I don't have much hope for this, but can you try the packages at:

http://download.devel.redhat.com/brewroot/scratch/bmarzins/task_5351856/

Those will fix the "mp->params too small" messages.

rpm -q device-mapper-multipath
device-mapper-multipath-0.4.9-64.el6.bz895110.x86_64

With this build the is no error messages, but it seems it still can only handle 51 paths.

iscsiadm -m session -P3 | grep sd | wc
    100     600    4274

multipath -l | grep sd | wc
     51     408    2448

There are some error messages when login out with iscsiadm -m node -u

device-mapper: table: 253:2: multipath: error getting device
device-mapper: table: 253:2: multipath: error getting device

So, did multipathd crash?  Would it be possible for me to get on this machine to take a look at it myself?

Actually, with just some debugging info added, everything worked just fine, with one small issue: "multipath -l" doesn't correctly display a multipath device that big.

# multipath -l | grep sd | wc -l
55

similar to your Comment 8 result.

However looking at the output

# multipath -l

...
|-+- policy='round-robin 0' prio=0 status=enabled
| `- 59:0:0:1  sdau 66:224 active undef running
|-+- policy='round-robin 0' prio=0 status=enabled
| `- 80:0:0:1  sdbh 67:176 active undef running
|-+- policy='round-robin 0' prio=0 s


It's pretty clearly cut off mid-line

Using multipathd to look at the device

# multipathd show topology map mpathc | grep sd | wc -l
100

does show all hundred paths, and look at /var/log/messages

Feb  7 11:21:08 storageqe-12 multipathd: mpathc: sdcp - directio checker reports path is up
Feb  7 11:21:08 storageqe-12 multipathd: 69:208: reinstated
Feb  7 11:21:08 storageqe-12 multipathd: mpathc: remaining active paths: 100

so the device does have 100 active paths.

I'll fix the multipath -l size issue.

The original corruption issue is more worrying.  I did expand a buffer that wasn't big enough, but I don't see where multipathd writes to it without size limiting, so unless I'm missing something, there still should be a overwrite, or a write after free, or something.  However, I've been totally unable to reproduce it, even with running valgrind with all the memory zeroing options.  That doesn't mean it's not there anymore, it just means the only way to find it is to notice it while reading the code.

If you can reproduce the corruption, please let me know.

thanks, if I reproduce the crash I'll let you know, and when login out of the sessions did you reproduce this error?

iscsiadm: initiator reported error (9 - internal error)
iscsiadm: Could not logout of [sid: 119, target: iqn.2009-10.com.redhat:storage-
1, portal: 127.0.0.1,3260].
iscsiadm: initiator reported error (9 - internal error)
iscsiadm: Could not logout of [sid: 118, target: iqn.2009-10.com.redhat:storage-
1, portal: 127.0.0.1,3260].
iscsiadm: initiator reported error (9 - internal error)
iscsiadm: Could not logout of [sid: 117, target: iqn.2009-10.com.redhat:storage-
1, portal: 127.0.0.1,3260].

*** Bug 723169 has been marked as a duplicate of this bug. ***

This should be fixed by the fix for 880121

*** This bug has been marked as a duplicate of bug 880121 ***