Created attachment 786466 [details]
multipath list output

I can't recreate this with FC attached storage.  I'm going to try with iscsi. I don't currently have any hardware that will let me try srp or iser.  Have you tried taking multipath out of the picture and verifying that you can run this with md raid10 directly on top of the scsi devices?

I'm also not able to recreate this using the in-kernel open-iscsi initiator, and the linux target framework software target.

Created attachment 789132 [details]
scsi id script

Created attachment 789137 [details]
multipath config file

Created attachment 789139 [details]
tgt configuration script

(In reply to Ben Marzinski from comment #4)
> I'm also not able to recreate this using the in-kernel open-iscsi initiator,
> and the linux target framework software target.

Verified again. The crash reproduce on raid over multipath but does not reproduce on raid over the raw devices.

Attached script used to create the targets and luns on the target side.
Attached multipath.conf with script used to create the DMs over 4 paths each on the initiator side.

the command used for the traffic is:
# dd if=/dev/md0 of=/dev/null iflag=direct count=1 bs=128K

discovery was done as follows:
# iscsiadm -m discovery --op=new --op=delete --type sendtargets --portal 172.30.4.71:3261 -l
# iscsiadm -m discovery --op=new --op=delete --type sendtargets --portal 172.30.4.71:3262 -l

Roi, Ben,

We have seen a similar NULL pointer dereference in __bio_add_page() when using multipath and raid10.

We have been able to create a consistence reproduction scenario with dm devices (no dm-multipath involved) and raid10. 

Following script reproduces the very same problem quickly on our system. Note that the script assumes no other loop devices or dm devices are present (please change script accordingly if you have any loop devices or dm devices in your system.

-----------
#!/bin/bash

T_DIR="/tmp/test_raid"

[[ ! -d ${T_DIR} ]] && mkdir ${T_DIR}
echo "using working directory: ${T_DIR}"

# Create the 8 files.
for i in `seq 1 8`; do
    [[ ! -f ${T_DIR}/f_$i} ]] && dd if=/dev/zero of=${T_DIR}/f_${i} count=1 bs=1048576

    if [[ ! -e /dev/loop${i} ]]; then
       echo "Make loop device: /dev/loop${i}"
       mknod -m660 /dev/loop${i} b 7 ${i}
       chown root.disk /dev/loop${i}
       chmod 666 /dev/loop${i}
    fi
    losetup /dev/loop${i} ${T_DIR}/f_${i}

    echo "0 `blockdev --getsize /dev/loop${i}` linear /dev/loop${i} 0" | dmsetup create dm-01${i}
done

[[ ! -e  /proc/mdstat ]] && modprobe md

mdadm -v --create --level=raid10 --raid-devices=8 /dev/md0 /dev/dm-*

sleep 5

mkfs -t ext2 /dev/md0
[[ ! -d /mnt/raid10 ]] && mkdir /mnt/raid10
mount /dev/md0 /mnt/raid10

cp /boot/config* /mnt/raid10/.
-----------
It crashes immediately after the copy occurs, the very same way it happened with multipath and raid10.

After researching upstream we found the following patch that fixed the crash.

   md/raid10: fix problem with on-stack allocation of r10bio structure.
   https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit?  id=e0ee778528bbaad28a5c69d2e219269a3a096607

Roi, You can try and see if this patch fixes your panic.