Description of problem: It has been seen that on a RHEL 4.8 host, iscsi service takes a long time to stop if large number of iscsi-based multipath maps are present. For 512 multipathed device (512 LUNs * 2 paths = 1024 SCSI device), it took almost 1 hour to search and flush iscsi-based multipath maps. The console remained in the following condition for entire time [root@localhost ~]# /etc/init.d/iscsi stop Searching for iscsi-based multipath maps Following code snippet of iscsi script is the reason for the long time, list_dms() { i=0 dev_sds=`iscsi-ls -l | grep Device | awk '{print $2}'| sed -e "s%/dev/%%"` dev_mpaths=`dmsetup table | grep multipath | awk '{print $1}' | sed -e "s/://"` echo "Searching for iscsi-based multipath maps" for dev_sd in $dev_sds; do for dev_mpath in $dev_mpaths; do # check to see if "cat /sys/block/$dev_sd/dev" matches "dmsetup table", and if so, save the "mpathN" device major_minor=`cat /sys/block/$dev_sd/dev` dmsetup_mpath=`dmsetup table $dev_mpath | grep -c $major_minor` if [ $dmsetup_mpath -eq 1 ]; then found=0 for ((j=0; j<$i ; j++)) do if [ ${iscsi_mpaths[$j]} = $dev_mpath ]; then found=1 fi done if [ $found -ne 1 ]; then iscsi_mpaths[$i]=$dev_mpath let "i += 1" fi fi done done echo "Found" $i "maps" } The above function has a time complexity of n^3. Hence, it takes such a long time to search iscsi-based multipath maps when there are 1024 SCSI devices and 512 multipath maps. Version-Release number of selected component (if applicable): iscsi-initiator-utils-4.0.3.0-8 How reproducible: Always Steps to Reproduce: 1.Map large number of LUNs to the host 2.Rescan the iscsi sessions 3.Create multipath maps of the LUNs 4.Stop the iscsi service using /etc/init.d/iscsi stop Actual results: It takes almost 1 hour to search iscsi based multipath maps when 512 LUNs with 2 paths are mapped to the host. Expected results: iscsi service should not take such a long time to stop. Additional info: I made some changes to the list_dm() as follows list_dms() { i=0 dev_sds=`iscsi-ls -l | grep Device | awk '{print $2}'| sed -e "s%/dev/%%"` dev_mpaths=`dmsetup table | grep multipath | awk '{print $1}' | sed -e "s/://"` echo "Searching for iscsi-based multipath maps" for dev_mpath in $dev_mpaths; do dmsetup_major_minor=`dmsetup deps $dev_mpath | cut -d')' -f1 | cut -d'(' -f2 | sed -e "s/, /:/"` for dev_sd in $dev_sds; do # check to see if "cat /sys/block/$dev_sd/dev" matches "dmsetup table", and if so, save the "mpathN" device sd_major_minor=`cat /sys/block/$dev_sd/dev` if [ "$dmsetup_major_minor" = "$sd_major_minor" ]; then iscsi_mpaths[$i]=$dev_mpath let "i += 1" break; fi done done echo "Found" $i "maps" } The time decreased considerably and it took around 5 minutes to search 512 iscsi-based multipath maps.
This code looks ok as an optimization. However, it assumes you only need to check one /dev/sd* (path) for each multipath map to determine whether it is iscsi. This assumption should be fine, unless a mix of iSCSI and non-iSCSI paths are exported to the host for the same LUN / multipath map. For example: # dmsetup deps mpath0 2 dependencies : (8, 0) (8, 16) But the code produces: # dmsetup deps mpath0 | cut -d')' -f1 | cut -d'(' -f2 | sed -e "s/, /:/" 8:0 So it only looks at the first path for each map. If a vendor exports paths to the host of both iscsi and FCP for example, this code might miss an iscsi based map. I pulled out the new "list_dms" function into and compared it with the old oneMy tests indicate a definite speedup though. (See attached scripts.) My test setup: - 100 luns, 2 paths, to netapp sim - compared old list_dms to new list_dms Results: old_list_dms output: Found 100 maps real 5m29.498s user 0m22.375s sys 2m34.946s new_list_dms output: Found 100 maps real 1m20.474s user 0m8.408s sys 0m59.669s
Created attachment 456281 [details] script to test new 'list_dms' function
Created attachment 456282 [details] original 'list_dms' function
Created attachment 456285 [details] Patch to iscsi script on top of mike's 4.9 build Mike - this is the patch you should be able to put into the 4.9 build - assumes previous lvs patch in https://bugzilla.redhat.com/show_bug.cgi?id=212265#c41
Thanks Dave. The fix is in iscsi-initiator-utils-4.0.3.0-10. You can download from here: http://people.redhat.com/mchristi/iscsi/rhel4.9/iscsi-initiator-utils/
Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: * Previously, the iSCSI shutdown process could take a long time when dm-multipath was used with iSCSI devices. This update improves the iSCSI shutdown process has been improved and the shutdown time is reduced. (BZ#493585)
Technical note updated. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1 +1 @@ -* Previously, the iSCSI shutdown process could take a long time when dm-multipath was used with iSCSI devices. This update improves the iSCSI shutdown process has been improved and the shutdown time is reduced. (BZ#493585)+* Previously, the iSCSI shutdown process could take a long time when dm-multipath was used with iSCSI devices. This update improves the iSCSI shutdown process has been improved and the shutdown time is reduced.
Setup 512 LUN (256 LUN for each controler) using tgtd. Create 512 mpath for each with only 1 path. Before the patch(iscsi-initiator-utils.x86_64 0:4.0.3.0-8): iscsi stop take: real 54m7.110s user 21m30.805s sys 47m0.375s With iscsi-initiator-utils.x86_64 0:4.0.3.0-10 real 11m27.977s user 4m5.848s sys 7m38.484s Set this bug as verified.
Technical note updated. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1 +1 @@ -* Previously, the iSCSI shutdown process could take a long time when dm-multipath was used with iSCSI devices. This update improves the iSCSI shutdown process has been improved and the shutdown time is reduced.+Previously, the iSCSI shutdown process could take a long time when dm-multipath was used with iSCSI devices. This update improves the iSCSI shutdown process has been improved and the shutdown time is reduced.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2011-0245.html