This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 493585 - iscsi stop takes a long time if large number of iscsi-based multipath maps are present
iscsi stop takes a long time if large number of iscsi-based multipath maps ar...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: iscsi-initiator-utils (Show other bugs)
4.8
All Linux
medium Severity medium
: rc
: 4.9
Assigned To: Mike Christie
Gris Ge
:
Depends On:
Blocks: 626414
  Show dependency treegraph
 
Reported: 2009-04-02 07:37 EDT by Tanvi
Modified: 2011-02-16 09:26 EST (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, the iSCSI shutdown process could take a long time when dm-multipath was used with iSCSI devices. This update improves the iSCSI shutdown process has been improved and the shutdown time is reduced.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-02-16 09:26:56 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
script to test new 'list_dms' function (916 bytes, application/octet-stream)
2010-10-28 12:34 EDT, Dave Wysochanski
no flags Details
original 'list_dms' function (885 bytes, application/octet-stream)
2010-10-28 12:35 EDT, Dave Wysochanski
no flags Details
Patch to iscsi script on top of mike's 4.9 build (1.53 KB, patch)
2010-10-28 12:40 EDT, Dave Wysochanski
no flags Details | Diff

  None (edit)
Description Tanvi 2009-04-02 07:37:13 EDT
Description of problem: 
It has been seen that on a RHEL 4.8 host, iscsi service takes a long time to stop if large number of iscsi-based multipath maps are present. For 512 multipathed device (512 LUNs * 2 paths = 1024 SCSI device), it took almost 1 hour to search and flush iscsi-based multipath maps. The console remained in the following condition for entire time

[root@localhost ~]# /etc/init.d/iscsi stop
Searching for iscsi-based multipath maps

Following code snippet of iscsi script is the reason for the long time,

list_dms()
{
        i=0
        dev_sds=`iscsi-ls -l | grep Device | awk '{print $2}'| sed -e "s%/dev/%%"`
        dev_mpaths=`dmsetup table | grep multipath | awk '{print $1}' | sed -e "s/://"`

        echo "Searching for iscsi-based multipath maps"
        for dev_sd in $dev_sds; do
                for dev_mpath in $dev_mpaths; do
                # check to see if "cat /sys/block/$dev_sd/dev" matches "dmsetup table", and if so, save the "mpathN" device
                        major_minor=`cat /sys/block/$dev_sd/dev`
                        dmsetup_mpath=`dmsetup table $dev_mpath | grep -c $major_minor`
                        if [ $dmsetup_mpath -eq 1 ]; then
                                found=0
                                for ((j=0; j<$i ; j++))
                                do
                                        if [ ${iscsi_mpaths[$j]} = $dev_mpath ]; then
                                                found=1
                                        fi
                                done
                                if [ $found -ne 1 ]; then
                                        iscsi_mpaths[$i]=$dev_mpath
                                        let "i += 1"
                                fi
                        fi
                done
        done
        echo "Found" $i "maps"
}

The above function has a time complexity of n^3. Hence, it takes such a long time to search iscsi-based multipath maps when there are 1024 SCSI devices and 512 multipath maps.

Version-Release number of selected component (if applicable):
iscsi-initiator-utils-4.0.3.0-8

How reproducible:
Always

Steps to Reproduce:
1.Map large number of LUNs to the host 
2.Rescan the iscsi sessions
3.Create multipath maps of the LUNs
4.Stop the iscsi service using /etc/init.d/iscsi stop
  
Actual results:
It takes almost 1 hour to search iscsi based multipath maps when 512 LUNs with 2 paths are mapped to the host.

Expected results:
iscsi service should not take such a long time to stop.

Additional info:
I made some changes to the list_dm() as follows
list_dms()
{
i=0
        dev_sds=`iscsi-ls -l | grep Device | awk '{print $2}'| sed -e "s%/dev/%%"`
        dev_mpaths=`dmsetup table | grep multipath | awk '{print $1}' | sed -e "s/://"`

        echo "Searching for iscsi-based multipath maps"
        for dev_mpath in $dev_mpaths; do
  dmsetup_major_minor=`dmsetup deps $dev_mpath | cut -d')' -f1 | cut -d'(' -f2 | sed -e "s/, /:/"`
                for dev_sd in $dev_sds; do
                # check to see if "cat /sys/block/$dev_sd/dev" matches "dmsetup table", and if so, save the "mpathN" device
                        sd_major_minor=`cat /sys/block/$dev_sd/dev`
                        if [ "$dmsetup_major_minor" = "$sd_major_minor" ]; then
                                iscsi_mpaths[$i]=$dev_mpath
                                        let "i += 1"
                                        break;
                        fi
                done
        done
        echo "Found" $i "maps"
}

The time decreased considerably and it took around 5 minutes to search 512 iscsi-based multipath maps.
Comment 9 Dave Wysochanski 2010-10-28 12:33:59 EDT
This code looks ok as an optimization.  However, it assumes you only need to check one /dev/sd* (path) for each multipath map to determine whether it is iscsi.  This assumption should be fine, unless a mix of iSCSI and non-iSCSI paths are exported to the host for the same LUN / multipath map.

For example:
# dmsetup deps mpath0
2 dependencies	: (8, 0) (8, 16)

But the code produces:
# dmsetup deps mpath0 | cut -d')' -f1 | cut -d'(' -f2 | sed -e "s/, /:/"
8:0

So it only looks at the first path for each map.

If a vendor exports paths to the host of both iscsi and FCP for example, this code might miss an iscsi based map.

I pulled out the new "list_dms" function into and compared it with the old oneMy tests indicate a definite speedup though.  (See attached scripts.)

My test setup:
- 100 luns, 2 paths, to netapp sim
- compared old list_dms to new list_dms

Results:
old_list_dms output:
Found 100 maps

real    5m29.498s
user    0m22.375s
sys     2m34.946s

new_list_dms output:
Found 100 maps

real    1m20.474s
user    0m8.408s
sys     0m59.669s
Comment 10 Dave Wysochanski 2010-10-28 12:34:50 EDT
Created attachment 456281 [details]
script to test new 'list_dms' function
Comment 11 Dave Wysochanski 2010-10-28 12:35:19 EDT
Created attachment 456282 [details]
original 'list_dms' function
Comment 12 Dave Wysochanski 2010-10-28 12:40:50 EDT
Created attachment 456285 [details]
Patch to iscsi script on top of mike's 4.9 build

Mike - this is the patch you should be able to put into the 4.9 build - assumes previous lvs patch in https://bugzilla.redhat.com/show_bug.cgi?id=212265#c41
Comment 13 Mike Christie 2010-10-29 00:56:27 EDT
Thanks Dave.

The fix is in iscsi-initiator-utils-4.0.3.0-10. You can download from here:
http://people.redhat.com/mchristi/iscsi/rhel4.9/iscsi-initiator-utils/
Comment 17 Florian Nadge 2011-01-03 09:42:49 EST
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
* Previously, the iSCSI shutdown process could take a long time when dm-multipath was used with iSCSI devices. This update improves the iSCSI shutdown process has been improved and the shutdown time is reduced. (BZ#493585)
Comment 18 Florian Nadge 2011-01-03 09:43:46 EST
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1 +1 @@
-* Previously, the iSCSI shutdown process could take a long time when dm-multipath was used with iSCSI devices. This update improves the iSCSI shutdown process has been improved and the shutdown time is reduced. (BZ#493585)+* Previously, the iSCSI shutdown process could take a long time when dm-multipath was used with iSCSI devices. This update improves the iSCSI shutdown process has been improved and the shutdown time is reduced.
Comment 19 Gris Ge 2011-01-18 05:10:14 EST
Setup 512 LUN (256 LUN for each controler) using tgtd.
Create 512 mpath for each with only 1 path.

Before the patch(iscsi-initiator-utils.x86_64 0:4.0.3.0-8):
iscsi stop take:
real    54m7.110s
user    21m30.805s
sys     47m0.375s

With iscsi-initiator-utils.x86_64 0:4.0.3.0-10
real    11m27.977s
user    4m5.848s
sys     7m38.484s

Set this bug as verified.
Comment 20 Florian Nadge 2011-02-14 03:04:50 EST
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1 +1 @@
-* Previously, the iSCSI shutdown process could take a long time when dm-multipath was used with iSCSI devices. This update improves the iSCSI shutdown process has been improved and the shutdown time is reduced.+Previously, the iSCSI shutdown process could take a long time when dm-multipath was used with iSCSI devices. This update improves the iSCSI shutdown process has been improved and the shutdown time is reduced.
Comment 21 errata-xmlrpc 2011-02-16 09:26:56 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0245.html

Note You need to log in before you can comment on or make changes to this bug.