Description of problem: iSCSI multipath has performance issue on volume detachment when multiple LUNs are exposed via single target(iqn). Version-Release number of selected component (if applicable): OSP 7.2 How reproducible: 100% when using more then 10 volumes in a test Steps to Reproduce: 1.Customer has a heat template that they use to launch 20 instances and attache 20 volumes. When the heat stack is deleted issue manifests 2. 3. Actual results: Expected results: Additional info: https://bugs.launchpad.net/nova/+bug/1454978
Could also potentially be the following upstream bug as well: https://bugs.launchpad.net/nova/+bug/1336683
*** Bug 1347441 has been marked as a duplicate of this bug. ***
Created attachment 1190176 [details] Other Tunables We Set
Verified as follows - volume detach took only ~240 seconds for all of the 20 volumes to be detached from the instances. [root@serverA]# nova list +--------------------------------------+------+--------+------------+-------------+---------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------+--------+------------+-------------+---------------------+ | 825d11df-71e6-4083-9e31-68e2e4fe5ef3 | vm1 | ACTIVE | - | Running | public=172.24.4.238 | | 00376f67-d201-4f94-9c9f-8f8677ef5926 | vm10 | ACTIVE | - | Running | private=10.0.0.21 | | c5f36a2b-9e4f-4872-bd15-c9e7c24b4a87 | vm11 | ACTIVE | - | Running | private=10.0.0.22 | | c4e6c3ff-453c-47cd-9a7e-287307d2be1f | vm12 | ACTIVE | - | Running | private=10.0.0.23 | | db45e2d4-e276-4993-90b5-f0732fc24e08 | vm13 | ACTIVE | - | Running | private=10.0.0.24 | | 999d9ed4-83ca-43dc-be87-98678895889e | vm14 | ACTIVE | - | Running | private=10.0.0.25 | | a7875765-ca75-4483-9ad7-d643dd1c3f82 | vm15 | ACTIVE | - | Running | private=10.0.0.26 | | b0e8d366-7701-4a31-ae85-ae127f20b28b | vm16 | ACTIVE | - | Running | private=10.0.0.27 | | e66fffc8-9ccf-4de0-923e-b430252b3e3b | vm17 | ACTIVE | - | Running | private=10.0.0.28 | | 811d0990-a186-439c-a2e5-c5b2361fb245 | vm18 | ACTIVE | - | Running | private=10.0.0.29 | | 44082553-897d-4fc5-b1af-64db8667e1f6 | vm19 | ACTIVE | - | Running | private=10.0.0.30 | | 83b835ae-d553-4b83-aa34-7cc2ad929ac0 | vm2 | ACTIVE | - | Running | private=10.0.0.13 | | 350d4e78-5019-444f-a341-086624a474d2 | vm20 | ACTIVE | - | Running | private=10.0.0.31 | | aeb47574-ec64-4b2d-b1aa-9d1b0cb8a179 | vm3 | ACTIVE | - | Running | private=10.0.0.14 | | 3e133191-6577-483f-bf78-f60d646e7a00 | vm4 | ACTIVE | - | Running | private=10.0.0.15 | | 63f2b0ad-ce9a-48e4-960a-f50844554231 | vm5 | ACTIVE | - | Running | private=10.0.0.16 | | e1fff9f9-2318-4861-820a-75c194c51026 | vm6 | ACTIVE | - | Running | private=10.0.0.17 | | 2d373df8-a6e5-42fc-8ab4-9d216a193b8a | vm7 | ACTIVE | - | Running | private=10.0.0.18 | | 248220b7-adbd-4952-bfcb-e8e6a6e68c43 | vm8 | ACTIVE | - | Running | private=10.0.0.19 | | ddc61572-a8ba-432d-ae76-a30b9ceb5537 | vm9 | ACTIVE | - | Running | private=10.0.0.20 | +--------------------------------------+------+--------+------------+-------------+---------------------+ [root@serverA]# cinder list +--------------------------------------+--------+-------+------+-------------+----------+--------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+--------+-------+------+-------------+----------+--------------------------------------+ | 0f3473b0-0b3d-4e57-8c18-97c0fe62a37d | in-use | vol20 | 1 | iscsi | false | 825d11df-71e6-4083-9e31-68e2e4fe5ef3 | | 18ace8f6-3cd1-4626-9fc5-dce4d40d68ac | in-use | vol7 | 1 | iscsi | false | 83b835ae-d553-4b83-aa34-7cc2ad929ac0 | | 211ddb41-c7fe-4ebe-9080-a1752201e1af | in-use | vol6 | 1 | iscsi | false | aeb47574-ec64-4b2d-b1aa-9d1b0cb8a179 | | 3861519a-d05e-4e91-ae10-e1e5cfca2e51 | in-use | vol1 | 1 | iscsi | false | 63f2b0ad-ce9a-48e4-960a-f50844554231 | | 3fca2f93-5725-49bb-a137-ece57c8970d3 | in-use | vol4 | 1 | iscsi | false | 63f2b0ad-ce9a-48e4-960a-f50844554231 | | 47f8afa0-0798-4e7d-864c-351bd5658652 | in-use | vol16 | 1 | iscsi | false | b0e8d366-7701-4a31-ae85-ae127f20b28b | | 4cce56d0-27a7-44d6-9c16-049764fce632 | in-use | vol8 | 1 | iscsi | false | 248220b7-adbd-4952-bfcb-e8e6a6e68c43 | | 50ce84bd-9883-46b5-bf65-529dffd44dda | in-use | vol3 | 1 | iscsi | false | aeb47574-ec64-4b2d-b1aa-9d1b0cb8a179 | | 623bf000-3c65-4dd3-957e-5a91636cb73c | in-use | vol19 | 1 | iscsi | false | 350d4e78-5019-444f-a341-086624a474d2 | | 71be98b7-bd2a-4f95-a8d7-52201df2ee35 | in-use | vol11 | 1 | iscsi | false | c5f36a2b-9e4f-4872-bd15-c9e7c24b4a87 | | 8a7d1805-5af9-4537-8ca8-70ef55ae72b1 | in-use | vol10 | 1 | iscsi | false | 00376f67-d201-4f94-9c9f-8f8677ef5926 | | 8b1ac8db-19b5-4ddf-838e-2145ad1a5e92 | in-use | vol2 | 1 | iscsi | false | 3e133191-6577-483f-bf78-f60d646e7a00 | | 975a7207-064a-4781-af52-d0abe6b6bef9 | in-use | vol9 | 1 | iscsi | false | ddc61572-a8ba-432d-ae76-a30b9ceb5537 | | 9f62593b-7df4-4c2d-8cd1-1166f5e924e6 | in-use | vol15 | 1 | iscsi | false | a7875765-ca75-4483-9ad7-d643dd1c3f82 | | a1fce700-15b4-4816-89db-17c0d1b55859 | in-use | vol17 | 1 | iscsi | false | 811d0990-a186-439c-a2e5-c5b2361fb245 | | b1c3b096-3f0d-42a4-9f70-559cb7378752 | in-use | vol14 | 1 | iscsi | false | 999d9ed4-83ca-43dc-be87-98678895889e | | d2d80ad4-6d61-40a6-97ab-d53e979e1146 | in-use | vol13 | 1 | iscsi | false | db45e2d4-e276-4993-90b5-f0732fc24e08 | | d459d166-b85d-4b27-8263-146fe94c0d77 | in-use | vol5 | 1 | iscsi | false | 2d373df8-a6e5-42fc-8ab4-9d216a193b8a | | daa6350b-880c-4cf6-8db1-2778fec32b26 | in-use | vol18 | 1 | iscsi | false | 44082553-897d-4fc5-b1af-64db8667e1f6 | | fd1b0f03-1724-4bac-8882-c3116cadf592 | in-use | vol12 | 1 | iscsi | false | c4e6c3ff-453c-47cd-9a7e-287307d2be1f | +--------------------------------------+--------+-------+------+-------------+----------+--------------------------------------+ [root@serverA]# multipath -ll 360014057d9f3732966a4c18923ff0af5 dm-54 LIO-ORG ,IBLOCK size=1.0G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 70:0:0:0 sdaj 66:48 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 71:0:0:0 sdak 66:64 active ready running 36001405782bb279728847bd8ba9d5b5c dm-34 LIO-ORG ,IBLOCK size=1.0G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 50:0:0:0 sdp 8:240 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 51:0:0:0 sdq 65:0 active ready running 360014054bf198386b2440359fbfe93f3 dm-32 LIO-ORG ,IBLOCK size=1.0G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 48:0:0:0 sdn 8:208 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 49:0:0:0 sdo 8:224 active ready running 360014054003df8173254b88b5c7724c8 dm-44 LIO-ORG ,IBLOCK size=1.0G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 60:0:0:0 sdz 65:144 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 61:0:0:0 sdaa 65:160 active ready running 36001405995a208c314045e2a27204cd1 dm-46 LIO-ORG ,IBLOCK size=1.0G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 62:0:0:0 sdab 65:176 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 63:0:0:0 sdac 65:192 active ready running 360014052c891d5965c54c0e9e82bf0d6 dm-28 LIO-ORG ,IBLOCK size=1.0G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 44:0:0:0 sdj 8:144 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 45:0:0:0 sdk 8:160 active ready running 36001405e53d00808bba4fb9adf3518a5 dm-50 LIO-ORG ,IBLOCK size=1.0G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 66:0:0:0 sdaf 65:240 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 67:0:0:0 sdag 66:0 active ready running 360014051efb9aacb7134cbeaa6bc98da dm-52 LIO-ORG ,IBLOCK size=1.0G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 68:0:0:0 sdah 66:16 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 69:0:0:0 sdai 66:32 active ready running 360014056945998506b44802aa1f35ac1 dm-48 LIO-ORG ,IBLOCK size=1.0G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 64:0:0:0 sdad 65:208 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 65:0:0:0 sdae 65:224 active ready running 36001405be614033696c4b5fa006aa460 dm-38 LIO-ORG ,IBLOCK size=1.0G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 54:0:0:0 sdt 65:48 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 55:0:0:0 sdu 65:64 active ready running 36001405f738658ff0d54e10b18c9f6f8 dm-42 LIO-ORG ,IBLOCK size=1.0G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 58:0:0:0 sdx 65:112 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 59:0:0:0 sdy 65:128 active ready running 3600140529c15ef4f1004ff79cd02e838 dm-26 LIO-ORG ,IBLOCK size=1.0G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 42:0:0:0 sdh 8:112 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 43:0:0:0 sdi 8:128 active ready running 36001405f6b1a822f27c4173b124cd1b5 dm-22 LIO-ORG ,IBLOCK size=1.0G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 38:0:0:0 sdd 8:48 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 39:0:0:0 sde 8:64 active ready running 36001405c6e6ec9fd7b740908d8740e67 dm-20 LIO-ORG ,IBLOCK size=1.0G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 36:0:0:0 sdb 8:16 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 37:0:0:0 sdc 8:32 active ready running 3600140518aef018a7d4485fa94d18f01 dm-56 LIO-ORG ,IBLOCK size=1.0G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 72:0:0:0 sdal 66:80 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 73:0:0:0 sdam 66:96 active ready running 360014051e4251b9894d44f5987edee2e dm-24 LIO-ORG ,IBLOCK size=1.0G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 40:0:0:0 sdf 8:80 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 41:0:0:0 sdg 8:96 active ready running 3600140598ec43c460fd4b829551e26c0 dm-40 LIO-ORG ,IBLOCK size=1.0G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 56:0:0:0 sdv 65:80 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 57:0:0:0 sdw 65:96 active ready running 360014057153f7e71665476e991694a98 dm-58 LIO-ORG ,IBLOCK size=1.0G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 74:0:0:0 sdan 66:112 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 75:0:0:0 sdao 66:128 active ready running 360014051215b71d3c11438fbe98e735a dm-30 LIO-ORG ,IBLOCK size=1.0G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 46:0:0:0 sdl 8:176 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 47:0:0:0 sdm 8:192 active ready running 36001405666aa9d27cac49d08515e35f4 dm-36 LIO-ORG ,IBLOCK size=1.0G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 52:0:0:0 sdr 65:16 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 53:0:0:0 sds 65:32 active ready running [root@serverA]# [root@serverA]# date Mon Aug 15 22:17:28 IDT 2016 [root@serverA]# cinder list +--------------------------------------+-----------+-------+------+-------------+----------+--------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+-------+------+-------------+----------+--------------------------------------+ | 0f3473b0-0b3d-4e57-8c18-97c0fe62a37d | available | vol20 | 1 | iscsi | false | | | 18ace8f6-3cd1-4626-9fc5-dce4d40d68ac | available | vol7 | 1 | iscsi | false | | | 211ddb41-c7fe-4ebe-9080-a1752201e1af | available | vol6 | 1 | iscsi | false | | | 3861519a-d05e-4e91-ae10-e1e5cfca2e51 | available | vol1 | 1 | iscsi | false | | | 3fca2f93-5725-49bb-a137-ece57c8970d3 | detaching | vol4 | 1 | iscsi | false | 63f2b0ad-ce9a-48e4-960a-f50844554231 | | 47f8afa0-0798-4e7d-864c-351bd5658652 | detaching | vol16 | 1 | iscsi | false | b0e8d366-7701-4a31-ae85-ae127f20b28b | | 4cce56d0-27a7-44d6-9c16-049764fce632 | detaching | vol8 | 1 | iscsi | false | 248220b7-adbd-4952-bfcb-e8e6a6e68c43 | | 50ce84bd-9883-46b5-bf65-529dffd44dda | detaching | vol3 | 1 | iscsi | false | aeb47574-ec64-4b2d-b1aa-9d1b0cb8a179 | | 623bf000-3c65-4dd3-957e-5a91636cb73c | detaching | vol19 | 1 | iscsi | false | 350d4e78-5019-444f-a341-086624a474d2 | | 71be98b7-bd2a-4f95-a8d7-52201df2ee35 | detaching | vol11 | 1 | iscsi | false | c5f36a2b-9e4f-4872-bd15-c9e7c24b4a87 | | 8a7d1805-5af9-4537-8ca8-70ef55ae72b1 | detaching | vol10 | 1 | iscsi | false | 00376f67-d201-4f94-9c9f-8f8677ef5926 | | 8b1ac8db-19b5-4ddf-838e-2145ad1a5e92 | detaching | vol2 | 1 | iscsi | false | 3e133191-6577-483f-bf78-f60d646e7a00 | | 975a7207-064a-4781-af52-d0abe6b6bef9 | detaching | vol9 | 1 | iscsi | false | ddc61572-a8ba-432d-ae76-a30b9ceb5537 | | 9f62593b-7df4-4c2d-8cd1-1166f5e924e6 | detaching | vol15 | 1 | iscsi | false | a7875765-ca75-4483-9ad7-d643dd1c3f82 | | a1fce700-15b4-4816-89db-17c0d1b55859 | detaching | vol17 | 1 | iscsi | false | 811d0990-a186-439c-a2e5-c5b2361fb245 | | b1c3b096-3f0d-42a4-9f70-559cb7378752 | detaching | vol14 | 1 | iscsi | false | 999d9ed4-83ca-43dc-be87-98678895889e | | d2d80ad4-6d61-40a6-97ab-d53e979e1146 | detaching | vol13 | 1 | iscsi | false | db45e2d4-e276-4993-90b5-f0732fc24e08 | | d459d166-b85d-4b27-8263-146fe94c0d77 | detaching | vol5 | 1 | iscsi | false | 2d373df8-a6e5-42fc-8ab4-9d216a193b8a | | daa6350b-880c-4cf6-8db1-2778fec32b26 | detaching | vol18 | 1 | iscsi | false | 44082553-897d-4fc5-b1af-64db8667e1f6 | | fd1b0f03-1724-4bac-8882-c3116cadf592 | detaching | vol12 | 1 | iscsi | false | c4e6c3ff-453c-47cd-9a7e-287307d2be1f | +--------------------------------------+-----------+-------+------+-------------+----------+--------------------------------------+ +--------------------------------------+-----------+-------+------+-------------+----------+--------------------------------------+ [root@serverA]# cinder list +--------------------------------------+-----------+-------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+-------+------+-------------+----------+-------------+ | 0f3473b0-0b3d-4e57-8c18-97c0fe62a37d | available | vol20 | 1 | iscsi | false | | | 18ace8f6-3cd1-4626-9fc5-dce4d40d68ac | available | vol7 | 1 | iscsi | false | | | 211ddb41-c7fe-4ebe-9080-a1752201e1af | available | vol6 | 1 | iscsi | false | | | 3861519a-d05e-4e91-ae10-e1e5cfca2e51 | available | vol1 | 1 | iscsi | false | | | 3fca2f93-5725-49bb-a137-ece57c8970d3 | available | vol4 | 1 | iscsi | false | | | 47f8afa0-0798-4e7d-864c-351bd5658652 | available | vol16 | 1 | iscsi | false | | | 4cce56d0-27a7-44d6-9c16-049764fce632 | available | vol8 | 1 | iscsi | false | | | 50ce84bd-9883-46b5-bf65-529dffd44dda | available | vol3 | 1 | iscsi | false | | | 623bf000-3c65-4dd3-957e-5a91636cb73c | available | vol19 | 1 | iscsi | false | | | 71be98b7-bd2a-4f95-a8d7-52201df2ee35 | available | vol11 | 1 | iscsi | false | | | 8a7d1805-5af9-4537-8ca8-70ef55ae72b1 | available | vol10 | 1 | iscsi | false | | | 8b1ac8db-19b5-4ddf-838e-2145ad1a5e92 | available | vol2 | 1 | iscsi | false | | | 975a7207-064a-4781-af52-d0abe6b6bef9 | available | vol9 | 1 | iscsi | false | | | 9f62593b-7df4-4c2d-8cd1-1166f5e924e6 | available | vol15 | 1 | iscsi | false | | | a1fce700-15b4-4816-89db-17c0d1b55859 | available | vol17 | 1 | iscsi | false | | | b1c3b096-3f0d-42a4-9f70-559cb7378752 | available | vol14 | 1 | iscsi | false | | | d2d80ad4-6d61-40a6-97ab-d53e979e1146 | available | vol13 | 1 | iscsi | false | | | d459d166-b85d-4b27-8263-146fe94c0d77 | available | vol5 | 1 | iscsi | false | | | daa6350b-880c-4cf6-8db1-2778fec32b26 | available | vol18 | 1 | iscsi | false | | | fd1b0f03-1724-4bac-8882-c3116cadf592 | available | vol12 | 1 | iscsi | false | | +--------------------------------------+-----------+-------+------+-------------+----------+-------------+ [root@serverA]# date Mon Aug 15 22:21:27 IDT 2016
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-1623.html