Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1498340

Summary: ISCSI targets with multiple discovery IP's provides only a single path for Direct LUN's to VM
Product: Red Hat Enterprise Virtualization Manager Reporter: Ribu Tho <rabraham>
Component: ovirt-engineAssignee: Idan Shaby <ishaby>
Status: CLOSED DUPLICATE QA Contact: Elad <ebenahar>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.1.5CC: aefrat, ebenahar, lsurette, Rhev-m-bugs, srevivo, tnisan, vvasilev, ylavi
Target Milestone: ovirt-4.3.0   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-08-06 08:58:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
logs-15.1.17
none
vdsm.log-netapp none

Description Ribu Tho 2017-10-04 05:06:49 UTC 
Description of problem:

ISCSI Targets with 2 portal's provide only a single path to the VM direct LUN's . Additionally the database also has entry for single path . In the event of a migration due to the single path existing on the DB , it creates only 1 path on the new host making the VM not so highly available. 


Version-Release number of selected component (if applicable):

ovirt-engine-4.1.5.2-0.1.el7.noarch
vdsm-4.19.28-1.el7ev.x86_64
iscsi-initiator-utils-6.2.0.874-4.el7.x86_64


How reproducible:


Steps to Reproduce:
1. Create an ISCSI target with 2 portal addresses.

2. Try to create a VM and add a new DISK via Direct LUN option and enter 1st discovery IP. 

2.1  Discover 1st portal IP  -> Login -> Choose host as A --> Add disk occurs 

2.2  Discovery 2nd portal IP -> Login -> Choose host as A --> Add disk is greyed out due to the same device which is fine. 

3.   When doing a multipath -ll and iscsi session on host A it gives  2 entries which is correct. 

4. Database contains only 1 entry for step [2.1].

5. Start the VM and try to migrate to another host. 

6. New Host shows only 1 path due to single portal address info available  from the database. 


Actual results:

- Fails to recognize the multiple path's existing for direct LUN's attached to VM's.

Expected results:

- Allow multiple paths for direct LUN's attached to VM's .


Additional info:
Comment 3 Allon Mureinik 2017-10-15 10:27:43 UTC 
The discovery IP is indeed used only for discovery.

AFAIK, such multipathing should be available already via the REST API (and thus also SDKs), but as noted above, not via the GUI.

Tal, can you take a look please?
Comment 6 Tal Nisan 2017-12-27 13:45:48 UTC 
Aviahi, you've tested a similar scenario later, can you share your results?
Comment 7 Yaniv Kaul 2017-12-27 13:47:57 UTC 
(In reply to Tal Nisan from comment #6)
> Aviahi, you've tested a similar scenario later, can you share your results?

Please make sure to test against a direct you do NOT have a storage domain on. Otherwise, it'll use the existing SD connections (which do have multipathing).
Comment 9 Elad 2018-01-15 14:08:12 UTC 
Created attachment 1381460 [details]
logs-15.1.17

I wasn't able to reproduce over [1].
Multiple sessions are opened from the same storage target to all the iSCSI portals it provides. Tested the following:

1) With a host that doesn't have any opened iSCSI sessions to the target
2) Added direct LUN (operation that involved iSCSI discovery and login) 

[1]
vdsm-4.20.11-1.el7ev.x86_64
rhvm-4.2.1-0.2.el7.noarch



2018-01-15 15:58:04,961+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-16) [76b24418-9a30-495f-96bf-0dc98effb41f] START, ConnectStorageServerVDSCommand(HostName = host_mixed_3, StorageServerConnectionManagementVDSParameters:{hostId='5c2ee991-6583-4dc2-a072-db27aa7615d7', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='ISCSI', connectionList='[StorageServerConnections:{id='null', connection='10.35.146.193', iqn='iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f6c04', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 403590ae

2018-01-15 15:58:09,566+0200 DEBUG (jsonrpc/7) [storage.Misc.excCmd] /usr/bin/taskset --cpu-list 0-0 /usr/bin/sudo -n /sbin/iscsiadm -m node -T iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f6c05 -I default 
-p 10.35.146.225:3260,1 -n node.startup -v manual --op=update (cwd None) (commands:65)



[root@storage-ge6-vdsm3 ~]# iscsiadm -m session 
tcp: [6] 10.35.146.129:3260,1 iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f6c00 (non-flash)
tcp: [7] 10.35.146.161:3260,1 iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f6c01 (non-flash)
tcp: [8] 10.35.146.193:3260,1 iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f6c04 (non-flash)
tcp: [9] 10.35.146.225:3260,1 iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f6c05 (non-flash)
Comment 10 Tal Nisan 2018-01-15 14:28:48 UTC 
Elad, can you please test with 4.1.8?
Comment 11 Elad 2018-01-17 12:13:49 UTC 
The same results, I tested with 4.1.9
Comment 12 Yaniv Kaul 2018-01-17 16:27:07 UTC 
Perhaps it depends on the storage - if it shows all IPs from all portals or not. I believe XtremIO does.
Comment 13 Elad 2018-01-18 12:42:42 UTC 
Discovery with send targets response from XtremIO returns all:

2018-01-15 15:57:56,218+0200 INFO  (jsonrpc/7) [vdsm.api] START discoverSendTargets(con={'ipv6_enabled': False, 'connection': '10.35.146.129', 'password': '', 'port': '3260', 'user': ''}, options=None) from=::ff
ff:10.35.161.127,60874, flow_id=2270034a-be95-4c1b-8487-a675884c45ec, task_id=6175a77a-95e8-445d-8e23-f9f189ba152a (api:46)





2018-01-15 15:57:56,921+0200 INFO  (jsonrpc/7) [vdsm.api] FINISH discoverSendTargets return={'fullTargets': ['10.35.146.129:3260,1 iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f6c00', '10.35.146.161:3260,1 
iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f6c01', '10.35.146.193:3260,1 iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f6c04', '10.35.146.225:3260,1 iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f
6c05'], 'targets': ['iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f6c00', 'iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f6c01', 'iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f6c04', 'iqn.2008-05.c
om.xtremio:xio00153500071-514f0c50023f6c05']} from=::ffff:10.35.161.127,60874, flow_id=2270034a-be95-4c1b-8487-a675884c45ec, task_id=6175a77a-95e8-445d-8e23-f9f189ba152a (api:52)
2018-01-15 15:57:56,923+0200 DEBUG (jsonrpc/7) [storage.TaskManager.Task] (Task='6175a77a-95e8-445d-8e23-f9f189ba152a') finished: {'fullTargets': ['10.35.146.129:3260,1 iqn.2008-05.com.xtremio:xio00153500071-514
f0c50023f6c00', '10.35.146.161:3260,1 iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f6c01', '10.35.146.193:3260,1 iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f6c04', '10.35.146.225:3260,1 iqn.2008-05.c
om.xtremio:xio00153500071-514f0c50023f6c05'], 'targets': ['iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f6c00', 'iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f6c01', 'iqn.2008-05.com.xtremio:xio0015350
0071-514f0c50023f6c04', 'iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f6c05']} (task:1201)
Comment 14 Yaniv Kaul 2018-01-18 13:54:22 UTC 
(In reply to Elad from comment #13)
> Discovery with send targets response from XtremIO returns all:

Exactly. Please check with other storage vendors. I believe there you'll have to login to each separately.
Comment 15 Elad 2018-01-23 15:43:53 UTC 
Created attachment 1384906 [details]
vdsm.log-netapp

The same with Netapp (vdsm-4.19.45-1.el7ev.x86_64, rhevm-4.1.9.1-0.1.el7.noarch):

2018-01-23 17:37:14,295+0200 INFO  (jsonrpc/5) [vdsm.api] START connectStorageServer(domType=3, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'id': '00000000-0000-0000-0000-000000000000', 'connection': '10.35.66.207', 'iqn': 'iqn.1992-08.com.netapp:sn.9a14187dd2e911e4bbb900a09861387c:vs.6', 'user': u'', 'tpgt': '1', 'password': '********', 'port': '3260'}], options=None) from=::ffff:10.35.69.107,55658, flow_id=c979a415-fc6e-4de4-8088-722eb19c9bc4, task_id=7bc7c1d5-d8cf-4f90-8013-d00d1c856bd6 (api:46)



2018-01-23 17:37:24,354+0200 INFO  (jsonrpc/0) [vdsm.api] FINISH getDeviceList return={'devList': [{'status': 'unknown', 'vendorID': 'NETAPP', 'capacity': '53687091200', 'fwrev': '8200', 'discard_zeroes_data': 0
, 'vgUUID': '', 'pvsize': '53687091200', 'pathlist': [{'connection': '10.35.66.207', 'iqn': 'iqn.1992-08.com.netapp:sn.9a14187dd2e911e4bbb900a09861387c:vs.6', 'portal': '1035', 'port': '3260', 'initiatorname': '
default'}, {'connection': '10.35.66.208', 'iqn': 'iqn.1992-08.com.netapp:sn.9a14187dd2e911e4bbb900a09861387c:vs.6', 'portal': '1036', 'port': '3260', 'initiatorname': 'default'}], 'logicalblocksize': '512', 'dis
card_max_bytes': 8388608, 'pathstatus': [{'type': 'iSCSI', 'physdev': 'sdg', 'capacity': '53687091200', 'state': 'active', 'lun': '5'}, {'type': 'iSCSI', 'physdev': 'sdaj', 'capacity': '53687091200', 'state': 'a
ctive', 'lun': '5'}], 'devtype': 'iSCSI', 'physicalblocksize': '4096', 'pvUUID': 'UgbpoV-gGzW-3cKO-1PIj-w7dh-RXfQ-niFD6l', 'serial': 'SNETAPP_LUN_C-Mode_804zb_FzytWC', 'GUID': '3600a09803830347a625d467a79745743'
, 'productID': 'LUN C-Mode'},
Comment 16 Yaniv Kaul 2018-02-21 11:43:18 UTC 
(In reply to Elad from comment #15)
> Created attachment 1384906 [details]
> vdsm.log-netapp
> 
> The same with Netapp (vdsm-4.19.45-1.el7ev.x86_64,

Does Netapp expose all IPs from any portal, like XtremIO does?

> rhevm-4.1.9.1-0.1.el7.noarch):
> 
> 2018-01-23 17:37:14,295+0200 INFO  (jsonrpc/5) [vdsm.api] START
> connectStorageServer(domType=3,
> spUUID='00000000-0000-0000-0000-000000000000', conList=[{'id':
> '00000000-0000-0000-0000-000000000000', 'connection': '10.35.66.207', 'iqn':
> 'iqn.1992-08.com.netapp:sn.9a14187dd2e911e4bbb900a09861387c:vs.6', 'user':
> u'', 'tpgt': '1', 'password': '********', 'port': '3260'}], options=None)
> from=::ffff:10.35.69.107,55658,
> flow_id=c979a415-fc6e-4de4-8088-722eb19c9bc4,
> task_id=7bc7c1d5-d8cf-4f90-8013-d00d1c856bd6 (api:46)
> 
> 
> 
> 2018-01-23 17:37:24,354+0200 INFO  (jsonrpc/0) [vdsm.api] FINISH
> getDeviceList return={'devList': [{'status': 'unknown', 'vendorID':
> 'NETAPP', 'capacity': '53687091200', 'fwrev': '8200', 'discard_zeroes_data':
> 0
> , 'vgUUID': '', 'pvsize': '53687091200', 'pathlist': [{'connection':
> '10.35.66.207', 'iqn':
> 'iqn.1992-08.com.netapp:sn.9a14187dd2e911e4bbb900a09861387c:vs.6', 'portal':
> '1035', 'port': '3260', 'initiatorname': '
> default'}, {'connection': '10.35.66.208', 'iqn':
> 'iqn.1992-08.com.netapp:sn.9a14187dd2e911e4bbb900a09861387c:vs.6', 'portal':
> '1036', 'port': '3260', 'initiatorname': 'default'}], 'logicalblocksize':
> '512', 'dis
> card_max_bytes': 8388608, 'pathstatus': [{'type': 'iSCSI', 'physdev': 'sdg',
> 'capacity': '53687091200', 'state': 'active', 'lun': '5'}, {'type': 'iSCSI',
> 'physdev': 'sdaj', 'capacity': '53687091200', 'state': 'a
> ctive', 'lun': '5'}], 'devtype': 'iSCSI', 'physicalblocksize': '4096',
> 'pvUUID': 'UgbpoV-gGzW-3cKO-1PIj-w7dh-RXfQ-niFD6l', 'serial':
> 'SNETAPP_LUN_C-Mode_804zb_FzytWC', 'GUID':
> '3600a09803830347a625d467a79745743'
> , 'productID': 'LUN C-Mode'},
Comment 17 Elad 2018-02-27 08:32:52 UTC 
Yes, with netapp it's the same
Comment 18 Allon Mureinik 2018-03-25 13:24:40 UTC 
Anything we do implicitly here may harm other flows that rely on today's (confusing [bad]) behavior.

Pushing out to 4.3 to consider a proper way to manage connections and their relationship to luns.
Comment 19 Yaniv Lavi 2018-08-06 08:58:39 UTC 

*** This bug has been marked as a duplicate of bug 1172354 ***