Bug 1202471 - [RFE] - adding new paths to iscsi data storage domains
Summary: [RFE] - adding new paths to iscsi data storage domains
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: RFEs
Version: ---
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
: ---
Assignee: Maor
QA Contact: Raz Tamir
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-03-16 17:19 UTC by pmdyermms
Modified: 2016-12-06 23:29 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Feature: add iSCSI paths to existing target Reason: bonding is not an option, multipath works better for SAN since there are no network switch configurations required. Result:
Clone Of:
Environment:
Last Closed: 2016-12-06 23:29:09 UTC
oVirt Team: Storage
ylavi: ovirt-future?
ylavi: planning_ack?
ylavi: devel_ack?
ylavi: testing_ack?


Attachments (Terms of Use)
engine log (384.80 KB, application/x-xz)
2015-03-16 17:19 UTC, pmdyermms
no flags Details
vdsm log (619.85 KB, application/x-xz)
2015-03-16 17:20 UTC, pmdyermms
no flags Details
network png (50.04 KB, image/png)
2016-02-01 17:50 UTC, pmdyermms
no flags Details
storage server connections png (24.14 KB, image/png)
2016-02-01 17:51 UTC, pmdyermms
no flags Details
add a new nic with iscsi bond (620.00 KB, application/x-gzip)
2016-04-07 11:48 UTC, Maor
no flags Details
host hooks event script (43.53 KB, image/png)
2016-05-02 22:47 UTC, pmdyermms
no flags Details
network table with 6 nics included (45.51 KB, image/png)
2016-12-06 16:14 UTC, pmdyermms
no flags Details

Description pmdyermms 2015-03-16 17:19:31 UTC
Created attachment 1002409 [details]
engine log

Description of problem:
added more nics to a host after the iSCSI storage domain was created.   The new paths do not get autologin during boot or after activating the host.

I looked in the engine table storage_server_connections and found that only the iscsi targets selected during the original storage domain create were present.


Version-Release number of selected component (if applicable):


How reproducible:
After installing the new hardware nics, from the Manager Console:

Steps to Reproduce:
1. Networks tab, New: add data32 for 10.251.8 network, add data33 for 10.251.9 network
2. Hosts tab, select host, Network Interfaces, Setup Host Networks, drag data32 to p4p1, drag data33 to p4p2, edit p4p1 and p4p2 to set protocol static and ipaddr/netmask
3. Storage tab, select kvm5DataDomain, Edit, Target to Luns, Discover targets 10.251.7.101, Login All, should get 8 targets
4. repeat Storage tab for each host, after moving SPM 

Actual results:
creates entries in /var/lib/iscsi/nodes for each of the targets added, along with login.   The "node.tpgt = 1" was set incorrectly for the case where it needed to be set to 2.

Expected results:
Set the node.tpgt correctly for this device type.   Add rows to engine table storage_server_connections so that the new paths would login at boot and activate.

Additional info:

Storage device is Dell MD3200i with 2 controllers, 4 nics each.  Controller 0, ports 0,1,2,3 uses portal target 1.    Controller 1, ports uses portal target 2.

The 4 group nics have different subnets.  Dell docs recommend this, and the storage devices keeps packets separated by these subnets.   For my setup, I have this config:
       controller 0      controller 1        host-1            host-2
nic0   10.251.6.101      10.251.6.102     10.251.6.135      10.251.6.136
nic1   10.251.7.101      10.251.7.102     10.251.7.135      10.251.7.136
nic2   10.251.8.101      10.251.8.102     10.251.8.135      10.251.8.136
nic3   10.251.9.101      10.251.9.102     10.251.9.135      10.251.9.136

I have attached the engine and vdsm logs.   I was working on this at about 11:40am Feb 4th.   At that time, I added only 1 addtional nic.

Comment 1 pmdyermms 2015-03-16 17:20:25 UTC
Created attachment 1002410 [details]
vdsm log

Comment 2 Allon Mureinik 2015-03-17 07:46:24 UTC
IIUC, you should use the iSCSI bond subtab.
Maor, can you please take a look?

Comment 3 Ori Gofen 2015-03-19 08:52:26 UTC
Hi pmdyermms,
For using multiple NICs with iSCSI Storage Domain it is better to configure iSCSI Bond (see [1])

[1]
http://www.ovirt.org/Feature/iSCSI-Multipath#User_Experience

Can u please try to configure one and let me know if that solved your problem

Comment 4 Maor 2015-03-19 08:57:22 UTC
(In reply to Ori Gofen from comment #3)
> Hi pmdyermms,
> For using multiple NICs with iSCSI Storage Domain it is better to configure
> iSCSI Bond (see [1])
> 
> [1]
> http://www.ovirt.org/Feature/iSCSI-Multipath#User_Experience
> 
> Can u please try to configure one and let me know if that solved your problem

I had a little problem with the bugzila session.
comment 3 was written by me (not ogofen), 

Can u please try to configure the iSCSI bond and let me know if that solved your problem

Thanks,
Maor

Comment 5 pmdyermms 2015-03-19 14:55:19 UTC
Hi,

thank you for the help.   Unfortunately, bond would not work for me, even if ovirt configures it correctly.

Dell support suggests the iscsi multipath is best for the MD3200i.   Bond is better when using a NAS device.    Second reason for not using bond: I do not have a level-3 switch.   In order to use bonding, the nics need a switch config done.   My switches are all level-2 switches.

The iscsi with multipath is setup to do round-robin, with failover between to 2 controllers on the device.   I have done manual setup, and it looks like this:

# multipath -ll
36f01faf000d7de06000002343a6fcf68 dm-0 DELL,MD32xxi
size=2.2T features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=14 status=active
| |- 6:0:0:2  sdg 8:96   active ready running
| |- 10:0:0:2 sdm 8:192  active ready running
| |- 5:0:0:2  sdp 8:240  active ready running
| `- 9:0:0:2  sdw 65:96  active ready running
`-+- policy='round-robin 0' prio=9 status=enabled
  |- 7:0:0:2  sdd 8:48   active ready running
  |- 8:0:0:2  sdl 8:176  active ready running
  |- 11:0:0:2 sdx 65:112 active ready running
  `- 12:0:0:2 sdy 65:128 active ready running

The prio=14 means that this controller does all the work.   The packets use sdg,sdm,sdp,sdw in a round-robin fashion.    If the first controller fails, then the prio=9 controller handles the work.

The storage_server_connections table only has 2 rows corresponding to sdg and sdd, because I setup the iscsi connection with only those nics originally.   If I had created the storage domain with all 8 nics, then I would have 8 rows in storage_server_connections.   Adding nics later, with iscsi discovery, should add those extra rows into storage_server_connections.  Just my opinion here.

Thanks,
Paul

Comment 6 pmdyermms 2015-04-07 15:53:21 UTC
Hi,

could this be changed from bug to enhancement?   It looks like it was never coded to add new iSCSI path in a multipath environment.

Thanks,
Paul

Comment 7 Red Hat Bugzilla Rules Engine 2015-10-19 10:53:27 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 8 Allon Mureinik 2016-01-20 17:34:28 UTC
(In reply to pmdyermms from comment #5)
> The storage_server_connections table only has 2 rows corresponding to sdg
> and sdd, because I setup the iscsi connection with only those nics
> originally.   If I had created the storage domain with all 8 nics, then I
> would have 8 rows in storage_server_connections.   Adding nics later, with
> iscsi discovery, should add those extra rows into
> storage_server_connections.  Just my opinion here.

So doens't the iSCSI Multipath feature do just that?
(if you use it form the front door, not discover manually on the host)

Comment 9 pmdyermms 2016-02-01 17:43:07 UTC
No, it does not for adding new paths.

Actually, only the network discovery from the time of data domain creation gets to the storage_server_connections table.    Anything added after that doesn't get managed by rhev/ovirt.

I have attached pictures of the network and storage_server_connections tables.   The connections 10.251.6.101 and 10.251.6.102 were present when the data domain was created.  The network picture show data30 logical network available first, then data31, data32 and data33 becoming available much later.   Their IP addresses do not appear in the storage_server_connections table.

On reboot of the physical host, I alway have to login and issue as root "iscsiadm -m node -L all" to get all the targets to login, and get multipath to see them.

Comment 10 pmdyermms 2016-02-01 17:50:58 UTC
Created attachment 1120203 [details]
network png

Comment 11 pmdyermms 2016-02-01 17:51:53 UTC
Created attachment 1120204 [details]
storage server connections png

Comment 12 Yaniv Kaul 2016-03-02 21:06:47 UTC
(In reply to pmdyermms from comment #5)
> Hi,
> 
> thank you for the help.   Unfortunately, bond would not work for me, even if
> ovirt configures it correctly.
> 
> Dell support suggests the iscsi multipath is best for the MD3200i.   Bond is
> better when using a NAS device.    Second reason for not using bond: I do
> not have a level-3 switch.   In order to use bonding, the nics need a switch
> config done.   My switches are all level-2 switches.

The term bond referred to in comment 4 did not refer to network bond, but to 'ISCSI bond' - which is a poor terminology to describe TWO (or more) outbound interfaces working to the target(s).
(Without it, you have a single point of failure on your host side, for example).

> 
> The iscsi with multipath is setup to do round-robin, with failover between
> to 2 controllers on the device.   I have done manual setup, and it looks
> like this:
> 
> # multipath -ll
> 36f01faf000d7de06000002343a6fcf68 dm-0 DELL,MD32xxi
> size=2.2T features='3 queue_if_no_path pg_init_retries 50' hwhandler='1
> rdac' wp=rw
> |-+- policy='round-robin 0' prio=14 status=active
> | |- 6:0:0:2  sdg 8:96   active ready running
> | |- 10:0:0:2 sdm 8:192  active ready running
> | |- 5:0:0:2  sdp 8:240  active ready running
> | `- 9:0:0:2  sdw 65:96  active ready running
> `-+- policy='round-robin 0' prio=9 status=enabled
>   |- 7:0:0:2  sdd 8:48   active ready running
>   |- 8:0:0:2  sdl 8:176  active ready running
>   |- 11:0:0:2 sdx 65:112 active ready running
>   `- 12:0:0:2 sdy 65:128 active ready running
> 
> The prio=14 means that this controller does all the work.   The packets use
> sdg,sdm,sdp,sdw in a round-robin fashion.    If the first controller fails,
> then the prio=9 controller handles the work.
> 
> The storage_server_connections table only has 2 rows corresponding to sdg
> and sdd, because I setup the iscsi connection with only those nics
> originally.   If I had created the storage domain with all 8 nics, then I
> would have 8 rows in storage_server_connections.   Adding nics later, with
> iscsi discovery, should add those extra rows into
> storage_server_connections.  Just my opinion here.
> 
> Thanks,
> Paul

Comment 13 pmdyermms 2016-03-08 00:32:30 UTC
>The term bond referred to in comment 4 did not refer to network bond, but to >'ISCSI bond' - which is a poor terminology to describe TWO (or more) outbound >interfaces working to the target(s).
>(Without it, you have a single point of failure on your host side, for example).

Actually, I do not have a single point of failure.   The server has 4 nics dedicated to reaching the storage device:
	
  em3  data30  10.251.6.136  90:b1:1c:fc:d2:f4
  em4  data31  10.251.7.136  90:b1:1c:fc:d2:f6
  p3p1 data32  10.251.8.136  00:15:17:3a:a1:6c
  p3p2 data33  10.251.9.136  00:15:17:3a:a1:6d

I am not using nic bonding in any fashion.   I have 4 iSCSI logins to the storage.  Multipath works with the Dell rdac device driver to implement round-robin across the multiple nics:

# multipath -l
36f01faf000d7de06000002343a6fcf68 dm-0 DELL,MD32xxi
size=2.2T features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=0 status=active
| |- 8:0:0:2  sdl  8:176  active undef running
| |- 6:0:0:2  sdo  8:224  active undef running
| |- 7:0:0:2  sdw  65:96  active undef running
| `- 12:0:0:2 sdaf 65:240 active undef running

The issue is that when I originally setup this storage domain in rhev/ovirt, only 2 nics were available.   After installing more nics, I needed to manually login to the extra iscsi targets, because rhev/ovirt has no knowledge of the new targets.   I run the "iscsiadm -n node -L all" command after a reboot and the "activate" host has completed.

It would be great if the postgresdb was aware of the new targets and did the login automatically, just as it does for the original 2 targets.   Even better would be to be able to add new iSCSI targets to an existing storage domain.

Comment 14 Maor 2016-04-07 11:44:34 UTC
Hi pmdyermms,

I think that the iSCSI bond feature should solve your problem regarding the manually login.
I simulate your issue in my env, and added some print screens demonstrating how the added new nic should be managed.

Regarding the DB, there is another table called "iscsi_bonds_networks_map" which should manage all the added nics.

Please let me know if this helps you.

Regards,
Maor

Comment 15 Maor 2016-04-07 11:48:27 UTC
Created attachment 1144690 [details]
add a new nic with iscsi bond

Comment 16 Maor 2016-04-07 11:54:01 UTC
BTW, if you want to check if the host has succeeded to connect to the iSCSI with the right interface you can also take a look at the vdsm log

This is the iscsi command which connects with ens7 network interface:

jsonrpc.Executor/7::DEBUG::2016-04-07 14:25:57,746::iscsiadm::117::Storage.Misc.excCmd::(_runCmd) /bin/taskset --cpu-list 0-1 /bin/sudo -n /sbin/iscsiadm -m node -T iqn.2015-07.com.mlipchuk3
.redhat:444 -I ens7 -p 10.35.16.55:3260,1 --op=new (cwd None)
jsonrpc.Executor/7::DEBUG::2016-04-07 14:25:57,763::iscsiadm::117::Storage.Misc.excCmd::(_runCmd) SUCCESS: <err> = ''; <rc> = 0
jsonrpc.Executor/7::DEBUG::2016-04-07 14:25:57,765::iscsiadm::117::Storage.Misc.excCmd::(_runCmd) /bin/taskset --cpu-list 0-1 /bin/sudo -n /sbin/iscsiadm -m node -T iqn.2015-07.com.mlipchuk3
.redhat:444 -I ens7 -p 10.35.16.55:3260,1 -l (cwd None)
jsonrpc.Executor/3::DEBUG::2016-04-07 14:25:57,774::__init__::511::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {'storagepoolID': 'bbcb48b2-3795-4
377-84ae-d01bc6bfb15a'}


and this is the iscsi command which connects with ens9 network interface:

jsonrpc.Executor/7::DEBUG::2016-04-07 14:25:58,716::iscsiadm::117::Storage.Misc.excCmd::(_runCmd) /bin/taskset --cpu-list 0-1 /bin/sudo -n /sbin/iscsiadm -m iface -I ens9 --op=new (cwd None)
jsonrpc.Executor/7::DEBUG::2016-04-07 14:25:58,730::iscsiadm::117::Storage.Misc.excCmd::(_runCmd) SUCCESS: <err> = ''; <rc> = 0
jsonrpc.Executor/7::DEBUG::2016-04-07 14:25:58,731::iscsiadm::117::Storage.Misc.excCmd::(_runCmd) /bin/taskset --cpu-list 0-1 /bin/sudo -n /sbin/iscsiadm -m iface -I ens9 -n iface.net_ifacen
ame -v network2 --op=update (cwd None)
jsonrpc.Executor/7::DEBUG::2016-04-07 14:25:58,745::iscsiadm::117::Storage.Misc.excCmd::(_runCmd) SUCCESS: <err> = ''; <rc> = 0
jsonrpc.Executor/7::DEBUG::2016-04-07 14:25:58,746::iscsiadm::117::Storage.Misc.excCmd::(_runCmd) /bin/taskset --cpu-list 0-1 /bin/sudo -n /sbin/iscsiadm -m iface -I ens9 -n iface.transport_
name -v tcp --op=update (cwd None)
jsonrpc.Executor/7::DEBUG::2016-04-07 14:25:58,779::iscsiadm::117::Storage.Misc.excCmd::(_runCmd) SUCCESS: <err> = ''; <rc> = 0
jsonrpc.Executor/7::DEBUG::2016-04-07 14:25:58,779::iscsiadm::117::Storage.Misc.excCmd::(_runCmd) /bin/taskset --cpu-list 0-1 /bin/sudo -n /sbin/iscsiadm -m node -T iqn.2015-07.com.mlipchuk3
.redhat:444 -I ens9 -p 10.35.16.55:3260,1 --op=new (cwd None)

Comment 17 pmdyermms 2016-05-02 22:46:10 UTC
Using the bonding setup looks promising, but it is not appropriate to my situation.    Bonding requires a few things that do not apply here.   

The network switch would need config changes to recognized MAC addresses for the different nics to be recognized as one.   This is not possible without a smart switch, which is not available.

The storage device being used is a Dell Powervault.   The vendor provides a driver that has all setup instructions for iscsi round-robin scheduling.   I spoke with the hardware support.   They say it is not intended to be used in a bonding setup.   He walked me through the setup for iscsi round-robin.   Indeed, it works well with this setup.

March 2015, I attempted to add an after_network_setup script in the "Host Hooks" tab.   I was not able to get it working.   Here is the script:

$ cat /usr/libexec/vdsm/hooks/after_network_setup/add-iscsi-paths 
#!/bin/bash
# logon to iscsi targets that vdsmd is not maintaining

/sbin/iscsiadm  -m node -L all
exit 0

I will attach a screenshot of the manager gui showing that it is attached.  The host hook did not work.   I was not able to debug the reason.

Paul

Comment 18 pmdyermms 2016-05-02 22:47:19 UTC
Created attachment 1153046 [details]
host hooks event script

Comment 19 Yaniv Kaul 2016-05-03 04:44:18 UTC
(In reply to pmdyermms from comment #17)
> Using the bonding setup looks promising, but it is not appropriate to my
> situation.    Bonding requires a few things that do not apply here.   
> 
> The network switch would need config changes to recognized MAC addresses for
> the different nics to be recognized as one.   This is not possible without a
> smart switch, which is not available.

We are misusing the term bond here - it's not really a bond, but real multipathing - multiple ongoing interfaces are being used for multiple paths to the storage.

Comment 20 Yaniv Lavi 2016-12-06 09:41:00 UTC
Did you resolve the issue with ISCSI bonding feature? Can this RFE be closed?

Comment 21 pmdyermms 2016-12-06 16:13:10 UTC
yes, this was resolved.  

I did not use the bonding feature of ovirt because I did not know the term bond was meant to mean multipath.

With ovirt 4.0, the hosts were reloaded with RHEL 7, and the rhev/ovirt was reinstalled.  At that point, the "network" table got all the rows relating to the 6 nics.   On reboot, all these nics get activated with iSCSI and become a multipath to the storage.    I will attach a picture of the table contents.

Thanks.

Comment 22 pmdyermms 2016-12-06 16:14:20 UTC
Created attachment 1228619 [details]
network table with 6 nics included


Note You need to log in before you can comment on or make changes to this bug.