Bug 1373640 (RHV_HE_on_iscsi_multipath) - [RFE] [downstream clone] [hosted-engine] [iSCSI multipath] Support hosted engine deployment based on multiple iSCSI initiators
Summary: [RFE] [downstream clone] [hosted-engine] [iSCSI multipath] Support hosted eng...
Keywords:
Status: CLOSED WORKSFORME
Alias: RHV_HE_on_iscsi_multipath
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-hosted-engine-setup
Version: 3.6.8
Hardware: x86_64
OS: Unspecified
high
high
Target Milestone: ovirt-4.2.6
: ---
Assignee: Simone Tiraboschi
QA Contact: Nikolai Sednev
URL:
Whiteboard:
Depends On: 1193961
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-09-06 20:35 UTC by Marina Kalinin
Modified: 2020-09-07 07:32 UTC (History)
26 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Honor iSCSI bonds as configured by the user on engine side.
Clone Of: 1193961
Environment:
Last Closed: 2018-08-09 10:47:44 UTC
oVirt Team: Integration
Target Upstream Version:
Embargoed:
mkalinin: Triaged+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1613763 0 high CLOSED [BLOCKED][RFE] - Allow ISCSI bonding in hosted engine setup 2022-04-20 16:08:39 UTC
Red Hat Knowledge Base (Solution) 2138951 0 None None None 2016-09-06 20:35:21 UTC

Internal Links: 1613763

Comment 1 Sandro Bonazzola 2018-07-16 13:59:11 UTC
Re-targeting to 4.2.6 being next build blockers only and this not being considered a blocker for 4.2.5.

Comment 2 Sandro Bonazzola 2018-07-17 06:30:12 UTC
Change is already in

Comment 4 Nikolai Sednev 2018-07-31 15:59:12 UTC
I've tried to separate iSCSI traffic over iSCSI bond and got all interfaces participating in iSCSI sessions. Instead of taking all iSCSI traffic over enp3s0f1, traffic was distributed over default interface enp3s0f0 and enp3s0f1 in this pattern:

alma04 ~]# iscsiadm -m session -P1
Target: iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f6c01 (non-flash)
        Current Portal: 10.35.146.161:3260,1
        Persistent Portal: 10.35.146.161:3260,1
                **********
                Interface:
                **********
                Iface Name: default
                Iface Transport: tcp
                Iface Initiatorname: iqn.1994-05.com.redhat:20179eabd3d
                Iface IPaddress: 10.35.92.4
                Iface HWaddress: <empty>
                Iface Netdev: <empty>
                SID: 1
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
Target: iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f6c00 (non-flash)
        Current Portal: 10.35.146.129:3260,1
        Persistent Portal: 10.35.146.129:3260,1
                **********
                Interface:
                **********
                Iface Name: default
                Iface Transport: tcp
                Iface Initiatorname: iqn.1994-05.com.redhat:20179eabd3d
                Iface IPaddress: 10.35.92.4
                Iface HWaddress: <empty>
                Iface Netdev: <empty>
                SID: 2
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
Target: iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f6c04 (non-flash)
        Current Portal: 10.35.146.193:3260,1
        Persistent Portal: 10.35.146.193:3260,1
                **********
                Interface:
                **********
                Iface Name: default
                Iface Transport: tcp
                Iface Initiatorname: iqn.1994-05.com.redhat:20179eabd3d
                Iface IPaddress: 10.35.92.4
                Iface HWaddress: <empty>
                Iface Netdev: <empty>
                SID: 3
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
Target: iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f6c05 (non-flash)
        Current Portal: 10.35.146.225:3260,1
        Persistent Portal: 10.35.146.225:3260,1
                **********
                Interface:
                **********
                Iface Name: enp3s0f1
                Iface Transport: tcp
                Iface Initiatorname: iqn.1994-05.com.redhat:20179eabd3d
                Iface IPaddress: 10.35.71.93
                Iface HWaddress: <empty>
                Iface Netdev: enp3s0f1
                SID: 4
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE

I can't verify this bug, due to the fact that its still impossible to separate iSCSI traffic from flowing over management interface.
Tested on these components on hosts:
rhvm-appliance-4.2-20180727.1.el7.noarch, 
Engine setup within the rhvm-appliance-4.2-20180727.1.el7.noarch is ovirt-engine-setup-4.2.5.2-0.1.el7ev.noarch.
ovirt-hosted-engine-ha-2.2.16-1.el7ev.noarch
ovirt-hosted-engine-setup-2.2.24-1.el7ev.noarch
rhvm-appliance-4.2-20180727.1.el7.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
Linux 3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.5 (Maipo)

-------------
Summary:
-------------
1.At present, there is no way to completely separate HE's storage traffic between default management network and storage traffic dedicated iSCSI bonded networks.
 
2.There is no way to detach default management network interface from HE storage target without loosing HE-VM on hosts.

3.There is no way to choose during HE's deployment stage, which multiple interfaces should be used for MPIO (aka iSCSI Bond).

4.There is no prediction on which of the networks "iscsi dedicated network" or the default management network, the HE's storage traffic will be forwarded,once there were created separate storage dedicated networks for HE's storage domain iSCSI target.

Due to the fact, that this RFE does not cover the customer requirements, I'm moving it back to assigned.

Comment 6 Simone Tiraboschi 2018-07-31 22:19:52 UTC
According to vdsm logs, ovirt-ha-agent sent:

2018-08-01 01:09:08,345+0300 INFO  (jsonrpc/0) [vdsm.api] START connectStorageServer(domType=3, spUUID=u'00000000-0000-0000-0000-000000000000', conList=[{u'netIfaceName': u'enp3s0f1', u'id': u'59ead077-fcbc-4cd5-85f5-ddf2a88c7f6c', u'connection': u'10.35.146.225', u'iqn': u'iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f6c05', u'user': u'', u'tpgt': u'1', u'ifaceName': u'enp3s0f1', u'password': '********', u'port': u'3260'}], options=None) from=::1,49818, task_id=2c39a45f-8f8e-4f86-8ef5-b757634e56b0 (api:46)
2018-08-01 01:09:09,666+0300 INFO  (jsonrpc/0) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'status': 0, 'id': u'59ead077-fcbc-4cd5-85f5-ddf2a88c7f6c'}]} from=::1,49818, task_id=2c39a45f-8f8e-4f86-8ef5-b757634e56b0 (api:52)


and the default interface is not in the connection list.

The connectStorageServer are not that different from what the engine sent.

2018-07-31 17:43:33,035+0300 INFO  (jsonrpc/0) [vdsm.api] START connectStorageServer(domType=3, spUUID=u'ff687f1e-93ea-11e8-a9b0-00163e7bb853', conList=[{u'netIfaceName': u'enp3s0f1', u'id': u'4b132b21-bad7-45ff-9006-0b3a69ac0bd4', u'connection': u'10.35.146.225', u'iqn': u'iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f6c05', u'user': u'', u'tpgt': u'1', u'ifaceName': u'enp3s0f1', u'password': '********', u'port': u'3260'}], options=None) from=::ffff:10.35.92.51,37436, flow_id=438f4f82, task_id=e65f0604-d2fc-4440-87f9-e7e6bde45097 (api:46)
2018-07-31 17:43:34,823+0300 INFO  (jsonrpc/0) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'status': 0, 'id': u'4b132b21-bad7-45ff-9006-0b3a69ac0bd4'}]} from=::ffff:10.35.92.51,37436, flow_id=438f4f82, task_id=e65f0604-d2fc-4440-87f9-e7e6bde45097 (api:52)

Comment 7 Simone Tiraboschi 2018-07-31 22:33:29 UTC
While the host is in maintenance mode a sequence of
  iscsiadm -m node -u
  iscsiadm -m node -I default --op=delete

brings to the desidered result and ovirt-ha-broker will simply use the iSCSI bonding as configured by the engine with expected results.

I fear that the issue is simply that the engine/vdsm never calls (or maybe it just fails because the ovirt-ha-broker keeps the iscsi sessions up?)
  iscsiadm -m node -I default --op=delete
re-configuring the iSCSI bonding on the host and so all the bindings between iSCSI interfaces and portals used in the past will remain on the host.

Comment 8 Simone Tiraboschi 2018-08-09 10:47:44 UTC
Let's close this with current state and let's add another patch to automatically configure iscsi bonding from setup as for 1613763

Comment 9 Ai90iV 2020-08-20 20:49:55 UTC
Dear Simone,

Do I understand correctly sessions default interface is legacy from initial configuration? I have situation with multiple path's duplicated originated from default interface. Can I remove them without impact, just leaving my 2 NICs in Multipath Bonding, right? they do not carry any service traffic or etc?

Comment 10 Simone Tiraboschi 2020-08-28 12:35:44 UTC
Yes, you should be able to do that.

Comment 11 Ai90iV 2020-09-02 05:16:56 UTC
Ok, clear. Thank you, Simone!

Also one question.

Is it mandatory to have both iSCSI Bonding members in same L3 network and same broadcast domain? 

I noticed that oVirt creates the file "vdsm.conf" under
/etc/sysctl.d, with these contents

net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.default.arp_announce = 2

This indirectly gives me a sign that this is done for purpose of having IP addresses from same L3 subnet on iSCSI Bonding member interfaces. 

But can I join interfaces to iSCSI Bonding, which are members of different L3 subnets and having different broadcast domains? For example eth0 with 192.168.0.0/24 and eth1 with 192.168.1.0/24, connected to different switches. 

I do not find in RH and oVirt documentation straightforward answer if members of iSCSI Bonding can be splitted across different L3 subnets, not routed between each other.

I would be very grateful for clarification.

Comment 12 Nikolai Sednev 2020-09-06 04:45:33 UTC
AFAIK we're not limiting this to the same L3 or L2 network. 

*
arp_ignore value set to 1 so that it will only reply to ARP requests on the interface which contains the target IP address. This acts much more like a traditional router and provides simplicity in troubleshooting and operation.

**
By setting the arp_announce parameter to 2, Linux uses the best local address for each ARP request, preferring primary addresses on the interface used to send the ARP. This most closely matches traditional router ARP request behavior.If no suitable local address is found we select the first local address we have on the outgoing interface or on all other interfaces, with the hope we will receive reply for our request and even sometimes no matter the source IP address we announce.

In general ARPs are working over L2.

I see no issue using different unrouted L3 networks inside separate L2 domains, as long as they can reach all relevant iSCSI storage targets just like other source NICs do.

https://www.ovirt.org/develop/release-management/features/storage/iscsi-multipath.html

"The User selects logical networks and iSCSI targets. The iSCSI bond must contain at least one logical network related to the Data Center."


Note You need to log in before you can comment on or make changes to this bug.