Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1053900

Summary: [RFE] Hypervisor RHEV-H only connects to the Dell Equallogic using one connection. it is expected to do multipathing and have 4 connection to it.
Product: Red Hat Enterprise Virtualization Manager Reporter: Sean Cohen <scohen>
Component: RFEsAssignee: Sergey Gotliv <sgotliv>
Status: CLOSED DUPLICATE QA Contact: Leonid Natapov <lnatapov>
Severity: high Docs Contact:
Priority: unspecified    
Version: unspecifiedCC: abaron, acanan, acathrow, baptiste.agasse, charles_rose, Donald_Williams, dossow, gklein, gouyang, hadong, howey.vernon, iheim, jboggs, jkt, leiwang, linux-bugs, lpeer, lyarwood, mgoldboi, mkalinin, nlevinki, ovirt-bugs, ovirt-maint, sebastian_antunez, s.kong, ycui, yeylon
Target Milestone: ---Keywords: FutureFeature, Triaged
Target Release: 3.4.0   
Hardware: x86_64   
OS: Linux   
Whiteboard: storage
Fixed In Version: Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: 953343 Environment:
Last Closed: 2014-01-27 07:48:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 953343    
Bug Blocks: 1078909, 1142926    

Description Sean Cohen 2014-01-15 23:10:42 UTC 
+++ This bug was initially created as a clone of Bug #953343 +++

Description of problem:

Hypervisor RHEV-H only connects to the Dell Equallogic using one connection. it is expected to do multipathing and have 4 connection to it.


Version-Release number of selected component (if applicable):
rhevh-6.4-20120318.1.el6_4


How reproducible:


Steps to Reproduce:
1. install rhev hypervisor
2. connect to rhev-m
3. configure iscsi storage
  
Actual results:

hypervisor only have one connection to Equallogic (i.e not using multipath)

Expected results:

suppose to have 4 connection using multipathing to the Dell Equallogic

Additional info:

=======================
the way that this was "fixed" (probably not the optimal config, but it works)

configure from console (F2) of the Hypervisor

-----------------------------------------------------------------------
[root@vh-7 network-scripts]# vi /var/lib/iscsi/ifaces/iSCSI:1
# BEGIN RECORD 6.2.0-873.2.el6
iface.iscsi_ifacename = iSCSI:1
iface.net_ifacename = iSCSI
iface.transport_name = tcp
iface.vlan_id = 0
iface.vlan_priority = 0
iface.iface_num = 0
iface.mtu = 0
iface.port = 0
# END RECORD

-----------------------------------------------------------------------
[root@vh-7 network-scripts]# vi /var/lib/iscsi/ifaces/iSCSI:2
# BEGIN RECORD 6.2.0-873.2.el6
iface.iscsi_ifacename = iSCSI:2
iface.net_ifacename = iSCSI
iface.transport_name = tcp
iface.vlan_id = 0
iface.vlan_priority = 0
iface.iface_num = 0
iface.mtu = 0
iface.port = 0
# END RECORD

-----------------------------------------------------------------------
[root@vh-7 admin]# vi /var/lib/iscsi/ifaces/iSCSI:3
# BEGIN RECORD 6.2.0-873.2.el6
iface.iscsi_ifacename = iSCSI:3
iface.net_ifacename = iSCSI
iface.transport_name = tcp
iface.vlan_id = 0
iface.vlan_priority = 0
iface.iface_num = 0
iface.mtu = 0
iface.port = 0
# END RECORD

-----------------------------------------------------------------------
# Add the folowing under section "devices {}"
[root@vh-7 ~]# vi /etc/multipath.conf
device {
	vendor			"EQLOGIC"
	product			"100E-00"
	path_grouping_policy	multibus
	getuid_callout          "/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/%n"
	features		"1 queue_if_no_path"
	path_checker		readsector0
	path_selector		"round-robin 0"
	failback		"immediate"
	rr_min_io		10
	rr_weight		priorities
}

-----------------------------------------------------------------------
vi /etc/mount.sh
#!/bin/sh

sleep 2m

iscsiadm -m discovery -t st -p 192.168.XXX.XXX:3260

iscsiadm -m node -l -T iqn.2001-05.com.equallogic:0-XXXXX-XXXX-XXXXXXXa-stor -p 192.168.XXX.XXX:3260

-----------------------------------------------------------------------
vi /etc/rc.local
nohup sh /etc/mount.sh > /dev/null &


-----------------------------------------------------------------------
vi /config/files
/etc/rc.local
/etc/mount.sh
/etc/multipath.conf
/var/lib/iscsi/ifaces/iSCSI:1
/var/lib/iscsi/ifaces/iSCSI:2
/var/lib/iscsi/ifaces/iSCSI:3

-----------------------------------------------------------------------
chmod +x /etc/mount.sh
cp /etc/rc.local /config/etc/rc.local
cp /etc/mount.sh /config/etc/mount.sh
cp /etc/multipath.conf /config/etc/multipath.conf
cp /var/lib/iscsi/ifaces/iSCSI:1 /config/var/lib/iscsi/ifaces/iSCSI:1
cp /var/lib/iscsi/ifaces/iSCSI:2 /config/var/lib/iscsi/ifaces/iSCSI:2
cp /var/lib/iscsi/ifaces/iSCSI:3 /config/var/lib/iscsi/ifaces/iSCSI:3

--- Additional comment from Mike Burns on 2013-04-18 09:04:11 EDT ---

We do have RFE's filed (don't have the numbers handy) for allowing changes to multipath.conf from the TUI.  

How does RHEL/Fedora work?  Do you have to write those files manually?

This seems more like something that needs to be configured in the oVirt Engine/RHEV-M configuration and pushed down to all hypervisors, rather than in ovirt-node/RHEV-H directly.

--- Additional comment from Simon Kong Win Chang on 2013-04-18 19:13:10 EDT ---

i configured the storage on the manager first, then the hypervisor was only have one connection to the Dell EqualLogic.

i then did the modification above to have it have 4 connection to the Equallogic.

yes the config should be from the manager, hence why i submitted this. also to help anyone with an equallogic make their setup work while waiting for this feature to be implemented to the manager.

also, the disadvantage now, is, every time i am to upgrade the hypervisor, i will need to manually go in and change those config again.

we are running the "Red Hat Enterprise Virtualization Manager" but i don't have access to submit a bug in that section, hence why i submitted it in the ovirt section.

--- Additional comment from Mike Burns on 2013-04-25 11:17:20 EDT ---

Moving this to ovirt-engine for triage.

As for issues with RHEV-M, you should file a support ticket with RH Support.  They can get a bug filed for you.

--- Additional comment from Simon Kong Win Chang on 2013-04-25 19:26:05 EDT ---

i have, case number 00811800

--- Additional comment from Simon Kong Win Chang on 2013-05-08 20:07:22 EDT ---

p.s upgrading the hypervisor using the RHEV-Manager did not wipe the config. still worked fine after the upgrade.

--- Additional comment from Simon Kong Win Chang on 2013-05-08 20:12:17 EDT ---

added the 2 mkdir line in this section


chmod +x /etc/mount.sh
cp /etc/rc.local /config/etc/rc.local
cp /etc/mount.sh /config/etc/mount.sh
cp /etc/multipath.conf /config/etc/multipath.conf
==mkdir /config/var/lib/iscsi==
==mkdir /config/var/lib/iscsi/ifaces==
cp /var/lib/iscsi/ifaces/iSCSI:1 /config/var/lib/iscsi/ifaces/iSCSI:1
cp /var/lib/iscsi/ifaces/iSCSI:2 /config/var/lib/iscsi/ifaces/iSCSI:2
cp /var/lib/iscsi/ifaces/iSCSI:3 /config/var/lib/iscsi/ifaces/iSCSI:3

--- Additional comment from Sebastian Antunez on 2013-06-23 13:53:14 EDT ---

Hello

I have installed RHEV 3.1 connected to an Equallogic storage, but only a connection is active. When configuring RHEV as the standard process under Redhat problem persists.

Is there any additional procedure to have 2 visible and not one path.

Regards

--- Additional comment from Simon Kong Win Chang on 2013-06-24 21:39:41 EDT ---

the workaround posted above is to add 3 path, therefore a total of 4.

if you only need two path, create all the above except for /var/lib/iscsi/ifaces/iSCSI:2 and /var/lib/iscsi/ifaces/iSCSI:3

--- Additional comment from Donald Williams on 2013-07-01 22:31:00 EDT ---

One thing I didn't see is the contents of the /etc/sysctl.conf file. 

When you have multiple NICs on the same subnet you need to add the following to the /etc/sysctl.conf file. 

As a test you should be able to ping -I <iscsi eth ports> <EQL Group IP>  Try all the ports. Without the settings below, only one interface will work. 

# EQL entries
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.all.rp_filter = 2

Optionally I have found the following helpful with Oracle OVMS and SQL servers. 
Also added to the /etc/sysctl.conf

# Increase network buffer sizes
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 8192 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.wmem_default = 262144
net.core.rmem_default = 262144

Run sysctl -p afterwards and try the ping again.  If that works then re-do the Discovery and login.

--- Additional comment from Donald Williams on 2013-07-01 22:37:41 EDT ---

Something else I just noticed, you ifaces iscsi_ifacenanme & net_ifacename aren't the same. I've never seen that work. Though I'm used to RHEL or Oracle Linux, ubuntu, etc... 

iface.iscsi_ifacename = iSCSI:3
iface.net_ifacename = iSCSI

On a standard config they are: 

# cat /etc/iscsi/ifaces/eth0
# BEGIN RECORD 2.0-873
iface.iscsi_ifacename = eth0
iface.net_ifacename = eth0
iface.transport_name = tcp
iface.vlan_id = 0
iface.vlan_priority = 0
iface.iface_num = 0
iface.mtu = 0
iface.port = 0
# END RECORD

--- Additional comment from Simon Kong Win Chang on 2013-07-02 01:31:04 EDT ---

Sorry, i should have given a bit more network configuration information with that.


the "iSCSI" is actually the name of the network interface which is configured with an ip address. the iSCSI interface is actually a bridge interface build on top of a LACP bond interface, which in turn is build on top of the physical interface. the config might make is less confusing.

emX interface are the physical interface.

bondX.X are the LACP bonding interface. we are actually bonding 4 physical interface.

iSCSI interface is actually the bridge interface. it could have been labled as br0.130 instead. if this was the case, then the config files would have been

# cat /etc/iscsi/ifaces/br0.130:1
# cat /etc/iscsi/ifaces/br0.130:2
# cat /etc/iscsi/ifaces/br0.130:3

the ":X" after the interface are usually use to configure alias interface in the network config. although in this case, no alias interface was configured in /etc/sysconfig/network-scripts/


=============network config files==============

# cat /etc/sysconfig/network-scripts/ifcfg-em1
DEVICE=em1
ONBOOT=yes
HWADDR=XX:XX:XX:XX:XX:XX
MASTER=bond0
SLAVE=yes
MTU=9000
NM_CONTROLLED=no
STP=no
PEERDNS=no

# cat /etc/sysconfig/network-scripts/ifcfg-em2
DEVICE=em2
ONBOOT=yes
HWADDR=XX:XX:XX:XX:XX:XX
MASTER=bond0
SLAVE=yes
MTU=9000
NM_CONTROLLED=no
STP=no
PEERDNS=no

# cat /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
ONBOOT=yes
BONDING_OPTS=mode=4
MTU=9000
NM_CONTROLLED=no
STP=no
PEERDNS=no

# cat /etc/sysconfig/network-scripts/ifcfg-bond0.130
DEVICE=bond0.130
ONBOOT=yes
VLAN=yes
BRIDGE=iSCSI
MTU=9000
NM_CONTROLLED=no
STP=no
PEERDNS=no

# cat /etc/sysconfig/network-scripts/ifcfg-iSCSI
DEVICE=iSCSI
ONBOOT=yes
TYPE=Bridge
DELAY=0
IPADDR=192.168.XXX.XXX
NETMASK=255.255.255.0
BOOTPROTO=none
MTU=9000
NM_CONTROLLED=no
STP=no
PEERDNS=no

====================================================
p.s ping works fine

a small extract of the output of
# iscsiadm -m session -P3

Target: iqn.2001-05.com.equallogic:0-XXXX
        Current Portal: 192.168.XXX.201:3260,1
        Persistent Portal: 192.168.XXX.200:3260,1
                **********
                Interface:
                **********
                Iface Name: default
                Iface Transport: tcp
                Iface Initiatorname: iqn.1994-05.com.XXXX
                Iface IPaddress: 192.168.XXX.148
                Iface HWaddress: <empty>
                Iface Netdev: <empty>
                SID: 1
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
.
.
.
       Current Portal: 192.168.XXX.202:3260,1
        Persistent Portal: 192.168.XXX.200:3260,1
                **********
                Interface:
                **********
                Iface Name: iSCSI:1
                Iface Transport: tcp
                Iface Initiatorname: iqn.1994-05.com.XXXX
                Iface IPaddress: 192.168.XXX.148 
                Iface HWaddress: <empty> 
                Iface Netdev: iSCSI
                SID: 2
.
.
.
        Current Portal: 192.168.XXX.203:3260,1
        Persistent Portal: 192.168.XXX.200:3260,1
                **********
                Interface:
                **********
                Iface Name: iSCSI:2
                Iface Transport: tcp
                Iface Initiatorname: iqn.1994-05.com.XXXX
                Iface IPaddress: 192.168.XXX.148
                Iface HWaddress: <empty>
                Iface Netdev: iSCSI
                SID: 3
.
.
.
        Current Portal: 192.168.XXX.204:3260,1
        Persistent Portal: 192.168.XXX.200:3260,1
                **********
                Interface:
                **********
                Iface Name: iSCSI:3
                Iface Transport: tcp
                Iface Initiatorname: iqn.1994-05.com.XXXX
                Iface IPaddress: 192.168.XXX.148
                Iface HWaddress: <empty>
                Iface Netdev: iSCSI
                SID: 4
.
.
.


===============
statistics info from the equallogic confirms that there is data transfer between all the interface.

hope this helps.

--- Additional comment from Simon Kong Win Chang on 2013-07-02 01:33:19 EDT ---

just to make is less confusing
the 192.168.XXX.148 is assagined to the iSCSI network interface


# cat /etc/sysconfig/network-scripts/ifcfg-iSCSI
DEVICE=iSCSI
ONBOOT=yes
TYPE=Bridge
DELAY=0
IPADDR=192.168.XXX.148
NETMASK=255.255.255.0
BOOTPROTO=none
MTU=9000
NM_CONTROLLED=no
STP=no
PEERDNS=no

--- Additional comment from Donald Williams on 2013-07-02 09:30:27 EDT ---

I have never heard of anyone tring to run MPIO over a bonded interface with an EQL array.  That does explain why ping works.

You're not generating new iSCSI sessions. EQL doesn't support bonding, so traffic over that bond isn't being spread over the available ports. One session ends up on one physical interface on the group. 

With access to each interface, open-iscsi will be able create unique sessions over those ports. Then devmapper can create multipath devices correct. 

I have two NICs definted for iSCSI.  eth0/eth1
When I discover them I get one entry for each defined interface. 

for example: 
# iscsiadm -m node
172.23.151.120:3260,1 iqn.2001-05.com.equallogic:4-52aed6-d8e3e0689-74c004a89f851c27-ubuntu-4k-vol-test
172.23.151.120:3260,1 iqn.2001-05.com.equallogic:4-52aed6-d8e3e0689-74c004a89f851c27-ubuntu-4k-vol-test

When I login, I get two logins, so now I have two iSCSI connections. 

iscsiadm -m node -T iqn.2001-05.com.equallogic:4-52aed6-d8e3e0689-74c004a89f851c27-ubuntu-4k-vol-test -l
Logging in to [iface: eth0, target: iqn.2001-05.com.equallogic:4-52aed6-d8e3e0689-74c004a89f851c27-ubuntu-4k-vol-test, portal: 172.23.151.120,3260] (multiple)
Logging in to [iface: eth1, target: iqn.2001-05.com.equallogic:4-52aed6-d8e3e0689-74c004a89f851c27-ubuntu-4k-vol-test, portal: 172.23.151.120,3260] (multiple)
Login to [iface: eth0, target: iqn.2001-05.com.equallogic:4-52aed6-d8e3e0689-74c004a89f851c27-ubuntu-4k-vol-test, portal: 172.23.151.120,3260] successful.
Login to [iface: eth1, target: iqn.2001-05.com.equallogic:4-52aed6-d8e3e0689-74c004a89f851c27-ubuntu-4k-vol-test, portal: 172.23.151.120,3260] successful.


#multipath -ll
mpath4 (364ed2a9568e0e3d8271c859fa804c074) dm-2 EQLOGIC,100E-00
size=20G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 15:0:0:0 sdd 8:48 active ready  running
  `- 16:0:0:0 sde 8:64 active ready  running

Now multipathd will round robin IO across those two iSCSI sessions. 

Compare my session output to yours. 

# iscsiadm -m session -P3
iSCSI Transport Class version 2.0-870
version 2.0-873
Target: iqn.2001-05.com.equallogic:4-52aed6-d8e3e0689-74c004a89f851c27-ubuntu-4k-vol-test
        Current Portal: 172.23.151.121:3260,1
        Persistent Portal: 172.23.151.120:3260,1
                **********
                Interface:
                **********
                Iface Name: eth0
                Iface Transport: tcp
                Iface Initiatorname: iqn.1993-08.org.debian:8ab9cf5340f0
                Iface IPaddress: 172.23.71.231
                Iface HWaddress: <empty>
                Iface Netdev: eth0
                SID: 11
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
                *********
                Timeouts:
                *********
                Recovery Timeout: 120
                Target Reset Timeout: 30
                LUN Reset Timeout: 30
                Abort Timeout: 15
                *****
                CHAP:
                *****
                username: <empty>
                password: ********
                username_in: <empty>
                password_in: ********
                ************************
                Negotiated iSCSI params:
                ************************
                HeaderDigest: None
                DataDigest: None
                MaxRecvDataSegmentLength: 262144
                MaxXmitDataSegmentLength: 65536
                FirstBurstLength: 65536
                MaxBurstLength: 262144
                ImmediateData: Yes
                InitialR2T: No
                MaxOutstandingR2T: 1
                ************************
                Attached SCSI devices:
                ************************
                Host Number: 15 State: running
                scsi15 Channel 00 Id 0 Lun: 0
                        Attached scsi disk sdd          State: running
        Current Portal: 172.23.151.122:3260,1
        Persistent Portal: 172.23.151.120:3260,1
                **********
                Interface:
                **********
                Iface Name: eth1
                Iface Transport: tcp
                Iface Initiatorname: iqn.1993-08.org.debian:8ab9cf5340f0
                Iface IPaddress: 172.23.74.186
                Iface HWaddress: <empty>
                Iface Netdev: eth1
                SID: 12
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
                *********
                Timeouts:
                *********
                Recovery Timeout: 120
                Target Reset Timeout: 30
                LUN Reset Timeout: 30
                Abort Timeout: 15
                *****
                CHAP:
                *****
                username: <empty>
                password: ********
                username_in: <empty>
                password_in: ********
                ************************
                Negotiated iSCSI params:
                ************************
                HeaderDigest: None
                DataDigest: None
                MaxRecvDataSegmentLength: 262144
                MaxXmitDataSegmentLength: 65536
                FirstBurstLength: 65536
                MaxBurstLength: 262144
                ImmediateData: Yes
                InitialR2T: No
                MaxOutstandingR2T: 1
                ************************
                Attached SCSI devices:
                ************************
                Host Number: 16 State: running
                scsi16 Channel 00 Id 0 Lun: 0
                        Attached scsi disk sde          State: running

--- Additional comment from Donald Williams on 2013-07-02 13:33:40 EDT ---

OK, I see what you did, but why do you have to create the BOND?  Why can't you use the network interfaces via their HyperVisor object names?
Comment 1 Sergey Gotliv 2014-01-25 18:52:33 UTC 
*** Bug 953343 has been marked as a duplicate of this bug. ***
Comment 2 Itamar Heim 2014-01-27 07:48:23 UTC 

*** This bug has been marked as a duplicate of bug 753541 ***