Bug 1094144

Summary: [engine-backend] [iSCSI multipath] No indication that updating an iSCSI multipath bond doesn't trigger any operation from vdsm side
Product: Red Hat Enterprise Virtualization Manager Reporter: Elad <ebenahar>
Component: ovirt-engineAssignee: Maor <mlipchuk>
Status: CLOSED CURRENTRELEASE QA Contact: Elad <ebenahar>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.4.0CC: amureini, gklein, iheim, lpeer, rbalakri, Rhev-m-bugs, scohen, sgotliv, tnisan, yeylon
Target Milestone: ---Keywords: ZStream
Target Release: 3.5.0Flags: amureini: Triaged+
Hardware: x86_64   
OS: Unspecified   
Whiteboard: storage
Fixed In Version: ovirt-3.5.0_rc1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1126428 (view as bug list) Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1126428, 1142923, 1156165    
Attachments:
Description Flags
logs from engine and host none

Description Elad 2014-05-05 07:10:26 UTC
Created attachment 892439 [details]
logs from engine and host

Description of problem:
When updating an iSCSI multipath bond, engine doesn't send any command to vdsm regarding the update. For example, when removing a network from an iscsi bond, engine doesn't order the hosts in the DC to disconnect one of the paths leading to the storage servers. vdsm still sees the targets via a network which was supposed to be removed from the iscsi multipath bond.

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
On a shared DC with 1 active iSCSI storage domain:
1. Create 2 new networks and attach them to the cluster with required check-box checked
2. Attach the networks to the cluster's hosts NICs 
3. Create a new iSCSI multipath bond (under DC tab -> pick the relevant DC -> iSCSI multipath sub-tab -> new) and add the new networks along with the targets to it.
4. remove one of the networks from the bond


Actual results:
Engine doesn't order the hosts in the DC to update their connections to the storage servers. In order for the changes to take effect, the hosts have to be disconnected from all of their paths to the storage servers by a manual intervention of maintenance and activate to the hosts or the storage domains.

Edit iSCSI bond as appears in engine.log:

2014-05-05 09:58:06,894 INFO  [org.ovirt.engine.core.bll.storage.EditIscsiBondCommand] (ajp-/127.0.0.1:8702-15) [3f8ec4e5] Running command: EditIscsiBondCommand internal: false. Entities affected :  ID: 00000002-0002-0002-0002-00000000030f Type: StoragePool
2014-05-05 09:58:06,910 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-15) [3f8ec4e5] Correlation ID: 3f8ec4e5, Call Stack: null, Custom Event ID: -1, Message: iSCSI bond 'iscsi' was successfully updated.

After the edit, when running 'iscsiadm -m session' on the host, the sessions between the host to the storage servers remain as they were before the update

Expected results:
When updating an existing iscsi multipath bond, the changes have to be sent to the hosts automatically, without the need to disconnect all the paths leading to the storage servers 

Additional info: logs from engine and host

Comment 1 Elad 2014-05-05 07:11:37 UTC
Version-Release number of selected component (if applicable):
AV7
rhevm-3.4.0-0.15.beta3.el6ev.noarch

Comment 2 Sergey Gotliv 2014-05-05 09:22:27 UTC
We discussed this limitation during development cycle, currently RHEV can't disconnect from the target. It will happen automatically after next VDSM restart.
Meantime there is no harm to the system.

Comment 3 Allon Mureinik 2014-05-11 22:13:28 UTC
(In reply to Sergey Gotliv from comment #2)
> We discussed this limitation during development cycle, currently RHEV can't
> disconnect from the target. It will happen automatically after next VDSM
> restart.
> Meantime there is no harm to the system.

Perhaps we should make this issue more visible?
Perhaps an audit log?

Comment 5 Maor 2014-08-03 12:40:14 UTC
Allon, how about the following audit log:

"The following Networks has been removed from the iSCSI bond ${IscsiBondName}: ${NetworkNames}. For the changes to take affect on the hosts, they should be moved to maintenance and get activated again."

Comment 6 Allon Mureinik 2014-08-03 12:57:10 UTC
Minor correction:

"The following Networks has been removed from the iSCSI bond ${IscsiBondName}: ${NetworkNames}. For these changes to take affect on the hosts, they should be moved to maintenance and then activated again."

Comment 7 Eyal Edri 2014-08-04 11:09:11 UTC
these bugs are candidates for z-stream, but not ready yet.
they were not included in 3.4.2 bug tracker [1] for critical bugs by gss,
and out of of scope for the 3.4.2 build.
moving to 3.4.3.


[1] https://bugzilla.redhat.com/show_bug.cgi?id=1123858

Comment 9 Elad 2014-08-10 15:03:14 UTC
When updating an iSCSI bond, the following message is shown to user in audit log:

2014-08-10 17:59:55,955 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [79a5a61a] Correlation ID: 79a5a61a, Call Stack: null, Custom Event ID: -1, Message: The following Networks has been removed from the iSCSI bond 1: s1. for those changes to take affect, the hosts must be moved to maintenance and activated again

Verified with ovirt-3.5 RC1

Comment 10 Allon Mureinik 2015-02-16 19:12:31 UTC
RHEV-M 3.5.0 has been released, closing this bug.

Comment 11 Allon Mureinik 2015-02-16 19:12:33 UTC
RHEV-M 3.5.0 has been released, closing this bug.