Bug 993871

Summary: [engine-backend] engine sends ConnectStorageServer to a host which is in maintenance, when detaching a local ISO domain sitting on an active host
Product: Red Hat Enterprise Virtualization Manager Reporter: Elad <ebenahar>
Component: ovirt-engineAssignee: Allon Mureinik <amureini>
Status: CLOSED CURRENTRELEASE QA Contact: Elad <ebenahar>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.3.0CC: acathrow, amureini, iheim, lpeer, Rhev-m-bugs, scohen, yeylon
Target Milestone: ---Flags: amureini: Triaged+
Target Release: 3.3.0   
Hardware: x86_64   
OS: Unspecified   
Whiteboard: storage
Fixed In Version: is12 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 974849    
Attachments:
Description Flags
logs none

Description Elad 2013-08-06 12:47:59 UTC 
Created attachment 783311 [details]
logs

Description of problem:
When detaching a local ISO domain from pool, for some reason, engine sends ConnectStorageToVdsCommand to another host (not the one that has the path of the ISO domain storage connection) which is in maintenance.


Version-Release number of selected component (if applicable):
rhevm-3.3.0-0.13.master.el6ev.noarch

How reproducible:
always

Steps to Reproduce:
1. on 2 hosts cluster with an active storage pool
2. create a local ISO domain on one of the hosts
3. put the other host in maintenance
4. maintenance the ISO domain and detach it.

Actual results:
engine sends DeactivateStorageDomain,  DisconnectStorageServerConnection, RemoveStorageDomain and DetachStorageDomainFromPool to the active host (which contains the ISO domain):



2013-08-06 15:27:56,808 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (pool-5-thread-46) [6875ec3c] Running command: DeactivateStorageDomainCommand internal: false. Entities affected :  ID: e6f7cae9-2607-4699-9
f4f-5e568df21caa Type: Storage
2013-08-06 15:27:56,814 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (pool-5-thread-46) [6875ec3c] START, DeactivateStorageDomainVDSCommand( storagePoolId = 072c2d76-8886-47ab-a1f9-d97f834115af, ign
oreFailoverLimit = false, storageDomainId = e6f7cae9-2607-4699-9f4f-5e568df21caa, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1), log id: 73a29a4c
2013-08-06 15:27:57,395 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (pool-5-thread-46) [6875ec3c] FINISH, DeactivateStorageDomainVDSCommand, log id: 73a29a4c
2013-08-06 15:27:57,395 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (pool-5-thread-46) [6875ec3c] Lock freed to object EngineLock [exclusiveLocks= key: e6f7cae9-2607-4699-9f4f-5e568df21caa value: STORAGE
, sharedLocks= ]
2013-08-06 15:27:57,398 INFO  [org.ovirt.engine.core.bll.storage.AfterDeactivateSingleAsyncOperation] (pool-5-thread-49) [6875ec3c] After deactivate treatment vds: nott-vds2,pool iscsi4
2013-08-06 15:27:57,402 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-5-thread-49) [6875ec3c] START, RefreshStoragePoolVDSCommand(HostName = nott-vds2, HostId = 5d7ce646-22c0-45e7-af68-e6120c97d94f, storagePoolId = 072c2d76-8886-47ab-a1f9-d97f834115af, masterStorageDomainId=c0e5eecc-53fc-4558-9477-0db4a6b9e873, masterVersion=1), log id: 13b4fcc1
2013-08-06 15:27:57,435 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-5-thread-49) [6875ec3c] FINISH, RefreshStoragePoolVDSCommand, log id: 13b4fcc1
2013-08-06 15:27:57,436 INFO  [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (pool-5-thread-46) Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
2013-08-06 15:27:57,443 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-5-thread-46) START, DisconnectStorageServerVDSCommand(HostName = nott-vds2, HostId = 5d7ce646-22c0-45e7-af68-e6120c97d94f, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: 96d1162f-dee2-4e61-8832-93506654fddd, connection: /opt/elad2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 315ba033
2013-08-06 15:28:00,748 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-5-thread-46) FINISH, DisconnectStorageServerVDSCommand, return: {96d1162f-dee2-4e61-8832-93506654fddd=0}, log id: 315ba033
2013-08-06 15:28:00,755 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-5-thread-46) Correlation ID: 6875ec3c, Job ID: 007ba222-0c39-4f13-a49b-29f2b013a668, Call Stack: null, Custom Event ID: -1, Message: Storage Domain iscsi4-isoTTTTT (Data Center iscsi4) was deactivated by admin@internal
2013-08-06 15:28:27,610 INFO  [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp-/127.0.0.1:8702-5) [43c1714f] Lock Acquired to object EngineLock [exclusiveLocks= key: e6f7cae9-2607-4699-9f4f-5e568df21caa value: STORAGE
, sharedLocks= ]
2013-08-06 15:28:27,660 INFO  [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (pool-5-thread-46) [43c1714f] Running command: RemoveStorageDomainCommand internal: false. Entities affected :  ID: e6f7cae9-2607-4699-9f4f-5e568df21caa Type: Storage
2013-08-06 15:28:27,684 INFO  [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-5-thread-46) Running command: DetachStorageDomainFromPoolCommand internal: true. Entities affected :  ID: e6f7cae9-2607-4699-9f4f-5e568df21caa Type: Storage
2013-08-06 15:28:27,684 INFO  [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-5-thread-46) Start detach storage domain
2013-08-06 15:28:27,687 INFO  [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-5-thread-46)  Detach storage domain: before connect
2013-08-06 15:28:27,694 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-5-thread-49) Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
2013-08-06 15:28:27,707 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-5-thread-49) START, ConnectStorageServerVDSCommand(HostName = nott-vds2, HostId = 5d7ce646-22c0-45e7-af68-e6120c97d94f, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: 96d1162f-dee2-4e61-8832-93506654fddd, connection: /opt/elad2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6bde2422
2013-08-06 15:28:27,729 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-5-thread-49) FINISH, ConnectStorageServerVDSCommand, return: {96d1162f-dee2-4e61-8832-93506654fddd=0}, log id: 6bde2422
2013-08-06 15:28:27,730 INFO  [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-5-thread-46)  Detach storage domain: after connect
2013-08-06 15:28:27,730 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (pool-5-thread-46) START, DetachStorageDomainVDSCommand( storagePoolId = 072c2d76-8886-47ab-a1f9-d97f834115af, ignoreFailoverLimit = false, storageDomainId = e6f7cae9-2607-4699-9f4f-5e568df21caa, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1, force = false), log id: 42fc2410
2013-08-06 15:28:32,684 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (pool-5-thread-46) FINISH, DetachStorageDomainVDSCommand, log id: 42fc2410
2013-08-06 15:28:32,684 INFO  [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-5-thread-46)  Detach storage domain: after detach in vds
2013-08-06 15:28:32,689 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-5-thread-49) START, RefreshStoragePoolVDSCommand(HostName = nott-vds2, HostId = 5d7ce646-22c0-45e7-af68-e6120c97d94f, storagePoolId = 072c2d76-8886-47ab-a1f9-d97f834115af, masterStorageDomainId=c0e5eecc-53fc-4558-9477-0db4a6b9e873, masterVersion=1), log id: 4584402b
2013-08-06 15:28:32,698 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-5-thread-49) FINISH, RefreshStoragePoolVDSCommand, log id: 4584402b
2013-08-06 15:28:32,700 INFO  [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (pool-5-thread-49) Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
2013-08-06 15:28:32,706 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-5-thread-49) START, DisconnectStorageServerVDSCommand(HostName = nott-vds2, HostId = 5d7ce646-22c0-45e7-af68-e6120c97d94f, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: 96d1162f-dee2-4e61-8832-93506654fddd, connection: /opt/elad2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 36e9e1b
2013-08-06 15:28:36,950 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-5-thread-49) FINISH, DisconnectStorageServerVDSCommand, return: {96d1162f-dee2-4e61-8832-93506654fddd=0}, log id: 36e9e1b
2013-08-06 15:28:36,951 INFO  [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-5-thread-46)  Detach storage domain: after disconnect storage
2013-08-06 15:28:36,955 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetISOPathVDSCommand] (pool-5-thread-46) START, ResetISOPathVDSCommand( storagePoolId = 072c2d76-8886-47ab-a1f9-d97f834115af, ignoreFailoverLimit = false), log id: 27029a91
2013-08-06 15:28:36,955 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetISOPathVDSCommand] (pool-5-thread-46) FINISH, ResetISOPathVDSCommand, log id: 27029a91
2013-08-06 15:28:36,955 INFO  [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-5-thread-46) End detach storage domain
2013-08-06 15:28:36,959 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-5-thread-46) Correlation ID: 39b475a0, Call Stack: null, Custom Event ID: -1, Message: Storage Domain iscsi4-isoTTTTT was detached from Data Center iscsi4 by admin@internal



and then, for some reason, engine sends ConnectStorageToVds to another host which is in maintenance:

2013-08-06 15:28:36,962 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-5-thread-46) Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
2013-08-06 15:28:36,964 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-5-thread-46) START, ConnectStorageServerVDSCommand(HostName = green-vdsa, HostId = c22ebae6-c72e-42c1-8d80-d55382888406, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: 96d1162f-dee2-4e61-8832-93506654fddd, connection: /opt/elad2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 26af0dd4
2013-08-06 15:28:37,571 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-5-thread-46) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host green-vdsa to the Storage Domains iscsi4-isoTTTTT.
2013-08-06 15:28:37,571 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-5-thread-46) FINISH, ConnectStorageServerVDSCommand, return: {96d1162f-dee2-4e61-8832-93506654fddd=469}, log id: 26af0dd4
2013-08-06 15:28:37,572 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-5-thread-46) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection /opt/elad2 returned by VDSM was: Permission settings on the specified path do not allow access to the storage.
Verify permission settings on the specified storage path.


The operation fails with this error: (which is related to this bug https://bugzilla.redhat.com/show_bug.cgi?id=993111)

2013-08-06 15:28:37,572 ERROR [org.ovirt.engine.core.bll.storage.LOCALFSStorageHelper] (pool-5-thread-46) The connection with details /opt/elad2 failed because of error code 469 and error message is: permission settings on the specified path do not allow access to the storage.
verify permission settings on the specified storage path.


Expected results:
engine should send ConnectStorageServer to an active host on cluster

Additional info: logs
Comment 1 Allon Mureinik 2013-09-17 14:51:56 UTC 
As part of bug 993111, the behavior changed so a local ISO domain can only be attached to a local DC.

If the single host in the DC is up - attach and detach work properly.
If the single host in the DC is not up - you cannot put the domain in maintenance anyway.
Moving to ON_QA to verify
Comment 2 Elad 2013-09-22 12:09:56 UTC 
There is no option to add more than 1 host to a local DC anymore.
Verified on is15:
rhevm-3.3.0-0.21.master.el6ev.noarch
Comment 3 Itamar Heim 2014-01-21 22:31:14 UTC 
Closing - RHEV 3.3 Released
Comment 4 Itamar Heim 2014-01-21 22:31:15 UTC 
Closing - RHEV 3.3 Released
Comment 5 Itamar Heim 2014-01-21 22:34:02 UTC 
Closing - RHEV 3.3 Released