Created attachment 783311 [details] logs Description of problem: When detaching a local ISO domain from pool, for some reason, engine sends ConnectStorageToVdsCommand to another host (not the one that has the path of the ISO domain storage connection) which is in maintenance. Version-Release number of selected component (if applicable): rhevm-3.3.0-0.13.master.el6ev.noarch How reproducible: always Steps to Reproduce: 1. on 2 hosts cluster with an active storage pool 2. create a local ISO domain on one of the hosts 3. put the other host in maintenance 4. maintenance the ISO domain and detach it. Actual results: engine sends DeactivateStorageDomain, DisconnectStorageServerConnection, RemoveStorageDomain and DetachStorageDomainFromPool to the active host (which contains the ISO domain): 2013-08-06 15:27:56,808 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (pool-5-thread-46) [6875ec3c] Running command: DeactivateStorageDomainCommand internal: false. Entities affected : ID: e6f7cae9-2607-4699-9 f4f-5e568df21caa Type: Storage 2013-08-06 15:27:56,814 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (pool-5-thread-46) [6875ec3c] START, DeactivateStorageDomainVDSCommand( storagePoolId = 072c2d76-8886-47ab-a1f9-d97f834115af, ign oreFailoverLimit = false, storageDomainId = e6f7cae9-2607-4699-9f4f-5e568df21caa, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1), log id: 73a29a4c 2013-08-06 15:27:57,395 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (pool-5-thread-46) [6875ec3c] FINISH, DeactivateStorageDomainVDSCommand, log id: 73a29a4c 2013-08-06 15:27:57,395 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (pool-5-thread-46) [6875ec3c] Lock freed to object EngineLock [exclusiveLocks= key: e6f7cae9-2607-4699-9f4f-5e568df21caa value: STORAGE , sharedLocks= ] 2013-08-06 15:27:57,398 INFO [org.ovirt.engine.core.bll.storage.AfterDeactivateSingleAsyncOperation] (pool-5-thread-49) [6875ec3c] After deactivate treatment vds: nott-vds2,pool iscsi4 2013-08-06 15:27:57,402 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-5-thread-49) [6875ec3c] START, RefreshStoragePoolVDSCommand(HostName = nott-vds2, HostId = 5d7ce646-22c0-45e7-af68-e6120c97d94f, storagePoolId = 072c2d76-8886-47ab-a1f9-d97f834115af, masterStorageDomainId=c0e5eecc-53fc-4558-9477-0db4a6b9e873, masterVersion=1), log id: 13b4fcc1 2013-08-06 15:27:57,435 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-5-thread-49) [6875ec3c] FINISH, RefreshStoragePoolVDSCommand, log id: 13b4fcc1 2013-08-06 15:27:57,436 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (pool-5-thread-46) Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System 2013-08-06 15:27:57,443 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-5-thread-46) START, DisconnectStorageServerVDSCommand(HostName = nott-vds2, HostId = 5d7ce646-22c0-45e7-af68-e6120c97d94f, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: 96d1162f-dee2-4e61-8832-93506654fddd, connection: /opt/elad2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 315ba033 2013-08-06 15:28:00,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-5-thread-46) FINISH, DisconnectStorageServerVDSCommand, return: {96d1162f-dee2-4e61-8832-93506654fddd=0}, log id: 315ba033 2013-08-06 15:28:00,755 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-5-thread-46) Correlation ID: 6875ec3c, Job ID: 007ba222-0c39-4f13-a49b-29f2b013a668, Call Stack: null, Custom Event ID: -1, Message: Storage Domain iscsi4-isoTTTTT (Data Center iscsi4) was deactivated by admin@internal 2013-08-06 15:28:27,610 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp-/127.0.0.1:8702-5) [43c1714f] Lock Acquired to object EngineLock [exclusiveLocks= key: e6f7cae9-2607-4699-9f4f-5e568df21caa value: STORAGE , sharedLocks= ] 2013-08-06 15:28:27,660 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (pool-5-thread-46) [43c1714f] Running command: RemoveStorageDomainCommand internal: false. Entities affected : ID: e6f7cae9-2607-4699-9f4f-5e568df21caa Type: Storage 2013-08-06 15:28:27,684 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-5-thread-46) Running command: DetachStorageDomainFromPoolCommand internal: true. Entities affected : ID: e6f7cae9-2607-4699-9f4f-5e568df21caa Type: Storage 2013-08-06 15:28:27,684 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-5-thread-46) Start detach storage domain 2013-08-06 15:28:27,687 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-5-thread-46) Detach storage domain: before connect 2013-08-06 15:28:27,694 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-5-thread-49) Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System 2013-08-06 15:28:27,707 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-5-thread-49) START, ConnectStorageServerVDSCommand(HostName = nott-vds2, HostId = 5d7ce646-22c0-45e7-af68-e6120c97d94f, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: 96d1162f-dee2-4e61-8832-93506654fddd, connection: /opt/elad2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6bde2422 2013-08-06 15:28:27,729 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-5-thread-49) FINISH, ConnectStorageServerVDSCommand, return: {96d1162f-dee2-4e61-8832-93506654fddd=0}, log id: 6bde2422 2013-08-06 15:28:27,730 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-5-thread-46) Detach storage domain: after connect 2013-08-06 15:28:27,730 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (pool-5-thread-46) START, DetachStorageDomainVDSCommand( storagePoolId = 072c2d76-8886-47ab-a1f9-d97f834115af, ignoreFailoverLimit = false, storageDomainId = e6f7cae9-2607-4699-9f4f-5e568df21caa, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1, force = false), log id: 42fc2410 2013-08-06 15:28:32,684 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (pool-5-thread-46) FINISH, DetachStorageDomainVDSCommand, log id: 42fc2410 2013-08-06 15:28:32,684 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-5-thread-46) Detach storage domain: after detach in vds 2013-08-06 15:28:32,689 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-5-thread-49) START, RefreshStoragePoolVDSCommand(HostName = nott-vds2, HostId = 5d7ce646-22c0-45e7-af68-e6120c97d94f, storagePoolId = 072c2d76-8886-47ab-a1f9-d97f834115af, masterStorageDomainId=c0e5eecc-53fc-4558-9477-0db4a6b9e873, masterVersion=1), log id: 4584402b 2013-08-06 15:28:32,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-5-thread-49) FINISH, RefreshStoragePoolVDSCommand, log id: 4584402b 2013-08-06 15:28:32,700 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (pool-5-thread-49) Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System 2013-08-06 15:28:32,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-5-thread-49) START, DisconnectStorageServerVDSCommand(HostName = nott-vds2, HostId = 5d7ce646-22c0-45e7-af68-e6120c97d94f, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: 96d1162f-dee2-4e61-8832-93506654fddd, connection: /opt/elad2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 36e9e1b 2013-08-06 15:28:36,950 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-5-thread-49) FINISH, DisconnectStorageServerVDSCommand, return: {96d1162f-dee2-4e61-8832-93506654fddd=0}, log id: 36e9e1b 2013-08-06 15:28:36,951 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-5-thread-46) Detach storage domain: after disconnect storage 2013-08-06 15:28:36,955 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetISOPathVDSCommand] (pool-5-thread-46) START, ResetISOPathVDSCommand( storagePoolId = 072c2d76-8886-47ab-a1f9-d97f834115af, ignoreFailoverLimit = false), log id: 27029a91 2013-08-06 15:28:36,955 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetISOPathVDSCommand] (pool-5-thread-46) FINISH, ResetISOPathVDSCommand, log id: 27029a91 2013-08-06 15:28:36,955 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-5-thread-46) End detach storage domain 2013-08-06 15:28:36,959 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-5-thread-46) Correlation ID: 39b475a0, Call Stack: null, Custom Event ID: -1, Message: Storage Domain iscsi4-isoTTTTT was detached from Data Center iscsi4 by admin@internal and then, for some reason, engine sends ConnectStorageToVds to another host which is in maintenance: 2013-08-06 15:28:36,962 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-5-thread-46) Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System 2013-08-06 15:28:36,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-5-thread-46) START, ConnectStorageServerVDSCommand(HostName = green-vdsa, HostId = c22ebae6-c72e-42c1-8d80-d55382888406, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: 96d1162f-dee2-4e61-8832-93506654fddd, connection: /opt/elad2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 26af0dd4 2013-08-06 15:28:37,571 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-5-thread-46) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host green-vdsa to the Storage Domains iscsi4-isoTTTTT. 2013-08-06 15:28:37,571 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-5-thread-46) FINISH, ConnectStorageServerVDSCommand, return: {96d1162f-dee2-4e61-8832-93506654fddd=469}, log id: 26af0dd4 2013-08-06 15:28:37,572 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-5-thread-46) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection /opt/elad2 returned by VDSM was: Permission settings on the specified path do not allow access to the storage. Verify permission settings on the specified storage path. The operation fails with this error: (which is related to this bug https://bugzilla.redhat.com/show_bug.cgi?id=993111) 2013-08-06 15:28:37,572 ERROR [org.ovirt.engine.core.bll.storage.LOCALFSStorageHelper] (pool-5-thread-46) The connection with details /opt/elad2 failed because of error code 469 and error message is: permission settings on the specified path do not allow access to the storage. verify permission settings on the specified storage path. Expected results: engine should send ConnectStorageServer to an active host on cluster Additional info: logs
As part of bug 993111, the behavior changed so a local ISO domain can only be attached to a local DC. If the single host in the DC is up - attach and detach work properly. If the single host in the DC is not up - you cannot put the domain in maintenance anyway. Moving to ON_QA to verify
There is no option to add more than 1 host to a local DC anymore. Verified on is15: rhevm-3.3.0-0.21.master.el6ev.noarch
Closing - RHEV 3.3 Released