Bug 1011569
Summary: | Cannot remove an iscsi storage connection not attached to any storage domain | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Virtualization Manager | Reporter: | Katarzyna Jachim <kjachim> | ||||
Component: | ovirt-engine-restapi | Assignee: | Daniel Erez <derez> | ||||
Status: | CLOSED CURRENTRELEASE | QA Contact: | Katarzyna Jachim <kjachim> | ||||
Severity: | unspecified | Docs Contact: | |||||
Priority: | unspecified | ||||||
Version: | 3.3.0 | CC: | abaron, abonas, acanan, acathrow, amureini, bazulay, iheim, kjachim, ncredi, oramraz, Rhev-m-bugs, scohen, yeylon | ||||
Target Milestone: | --- | Flags: | abaron:
Triaged+
|
||||
Target Release: | 3.3.0 | ||||||
Hardware: | Unspecified | ||||||
OS: | Unspecified | ||||||
Whiteboard: | storage | ||||||
Fixed In Version: | is18 | Doc Type: | Bug Fix | ||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | Environment: | ||||||
Last Closed: | Type: | Bug | |||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | Storage | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Attachments: |
|
Description
Katarzyna Jachim
2013-09-24 14:46:42 UTC
Created attachment 802271 [details]
vdsm.log + engine.log + server.log + db dump
Do you have direct LUNs that may be using this connection? Looking at the db dump, it seems that there are still leftovers mentioning this connection related to a lun in the luns-connections table: lun_storage_server_connection_map (lun_id, storage_server_connection) FROM stdin; 1kjachim02 b001aac3-3066-4eaf-aa70-a5d217ce678e And the lun 1kjachim02 also still exists in the luns table, with a volumeGroupId. volumeGroupId is an indication of the fact that storage domain is using the lun (and respectively - the connection) even that in this probably not clean setup the storage domain was deleted without cleanup of its luns. The engine log has this (mention of domain sd_288968_2): 2013-09-24 13:31:46,236 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-1) Lock Acquired to object EngineLock [exclusiveLocks= key: null value: STORAGE_CONNECTION key: b001aac3-3066-4eaf-aa70-a5d217ce678e value: STORAGE_CONNECTION , sharedLocks= ] 2013-09-24 13:31:46,238 WARN [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-1) CanDoAction of action RemoveStorageServerConnection failed. Reasons:VAR__ACTION__REMOVE,VAR__TYPE__STORAGE__CONNECTION,$domainNames sd_288968_2,sd_288968_2,ACTION_TYPE_FAILED_STORAGE_CONNECTION_BELONGS_TO_SEVERAL_STORAGE_DOMAINS 2013-09-24 13:31:46,239 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-1) Lock freed to object EngineLock [exclusiveLocks= key: null value: STORAGE_CONNECTION key: b001aac3-3066-4eaf-aa70-a5d217ce678e value: STORAGE_CONNECTION , sharedLocks= ] 2013-09-24 13:31:46,297 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp-/127.0.0.1:8702-1) Operation Failed: [Cannot remove Storage Connection. Storage connection parameters are used by the following storage domains : sd_288968_2,sd_288968_2.] According to the audit log, there was an attempt to remove this domain but it failed - so this might cause not a clean removal that left the luns not deleted: 3ae219c3-9be6-4f69-8834-c2c326bd0970 RemoveStorageDomain Removing Storage Domain sd_288968_2 from Data Center <UNKNOWN> FAILED (In reply to Alissa from comment #3) > Looking at the db dump, it seems that there are still leftovers mentioning > this connection related to a lun in the luns-connections table: > > lun_storage_server_connection_map (lun_id, storage_server_connection) FROM > stdin; > 1kjachim02 b001aac3-3066-4eaf-aa70-a5d217ce678e > > And the lun 1kjachim02 also still exists in the luns table, with a > volumeGroupId. > volumeGroupId is an indication of the fact that storage domain is using the > lun (and respectively - the connection) even that in this probably not clean > setup the storage domain was deleted without cleanup of its luns. > > > The engine log has this (mention of domain sd_288968_2): > 2013-09-24 13:31:46,236 INFO > [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] > (ajp-/127.0.0.1:8702-1) Lock Acquired to object EngineLock [exclusiveLocks= > key: null value: STORAGE_CONNECTION > key: b001aac3-3066-4eaf-aa70-a5d217ce678e value: STORAGE_CONNECTION > , sharedLocks= ] > 2013-09-24 13:31:46,238 WARN > [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] > (ajp-/127.0.0.1:8702-1) CanDoAction of action RemoveStorageServerConnection > failed. > Reasons:VAR__ACTION__REMOVE,VAR__TYPE__STORAGE__CONNECTION,$domainNames > sd_288968_2,sd_288968_2, > ACTION_TYPE_FAILED_STORAGE_CONNECTION_BELONGS_TO_SEVERAL_STORAGE_DOMAINS > 2013-09-24 13:31:46,239 INFO > [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] > (ajp-/127.0.0.1:8702-1) Lock freed to object EngineLock [exclusiveLocks= > key: null value: STORAGE_CONNECTION > key: b001aac3-3066-4eaf-aa70-a5d217ce678e value: STORAGE_CONNECTION > , sharedLocks= ] > 2013-09-24 13:31:46,297 ERROR > [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] > (ajp-/127.0.0.1:8702-1) Operation Failed: [Cannot remove Storage Connection. > Storage connection parameters are used by the following storage domains : > sd_288968_2,sd_288968_2.] > > According to the audit log, there was an attempt to remove this domain but > it failed - so this might cause not a clean removal that left the luns not > deleted: > > 3ae219c3-9be6-4f69-8834-c2c326bd0970 RemoveStorageDomain Removing Storage > Domain sd_288968_2 from Data Center <UNKNOWN> FAILED How can the user get out of this state? I am not sure this is a classic user case. This is a test environment and I don't know how it was cleaned, or in which method entities were deleted. Having said that, I think that removal of storage domain should be atomical if it isn't already. If removal of storage domain consists of several deletions from several db tables, it should be either all (commit) or nothing (rollback) - without leftovers. That way, there will be no leftovers in db and user will not get into this kind of situation. Hi Katarzyna, In which API calls did you use to clean the environment? Specifically, how did you remove sd_288968_2 storage domain? from engine.log: 2013-09-24 13:35:42,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FormatStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-11) START, FormatStorageDomainVDSCommand(HostName = 10.34.63.216, HostId = 44cfe251-31ba-49c1-9980-e266562d016b, storageDomainId=5c4578e2-b901-47e2-a167-bbd0457203e4), log id: 4ccc57f I killed my test at 2013-09-24 15:33:20,261 (if you want, I may attach also test log) and tried to clean my RHEV-M setup manually, so removed everything which is possible via GUI and then tried to remove left storage connections via REST API. (In reply to Katarzyna Jachim from comment #7) > from engine.log: > > 2013-09-24 13:35:42,151 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.FormatStorageDomainVDSCommand] > (ajp-/127.0.0.1:8702-11) START, FormatStorageDomainVDSCommand(HostName = > 10.34.63.216, HostId = 44cfe251-31ba-49c1-9980-e266562d016b, > storageDomainId=5c4578e2-b901-47e2-a167-bbd0457203e4), log id: 4ccc57f > > I killed my test at 2013-09-24 15:33:20,261 (if you want, I may attach also > test log) and tried to clean my RHEV-M setup manually, so removed everything > which is possible via GUI and then tried to remove left storage connections > via REST API. - Yes, please attach the test log. - Have you used remove or force remove/destroy from the GUI? I don't have an exact scenario for verification, but I haven't seen it in the newest versions, so I assume it is fixed. Closing - RHEV 3.3 Released Closing - RHEV 3.3 Released Closing - RHEV 3.3 Released |