Bug 969026

Summary: [engine] Storage Domain that was destroyed from GUI did not get completely removed from DB
Product: Red Hat Enterprise Virtualization Manager Reporter: Gadi Ickowicz <gickowic>
Component: ovirt-engineAssignee: Liron Aravot <laravot>
Status: CLOSED CURRENTRELEASE QA Contact: Ori Gofen <ogofen>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.2.0CC: acanan, amureini, bsettle, dron, iheim, lpeer, nlevinki, rbalakri, Rhev-m-bugs, scohen, tnisan, yeylon
Target Milestone: ---Flags: amureini: Triaged+
Target Release: 3.5.0   
Hardware: All   
OS: Linux   
Whiteboard: storage
Fixed In Version: ovirt-3.5.0-alpha2 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-02-16 19:10:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1142923, 1156165    
Attachments:
Description Flags
vdsm + engine logs none

Description Gadi Ickowicz 2013-05-30 13:58:51 UTC
Created attachment 754868 [details]
vdsm + engine logs

Description of problem:
After creating a new DC with 1 host and 1 storage domain, and 1 additional domain which is not attached, all storages were destroyed (directly on the storage server). To clean the RHEVM the host was moved to maintenance, DC was force-removed and additional domain was destroyed. However the domain remained in the storage_domain_static table. Now it is not possible to add another domain to the RHEVM with the same name as that one.


Queries to the DB:
# psql -d engine -U postgres -c "select * from storage_domain_static;"
                 id                  |                storage                 | storage_name | storage_domain_type | storage_type | storage_domain_format_type |         _create_date          |         _update_dat
--------------------------------------+----------------------------------------+--------------+---------------------+--------------+----------------------------+-------------------------------+--------------------
 feb6aa0e-4302-42b9-86a7-84d902fea07e | AxHc8t-CRgE-FKft-qrSl-PG8O-Svs5-9m5dL3 | iscsi_2      |                   1 |            3 | 3                          | 2013-05-26 14:39:33.223723+03 |                    
 7d3d2d42-270f-45dc-b6e4-20fc1f389e37 | iDGExr-4Kch-TKl0-GzEV-BxgG-bvsP-rfK6Ly | iscsi_0      |                   0 |            3 | 3                          | 2013-05-30 14:25:00.450573+03 | 2013-05-30 14:25:04
 f1103bf0-65cb-4d7a-a7cd-1656ea1252cd | RYPDXv-AfXt-7Dyz-SA4d-nVwI-xFOe-I0FjU6 | iscsi_1      |                   1 |            3 | 0                          | 2013-05-30 14:26:22.152056+03 | 2013-05-30 14:26:22



# # psql -d engine -U postgres -c "select * from storage_domains;"
                  id                  |                storage                 | storage_name |           storage_pool_id            | available_disk_size | used_disk_size | commited_disk_size | status | storage_p
--------------------------------------+----------------------------------------+--------------+--------------------------------------+---------------------+----------------+--------------------+--------+----------
 7d3d2d42-270f-45dc-b6e4-20fc1f389e37 | iDGExr-4Kch-TKl0-GzEV-BxgG-bvsP-rfK6Ly | iscsi_0      | 7cc34b9a-ea43-49ba-a73d-f667bb93f664 |                     |                |                  0 |      4 | TestDC   
 f1103bf0-65cb-4d7a-a7cd-1656ea1252cd | RYPDXv-AfXt-7Dyz-SA4d-nVwI-xFOe-I0FjU6 | iscsi_1      |                                      |                  10 |              4 |                  0 |        |          
(2 rows)



Version-Release number of selected component (if applicable):
rhevm-3.2.0-11.28.el6ev.noarch

How reproducible:
?

Steps to Reproduce:
1. Create datacenter, cluster and add 1 host
2. Create storage domain and attach it to DC
3. Create additonal storage domain (not attached to DC)
4. Destroy storage (directly on the storage server)
5. When host is in non-operational move it to maintenance and force-remove DC
6. Destroy remaining storage domain

Actual results:
After repeating this procedure several times I was suddenly unable to re-add a storage domain with the same name as the additional domain. Checking the DB showed that the storage domain entry remained in the storage_domain_static table, but not in the storage_domains table

Expected results:
destroying a domain should remove it completely from the DB


Additional info:

Comment 1 Alissa 2013-07-09 12:40:03 UTC
What did you mean by storage_domains table? there is no such table in the db... storage_domains is a view. in which specific table did you see the records remained?

Also, you wrote "After repeating this procedure several times" -is this consistent? How many times the procedure needs to be repeated for reproduction?

Comment 3 Alissa 2013-08-19 14:47:22 UTC
Some information and the cause for the problem in the bug:
relevant storage domain id is feb6aa0e-4302-42b9-86a7-84d902fea07e
1. ReconstructMasterDomainCommand starts, compensates storage_domain_static. 
2. ForceRemoveStorageDomainCommand starts.
It removes the storage domain from the db.
3. ReconstructMasterDomainCommand fails.
It triggers the compensation of the storage domain. Since it was removed from db by step 2 (ForceRemoveStorageDomainCommand), the compensation gets storage domain inserted  again to storage_domain_static table.

Comment 5 Allon Mureinik 2014-02-16 09:09:22 UTC
Liron, was this solved as part of your on reconstruct?

Comment 6 Liron Aravot 2014-03-02 13:51:29 UTC
Allon, no.
I provided patches to solve the issue, added to the tracker.

Comment 7 Kevin Alon Goldblatt 2014-07-07 07:19:48 UTC
All instances of the storage domains were removed from the database after performing the above scenario ONCE. Is this sufficient to move this defect to Verified? Gadi?

Comment 8 Gadi Ickowicz 2014-07-08 08:16:41 UTC
(In reply to Kevin Alon Goldblatt from comment #7)
> All instances of the storage domains were removed from the database after
> performing the above scenario ONCE. Is this sufficient to move this defect
> to Verified? Gadi?

If it worked, then I guess so

Comment 9 Ori Gofen 2014-07-29 07:51:16 UTC
verified on oVirt beta.2

Comment 11 Allon Mureinik 2015-02-16 19:10:13 UTC
RHEV-M 3.5.0 has been released, closing this bug.