Bug 1007507

Summary: Failures to remove images from an import domain result in imported images on data domains being marked as illegal.
Product: Red Hat Enterprise Virtualization Manager Reporter: rhev-integ
Component: ovirt-engineAssignee: Lee Yarwood <lyarwood>
Status: CLOSED ERRATA QA Contact: Aharon Canan <acanan>
Severity: high Docs Contact:
Priority: high    
Version: 3.2.0CC: abaron, acathrow, amureini, iheim, lpeer, Rhev-m-bugs, scohen, yeylon
Target Milestone: ---Keywords: ZStream
Target Release: 3.2.4Flags: lsvaty: testing_plan_complete-
Hardware: All   
OS: Linux   
Whiteboard: storage
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Failure to remove volumes from an import domain marked the imported volumes on the data domain as illegal. With this update, the engine now checks if the parent command is an image or template removal command before setting the volume as illegal.
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-10-25 18:43:39 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1002664    
Bug Blocks:    

Description rhev-integ 2013-09-12 15:45:35 UTC
+++ This bug is a RHEV-M zstream clone. The original bug is: +++
+++   https://bugzilla.redhat.com/show_bug.cgi?id=1002664. +++

======================================================================



----------------------------------------------------------------------
Following comment by lyarwood on August 29 at 16:54:05, 2013

Description of problem:
Failures to remove volumes from an import domain result in imported volumes on data domains being marked as illegal.

Version-Release number of selected component (if applicable):
rhevm-3.2.2-0.41

How reproducible:
Always.

Steps to Reproduce:
1. Export a guest to an import domain.
2. Manually remove the volume from the import domain.
3. Attempt to remove the volume from the import domain.
4. The original volume on the data domain is now marked as illegal.

Actual results:
Original volumes marked as illegal.

Expected results:
Imported volumes are not touched as a result of any failures to remove volumes from the import domain.

Additional info:

----------------------------------------------------------------------
Following comment by lyarwood on August 29 at 16:56:21, 2013

For example, in the attached case we see the following in engine.log  for a given delete task (will attach) :

# grep 31b7336a var/log/ovirt-engine/engine.log
2013-08-19 15:52:00,557 INFO  [org.ovirt.engine.core.bll.RemoveVmFromImportExportCommand] (pool-4-thread-49) [31b7336a] Running command: RemoveVmFromImportExportCommand internal: false. Entities affected :  ID: 39196f33-7940-4e49-8921-3786b0306cab Type: VM
2013-08-19 15:52:00,558 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.RemoveVMVDSCommand] (pool-4-thread-49) [31b7336a] START, RemoveVMVDSCommand( storagePoolId = 196592fe-7b16-458d-b7da-cb31d0b4cae3, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 9e384616-9f4d-4b99-8188-1fe7209f6964, vmGuid = 39196f33-7940-4e49-8921-3786b0306cab), log id: 2a6a683f
2013-08-19 15:52:00,622 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.RemoveVMVDSCommand] (pool-4-thread-49) [31b7336a] FINISH, RemoveVMVDSCommand, log id: 2a6a683f
2013-08-19 15:52:00,653 INFO  [org.ovirt.engine.core.bll.RemoveAllVmImagesCommand] (pool-4-thread-49) [31b7336a] Running command: RemoveAllVmImagesCommand internal: true. Entities affected :  ID: 39196f33-7940-4e49-8921-3786b0306cab Type: VM
2013-08-19 15:52:00,659 INFO  [org.ovirt.engine.core.bll.RemoveImageCommand] (pool-4-thread-49) [31b7336a] Running command: RemoveImageCommand internal: true. Entities affected :  ID: 00000000-0000-0000-0000-000000000000 Type: Storage
2013-08-19 15:52:00,661 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (pool-4-thread-49) [31b7336a] START, DeleteImageGroupVDSCommand( storagePoolId = 196592fe-7b16-458d-b7da-cb31d0b4cae3, ignoreFailoverLimit = false, compatabilityVersion = 3.1, storageDomainId = 9e384616-9f4d-4b99-8188-1fe7209f6964, imageGroupId = 687cda1a-d6b0-416d-8b35-1a86d6a7e720, postZeros = false, forceDelete = false), log id: 36d6e5f
2013-08-19 15:52:35,286 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (pool-4-thread-49) [31b7336a] FINISH, DeleteImageGroupVDSCommand, log id: 36d6e5f
2013-08-19 15:52:35,292 INFO  [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-49) [31b7336a] EntityAsyncTask::Adding EntityMultiAsyncTasks object for entity 39196f33-7940-4e49-8921-3786b0306cab
2013-08-19 15:52:35,292 INFO  [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (pool-4-thread-49) [31b7336a] EntityMultiAsyncTasks::AttachTask: Attaching task 830b8c38-03c5-4c78-a0a2-8f5b9bd1aefe to entity 39196f33-7940-4e49-8921-3786b0306cab.
2013-08-19 15:52:35,340 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (pool-4-thread-49) [31b7336a] Adding task 830b8c38-03c5-4c78-a0a2-8f5b9bd1aefe (Parent Command RemoveVmFromImportExport, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet..
2013-08-19 15:52:35,353 INFO  [org.ovirt.engine.core.bll.RemoveImageCommand] (pool-4-thread-49) [31b7336a] Running command: RemoveImageCommand internal: true. Entities affected :  ID: 00000000-0000-0000-0000-000000000000 Type: Storage
2013-08-19 15:52:35,354 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (pool-4-thread-49) [31b7336a] START, DeleteImageGroupVDSCommand( storagePoolId = 196592fe-7b16-458d-b7da-cb31d0b4cae3, ignoreFailoverLimit = false, compatabilityVersion = 3.1, storageDomainId = 9e384616-9f4d-4b99-8188-1fe7209f6964, imageGroupId = 1f205d2c-ae58-4b9a-9dd3-ddefedb6f177, postZeros = false, forceDelete = false), log id: 329f99f0
2013-08-19 15:53:35,612 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-4-thread-49) [31b7336a] Failed in DeleteImageGroupVDS method
2013-08-19 15:53:35,613 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-4-thread-49) [31b7336a] Error code GeneralException and error message IRSGenericException: IRSErrorException: Failed to DeleteImageGroupVDS, error = 
2013-08-19 15:53:35,613 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-49) [31b7336a] IrsBroker::Failed::DeleteImageGroupVDS due to: IRSErrorException: IRSGenericException: IRSErrorException: Failed to DeleteImageGroupVDS, error = 
2013-08-19 15:53:35,681 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (pool-4-thread-49) [31b7336a] START, SpmStopVDSCommand(HostName = ecp06.thw.itio.com, HostId = d25fb9e4-a8f2-11e2-9088-001a645cbc08, storagePoolId = 196592fe-7b16-458d-b7da-cb31d0b4cae3), log id: 1be03a71
2013-08-19 15:53:35,717 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (pool-4-thread-49) [31b7336a] FINISH, SpmStopVDSCommand, log id: 1be03a71
2013-08-19 15:53:35,717 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-49) [31b7336a] IRS failover failed - cant allocate vds server
2013-08-19 15:53:35,718 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (pool-4-thread-49) [31b7336a] FINISH, DeleteImageGroupVDSCommand, log id: 329f99f0
2013-08-19 15:53:35,718 ERROR [org.ovirt.engine.core.bll.RemoveImageCommand] (pool-4-thread-49) [31b7336a] Command org.ovirt.engine.core.bll.RemoveImageCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException: IRSGenericException: IRSErrorException: Failed to DeleteImageGroupVDS, error = 
2013-08-19 15:53:35,720 ERROR [org.ovirt.engine.core.bll.RemoveAllVmImagesCommand] (pool-4-thread-49) [31b7336a] Cant remove image id: 9654e7e3-0336-4257-b9e7-732ef13786b7 for VM id: 39196f33-7940-4e49-8921-3786b0306cab due to: GeneralException. Image will be set at illegal state with no snapshot id.
2013-08-19 15:53:35,744 INFO  [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-49) [31b7336a] BaseAsyncTask::StartPollingTask: Starting to poll task 830b8c38-03c5-4c78-a0a2-8f5b9bd1aefe.
2013-08-19 15:53:35,744 INFO  [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (pool-4-thread-49) [31b7336a] EntityMultiAsyncTasks::StartPollingTask: Current Action Type for entity 39196f33-7940-4e49-8921-3786b0306cab is RemoveVmFromImportExport (determined by task 830b8c38-03c5-4c78-a0a2-8f5b9bd1aefe)

Where the guest volumes end up like this :

engine=> select vm_names, imagestatus, storage_name, image_group_id from vm_images_view where vm_names = 'perldev01';
 vm_names  | imagestatus |  storage_name   |            image_group_id            
-----------+-------------+-----------------+--------------------------------------
 perldev01 |           4 | THW_B_SATA_DATA | 1f205d2c-ae58-4b9a-9dd3-ddefedb6f177
 perldev01 |           1 | THW_B_SATA_DATA | 687cda1a-d6b0-416d-8b35-1a86d6a7e720
(2 rows)

----------------------------------------------------------------------
Following comment by lyarwood on August 29 at 16:59:59, 2013

Created attachment 791895 [details]
SFDC#00924870 - example engine.log

----------------------------------------------------------------------
Following comment by lyarwood on August 29 at 22:36:14, 2013

I've attached a quick patch to workaround this by checking if the parent command is an image/template removal command for an import domain.

----------------------------------------------------------------------
Following comment by amureini on September 11 at 08:30:03, 2013

Upstream patch given in the external tracker was merged.

I'm leaving the bug on POST since is14 build is currently blocked, and its unclear what commit it will eventually use when its released.
Once this issue is cleared, the bug should either be moved to MODIFIED if is14 includes it or backported to d/s gerrit.

----------------------------------------------------------------------
Following comment by amureini on September 12 at 15:07:43, 2013

(In reply to Allon Mureinik from comment #4)
> Upstream patch given in the external tracker was merged.
> 
> I'm leaving the bug on POST since is14 build is currently blocked, and its
> unclear what commit it will eventually use when its released.
> Once this issue is cleared, the bug should either be moved to MODIFIED if
> is14 includes it or backported to d/s gerrit.
We are still rebasing - This patch will be autoamtically delivered in IS15.

Comment 2 Aharon Canan 2013-10-14 14:19:47 UTC
verified using sf21

verification steps - 
1. create DC with data domain and export domain
2. create VM and export it
3. manually remove the disk from the export domain path
4. on engine remove the VM from the export domain

original disk is OK, everything looks fine

Comment 4 errata-xmlrpc 2013-10-25 18:43:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1468.html