Bug 1121089

Summary: [RHEV] VM removed while migrating, resulting in "VM not managed by engine"
Product: Red Hat Enterprise Virtualization Manager Reporter: Evgheni Dereveanchin <ederevea>
Component: ovirt-engineAssignee: Michal Skrivanek <michal.skrivanek>
Status: CLOSED ERRATA QA Contact: Israel Pinto <ipinto>
Severity: high Docs Contact:
Priority: medium    
Version: 3.3.0CC: anande, bazulay, ederevea, eedri, iheim, ipinto, istein, juan.hernandez, lpeer, mavital, mgoldboi, michal.skrivanek, mkalinin, oramraz, rbalakri, Rhev-m-bugs, sapandit, sherold, tdosek, usurse, yeylon, ykaul
Target Milestone: ovirt-3.6.0-rcFlags: mgoldboi: Triaged+
Target Release: 3.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previously, in rare cases a virtual machine disappeared during migration in the UI and reappeared as an external virtual machine that could no longer be managed. This error has now been fixed.
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-03-09 20:35:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Virt RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 947965, 1253043    
Bug Blocks:    

Description Evgheni Dereveanchin 2014-07-18 10:37:10 UTC
In RHEV 3.3 there seems to be a race condition in which a VM remove command is executed when in fact the guest is migrating between hosts. This results in the guest being removed from the source host but coming up on destination. Disks are also removed so the guest is actually unusable and may hang, appearing on the manager with "external-" prefix.

The VM creation/removal is done via the API.

Comment 6 Omer Frenkel 2014-08-07 13:00:58 UTC
exact race flow, for documentation:
migration ends, and vm moves to up on the destination.
in the engine, there are 2 monitoring threads that run in parallel (src,dst)
dst finish first, and move the vm to up, and release the migration lock.
then src finish, but override the status to migratingTo (hand-over), because it is not aware to the other update.

now we have vm in migratingTo, without a lock.
resolution is to mark this status as a 'running' status, like migratingFrom, and user will be blocked from removing based on vm status, when this case happens.

i was able to reproduce this (using debugger).

Comment 7 Omer Frenkel 2014-09-01 11:56:26 UTC
The fix for bug 947965 also fix this issue, as it will not allow removing a vm in migrating-to, and that fix is much safer than the fix proposed in this bug.

Comment 9 Anand Nande 2015-06-05 12:34:59 UTC
Hi,

I am not sure if the following should be part of this bug OR a new bug.

Suddenly some virtual machines appeared in Manager, prefixed with 'external'. It seems they were removed from Export Domain but now imported again. 

All the VMs/Windows guests are from storage_domain="FC-NetAPP-Windows0"

engine=> SELECT vm_id,vm_name,log_time,log_type_name,message from audit_log where message like '%external%';
                vm_id                 |           vm_name            |          log_time          | log_type_name |                          message                          
--------------------------------------+------------------------------+----------------------------+---------------+-----------------------------------------------------------
 5ef17f98-6ac4-4cd7-ba14-4f52dfac8bec | external-eevrt-002-big-local | 2015-05-26 16:39:10.603-04 | USER_ADD_VM   | VM external-eevrt-002-big-local was created by <UNKNOWN>.
 bd1e8f7f-db44-4207-8215-6394a239ee3a | external-skoor0-big-local    | 2015-05-26 16:39:10.453-04 | USER_ADD_VM   | VM external-skoor0-big-local was created by <UNKNOWN>.
 cb93d661-3fc7-4d13-beb4-765a2867febc | external-2008R2TEST          | 2015-05-27 14:26:47.257-04 | USER_ADD_VM   | VM external-2008R2TEST was created by <UNKNOWN>.
 474a70bb-8ba2-4b85-8aa8-8a79fbf18a15 | external-win2008R2-temp      | 2015-05-27 14:28:37.111-04 | USER_ADD_VM   | VM external-win2008R2-temp was created by <UNKNOWN>.
 4b76def7-74b6-4a0f-8a69-8a339df82144 | external-es3-test-big-local  | 2015-05-27 14:28:36.776-04 | USER_ADD_VM   | VM external-es3-test-big-local was created by <UNKNOWN>.
 908047a3-8d05-4273-8314-61023996e081 | external-es1.cloud.big.local | 2015-05-27 14:28:36.93-04  | USER_ADD_VM   | VM external-es1.cloud.big.local was created by <UNKNOWN>.
 962eb781-4dfa-46a0-a43a-957c54dabc11 | external-sql-dev-big-local   | 2015-05-27 14:28:36.588-04 | USER_ADD_VM   | VM external-sql-dev-big-local was created by <UNKNOWN>.
(7 rows)



engine.log
-----------
2015-05-27 21:28:33,157 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) [3c0d402c] Importing VM sql-dev-big-local as external-sql-dev-big-local, as it is running on the on Host, but does not exist in the engine.
2015-05-27 21:28:33,157 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) [3c0d402c] Importing VM es3-test-big-local as external-es3-test-big-local, as it is running on the on Host, but does not exist in the engine.
2015-05-27 21:28:33,157 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) [3c0d402c] Importing VM es1.cloud.big.local as external-es1.cloud.big.local, as it is running on the on Host, but does not exist in the engine.
2015-05-27 21:28:33,157 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) [3c0d402c] Importing VM win2008R2-temp as external-win2008R2-temp, as it is running on the on Host, but does not exist in the engine.
...
2015-05-27 21:28:36,594 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) [487642b6] Correlation ID: 487642b6, Call Stack: null, Custom Event ID: -1, Message: VM external-sql-dev-big-local was created by <UNKNOWN>

Comment 10 Omer Frenkel 2015-06-07 07:39:08 UTC
it's not a bug, the engine imports vms detected running on the host and allows you to see them and manage them (partially).
these are probably vms that were run by some kind of previous setup and were not cleared well (usually when force removing host or storage, or engine-reinstall)

please mind the difference between this case of automatically importing vms that are RUNNING on the host and regular import from export domain.

Comment 28 Israel Pinto 2015-11-05 12:05:50 UTC
Verify with:
RHEVM: rhevm-tools-3.6.0.3-0.1
vdsm:vdsm-4.17.10.1-0.el7ev
libvirt: libvirt-1.2.17-13.el7

Scenario:
1. Migration VM
2. While VM being migrate send request via rest to delete VM.

Results:
Delete action failed.
<fault>
    <reason>Operation Failed</reason>
    <detail>[Cannot remove VM. VM golden_env_mixed_virtio_0 is being migrated.]</detail>
</fault>

Comment 30 errata-xmlrpc 2016-03-09 20:35:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0376.html