Bug 1121089 - [RHEV] VM removed while migrating, resulting in "VM not managed by engine"
Summary: [RHEV] VM removed while migrating, resulting in "VM not managed by engine"
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 3.3.0
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ovirt-3.6.0-rc
: 3.6.0
Assignee: Michal Skrivanek
QA Contact: Israel Pinto
URL:
Whiteboard:
Depends On: 947965 1253043
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-07-18 10:37 UTC by Evgheni Dereveanchin
Modified: 2019-08-15 03:54 UTC (History)
22 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, in rare cases a virtual machine disappeared during migration in the UI and reappeared as an external virtual machine that could no longer be managed. This error has now been fixed.
Clone Of:
Environment:
Last Closed: 2016-03-09 20:35:39 UTC
oVirt Team: Virt
Target Upstream Version:
mgoldboi: Triaged+


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:0376 normal SHIPPED_LIVE Red Hat Enterprise Virtualization Manager 3.6.0 2016-03-10 01:20:52 UTC
oVirt gerrit 31204 master MERGED core: migrating-to is a running state as well Never
oVirt gerrit 46518 ovirt-engine-3.6 MERGED core: migrating-to is a running state as well Never
Red Hat Knowledge Base (Solution) 1472473 None None None 2016-04-04 17:27:32 UTC

Description Evgheni Dereveanchin 2014-07-18 10:37:10 UTC
In RHEV 3.3 there seems to be a race condition in which a VM remove command is executed when in fact the guest is migrating between hosts. This results in the guest being removed from the source host but coming up on destination. Disks are also removed so the guest is actually unusable and may hang, appearing on the manager with "external-" prefix.

The VM creation/removal is done via the API.

Comment 6 Omer Frenkel 2014-08-07 13:00:58 UTC
exact race flow, for documentation:
migration ends, and vm moves to up on the destination.
in the engine, there are 2 monitoring threads that run in parallel (src,dst)
dst finish first, and move the vm to up, and release the migration lock.
then src finish, but override the status to migratingTo (hand-over), because it is not aware to the other update.

now we have vm in migratingTo, without a lock.
resolution is to mark this status as a 'running' status, like migratingFrom, and user will be blocked from removing based on vm status, when this case happens.

i was able to reproduce this (using debugger).

Comment 7 Omer Frenkel 2014-09-01 11:56:26 UTC
The fix for bug 947965 also fix this issue, as it will not allow removing a vm in migrating-to, and that fix is much safer than the fix proposed in this bug.

Comment 9 Anand Nande 2015-06-05 12:34:59 UTC
Hi,

I am not sure if the following should be part of this bug OR a new bug.

Suddenly some virtual machines appeared in Manager, prefixed with 'external'. It seems they were removed from Export Domain but now imported again. 

All the VMs/Windows guests are from storage_domain="FC-NetAPP-Windows0"

engine=> SELECT vm_id,vm_name,log_time,log_type_name,message from audit_log where message like '%external%';
                vm_id                 |           vm_name            |          log_time          | log_type_name |                          message                          
--------------------------------------+------------------------------+----------------------------+---------------+-----------------------------------------------------------
 5ef17f98-6ac4-4cd7-ba14-4f52dfac8bec | external-eevrt-002-big-local | 2015-05-26 16:39:10.603-04 | USER_ADD_VM   | VM external-eevrt-002-big-local was created by <UNKNOWN>.
 bd1e8f7f-db44-4207-8215-6394a239ee3a | external-skoor0-big-local    | 2015-05-26 16:39:10.453-04 | USER_ADD_VM   | VM external-skoor0-big-local was created by <UNKNOWN>.
 cb93d661-3fc7-4d13-beb4-765a2867febc | external-2008R2TEST          | 2015-05-27 14:26:47.257-04 | USER_ADD_VM   | VM external-2008R2TEST was created by <UNKNOWN>.
 474a70bb-8ba2-4b85-8aa8-8a79fbf18a15 | external-win2008R2-temp      | 2015-05-27 14:28:37.111-04 | USER_ADD_VM   | VM external-win2008R2-temp was created by <UNKNOWN>.
 4b76def7-74b6-4a0f-8a69-8a339df82144 | external-es3-test-big-local  | 2015-05-27 14:28:36.776-04 | USER_ADD_VM   | VM external-es3-test-big-local was created by <UNKNOWN>.
 908047a3-8d05-4273-8314-61023996e081 | external-es1.cloud.big.local | 2015-05-27 14:28:36.93-04  | USER_ADD_VM   | VM external-es1.cloud.big.local was created by <UNKNOWN>.
 962eb781-4dfa-46a0-a43a-957c54dabc11 | external-sql-dev-big-local   | 2015-05-27 14:28:36.588-04 | USER_ADD_VM   | VM external-sql-dev-big-local was created by <UNKNOWN>.
(7 rows)



engine.log
-----------
2015-05-27 21:28:33,157 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) [3c0d402c] Importing VM sql-dev-big-local as external-sql-dev-big-local, as it is running on the on Host, but does not exist in the engine.
2015-05-27 21:28:33,157 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) [3c0d402c] Importing VM es3-test-big-local as external-es3-test-big-local, as it is running on the on Host, but does not exist in the engine.
2015-05-27 21:28:33,157 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) [3c0d402c] Importing VM es1.cloud.big.local as external-es1.cloud.big.local, as it is running on the on Host, but does not exist in the engine.
2015-05-27 21:28:33,157 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) [3c0d402c] Importing VM win2008R2-temp as external-win2008R2-temp, as it is running on the on Host, but does not exist in the engine.
...
2015-05-27 21:28:36,594 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) [487642b6] Correlation ID: 487642b6, Call Stack: null, Custom Event ID: -1, Message: VM external-sql-dev-big-local was created by <UNKNOWN>

Comment 10 Omer Frenkel 2015-06-07 07:39:08 UTC
it's not a bug, the engine imports vms detected running on the host and allows you to see them and manage them (partially).
these are probably vms that were run by some kind of previous setup and were not cleared well (usually when force removing host or storage, or engine-reinstall)

please mind the difference between this case of automatically importing vms that are RUNNING on the host and regular import from export domain.

Comment 28 Israel Pinto 2015-11-05 12:05:50 UTC
Verify with:
RHEVM: rhevm-tools-3.6.0.3-0.1
vdsm:vdsm-4.17.10.1-0.el7ev
libvirt: libvirt-1.2.17-13.el7

Scenario:
1. Migration VM
2. While VM being migrate send request via rest to delete VM.

Results:
Delete action failed.
<fault>
    <reason>Operation Failed</reason>
    <detail>[Cannot remove VM. VM golden_env_mixed_virtio_0 is being migrated.]</detail>
</fault>

Comment 30 errata-xmlrpc 2016-03-09 20:35:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0376.html


Note You need to log in before you can comment on or make changes to this bug.