Bug 1052097

Summary: VM migration back to original host fail on "file exists"
Product: Red Hat Enterprise Virtualization Manager Reporter: Ilanit Stein <istein>
Component: vdsmAssignee: Vinzenz Feenstra [evilissimo] <vfeenstr>
Status: CLOSED ERRATA QA Contact: Ilanit Stein <istein>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.3.0CC: acathrow, bazulay, gwatson, iheim, jhunsaker, lpeer, michal.skrivanek, Rhev-m-bugs, rhodain, sherold, ssekidde, yeylon, zdover
Target Milestone: ---Keywords: ZStream
Target Release: 3.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: virt
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previously, when links were left behind on a host, migration of a virtual machine to that host failed and the virtual machine was shut down on the host. This happened in cases in which the version of VDSM supported by Red Hat Enterprise Virtualization 3.2 (4.10.2) was present on the host. This was caused by a failure to remove guest socket files. Only symlinks to the guest socket files were removed. Links are now cleaned when virtual machines are migrated from hosts (the guest socket files linked to are now removed prior to the removal of the symlinks). Virtual machines no longer shut down when they are migrated back to those hosts from other hosts.
Story Points: ---
Clone Of:
: 1082986 1088906 (view as bug list) Environment:
Last Closed: 2014-06-09 13:27:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1082986    
Bug Blocks: 1088906    
Attachments:
Description Flags
engine.log migration start & fail @ 2014-Jan-12, 16:21
none
host1 logs
none
host2 logs none

Description Ilanit Stein 2014-01-13 10:20:41 UTC
Description of problem:

On rhevm 3.3 (is30), Have 2 3.3 hosts in 3.2 cluster.
Downgrade vdsm of host1 (remove the 3.3 vdsm, and install 3.2 vdsm).
Run a VM on host1. 
Migrate the VM to host2.
Upgrade host1 vdsm to 3.3.
Migrate VM back to original host, host1
Migration failed on "file exists" error.

Version-Release number of selected component (if applicable):
is30
vdsm 3.3: (is30  ) vdsm-4.13.2-0.5.el6ev.x86_64.rpm
vdsm 3.2: (sf22.4) vdsm-4.10.2-30.1.el6ev.x86_64.rpm

Additional info:
event error on rhevm: VM mig_ver is down. Exit message: [Errno 17] File exists.

vdsm error on destination host (host1):
Thread-38::ERROR::2014-01-12 16:21:09,752::vm::2162::vm.Vm::(_startUnderlyingVm) vmId=`53396dbd-388f-47be-a936-4a92a5dc6a6a`::The vm start process failed
Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 2140, in _startUnderlyingVm
    self._waitForIncomingMigrationFinish()
  File "/usr/share/vdsm/vm.py", line 3575, in _waitForIncomingMigrationFinish
    self._domDependentInit()
  File "/usr/share/vdsm/vm.py", line 2926, in _domDependentInit
    self._updateAgentChannels()
  File "/usr/share/vdsm/vm.py", line 2912, in _updateAgentChannels
    os.symlink(path, socketPath)
OSError: [Errno 17] File exists
Thread-38::DEBUG::2014-01-12 16:21:09,766::vm::2615::vm.Vm::(setDownStatus) vmId=`53396dbd-388f-47be-a936-4a92a5dc6a6a`::Changed state to Down: [Errno 17] File exists

Comment 1 Ilanit Stein 2014-01-13 10:21:47 UTC
Created attachment 849248 [details]
engine.log migration start & fail @ 2014-Jan-12, 16:21

Comment 2 Ilanit Stein 2014-01-13 10:22:52 UTC
Created attachment 849249 [details]
host1 logs

Comment 3 Ilanit Stein 2014-01-13 10:23:30 UTC
Created attachment 849250 [details]
host2 logs

Comment 4 Michal Skrivanek 2014-02-03 15:45:29 UTC
not sure if 3.3.z needed, let's review

Comment 5 Vinzenz Feenstra [evilissimo] 2014-02-04 11:36:32 UTC
*** Bug 1052841 has been marked as a duplicate of this bug. ***

Comment 7 Vinzenz Feenstra [evilissimo] 2014-04-01 09:18:18 UTC
Merged to u/s master as http://gerrit.ovirt.org/gitweb?p=vdsm.git;a=commit;h=4a7af0277aa441478b040163190ded27a9d7dcc2

Comment 8 Vinzenz Feenstra [evilissimo] 2014-04-01 13:46:02 UTC
Merged u/s to ovirt-3.4 branch as http://gerrit.ovirt.org/gitweb?p=vdsm.git;a=commit;h=be6302a938c1a1f1dac1e03b65ce10c7fecceb92

Comment 10 Michal Skrivanek 2014-04-16 11:30:51 UTC
discovered an issue described in bug 1085801. We need an additional fix and possibly a backport to 3.3 as more and more people are hitting this

Comment 11 Michal Skrivanek 2014-04-16 11:31:42 UTC
*** Bug 1085801 has been marked as a duplicate of this bug. ***

Comment 14 Ilanit Stein 2014-05-01 21:35:37 UTC
Verified on rhevm3.4 av8.1:

Had 3.2 cluster.
host_1 3.3 vdsm vdsm-4.13.2-0.14.el6ev (from is36)
host_2 3.4 vdsm vdsm-4.14.7-0.2.rc.el6ev.x86_64
Run a VM on host_1
Migrate VM to host_2
Upgrade vdsm on host_1 from 3.3 to 3.4
Migrate VM back to host_1

NO migration failures.

Comment 15 errata-xmlrpc 2014-06-09 13:27:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0504.html