Bug 1274670 - VM migration doesn't work with current VDSM master
VM migration doesn't work with current VDSM master
Status: CLOSED CURRENTRELEASE
Product: vdsm
Classification: oVirt
Component: General (Show other bugs)
---
Unspecified Linux
unspecified Severity urgent (vote)
: ovirt-4.0.0-alpha
: 4.17.999
Assigned To: Michal Skrivanek
Israel Pinto
: Reopened
: 1282054 (view as bug list)
Depends On:
Blocks: 1188543
  Show dependency treegraph
 
Reported: 2015-10-23 06:24 EDT by Milan Zamazal
Modified: 2016-07-05 03:48 EDT (History)
7 users (show)

See Also:
Fixed In Version: ovirt 4.0.0 alpha1
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-07-05 03:48:49 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Virt
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
rule-engine: ovirt‑4.0.0+
ipinto: testing_plan_complete+
rule-engine: planning_ack+
rule-engine: devel_ack+
mavital: testing_ack+


Attachments (Terms of Use)
Source and destination VDSM logs (443.62 KB, text/plain)
2015-10-23 06:24 EDT, Milan Zamazal
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 48803 master MERGED stomp: Drain pending bytes from ssl socket Never

  None (edit)
Description Milan Zamazal 2015-10-23 06:24:01 EDT
Created attachment 1085781 [details]
Source and destination VDSM logs

Description of problem:

When I initiate VM migration from one host to another or the other way round in oVirt engine web frontend, it fails with current VDSM master.

Frontend message: Migration failed
VDSM source host message: No response for JSON-RPC VM.migrationCreate request.

More information from VDSM logs is attached.

Version-Release number of selected component (if applicable):

CentOS 7 installed in virtual machines on my computer.
Engine version: ovirt-engine-3.6.0.2-1.el7.centos.noarch
VDSM version on the hosts: vdsm-4.17.999-53.git431ac02.el7.noarch (current master) 

How reproducible:

Always reproducible on my machine.

Steps to Reproduce:
1. Select "Virtual Machines" tab in oVirt engine web frontend.
2. Select one of the VMs.
3. Press "Run Once" and then "OK".
4. Wait until the VM starts.
5. Press "Migrate" and then "OK".

Actual results:

The frontend displays VM migration progress bar with 0%, it reports migration failure after about 15 seconds and the VM remains running on the source host. 

Expected results:

The VM is migrated to the destination host.

Additional info:
Comment 1 Milan Zamazal 2015-10-23 08:11:49 EDT
Results with 3.6:
I tried to add two new hosts with vdsm-4.17.10-0.el7.centos.noarch (3.6 repo) to my setup. Migration works between those hosts.
Comment 2 Piotr Kliczewski 2015-10-28 11:55:57 EDT
I reproduced the issue and it seems that it is not related to communication code.
The issue is that when we send VM.migrationCreate there is no response and we timeout based on the setting (15 seconds). Even though i change the timeout to 25 mins there is no response from destination host and it timeouts.

We need to understand why there is no response in timely manner.
Comment 3 Piotr Kliczewski 2015-11-16 03:11:34 EST
*** Bug 1282054 has been marked as a duplicate of this bug. ***
Comment 4 Piotr Kliczewski 2015-11-19 07:24:11 EST
Please check whether the problem still exists when [1] is applied.

[1] https://gerrit.ovirt.org/#/c/48803/
Comment 5 Milan Zamazal 2015-11-19 07:46:05 EST
Migration works with https://gerrit.ovirt.org/#/c/48803/ .
Comment 6 Nir Soffer 2015-11-19 07:49:14 EST
(In reply to Milan Zamazal from comment #5)
> Migration works with https://gerrit.ovirt.org/#/c/48803/ .

Do you want to add your verification to the patch?
Comment 7 Milan Zamazal 2015-11-19 08:00:58 EST
(In reply to Nir Soffer from comment #6)
> Do you want to add your verification to the patch?

Done.
Comment 8 Red Hat Bugzilla Rules Engine 2015-11-20 11:32:56 EST
Fixed bug tickets must have version flags set prior to fixing them. Please set the correct version flags and move the bugs back to the previous status after this is corrected.
Comment 9 Michal Skrivanek 2015-12-07 06:26:17 EST
(In reply to Milan Zamazal from comment #1)
> Results with 3.6:
> I tried to add two new hosts with vdsm-4.17.10-0.el7.centos.noarch (3.6
> repo) to my setup. Migration works between those hosts.

in 1112594 we introduced the jsonrpc code to migration, can you please doublecheck the behavior, it should be broken in 3.6 too
Comment 10 Francesco Romani 2015-12-07 07:16:37 EST
(In reply to Michal Skrivanek from comment #9)
> (In reply to Milan Zamazal from comment #1)
> > Results with 3.6:
> > I tried to add two new hosts with vdsm-4.17.10-0.el7.centos.noarch (3.6
> > repo) to my setup. Migration works between those hosts.
> 
> in 1112594 we introduced the jsonrpc code to migration, can you please
> doublecheck the behavior, it should be broken in 3.6 too

I agree to doublecheck, but the breaking patch was merged among some JSON-RPC notification enhancements.
I believe both jsonrpc migrations and json notification we added in 3.6.0 are safe.
Not sure about further backports, however.
Comment 11 Nir Soffer 2015-12-07 07:24:50 EST
(In reply to Michal Skrivanek from comment #9)
> (In reply to Milan Zamazal from comment #1)
> > Results with 3.6:
> > I tried to add two new hosts with vdsm-4.17.10-0.el7.centos.noarch (3.6
> > repo) to my setup. Migration works between those hosts.
> 
> in 1112594 we introduced the jsonrpc code to migration, can you please
> doublecheck the behavior, it should be broken in 3.6 too

The root cause is https://gerrit.ovirt.org/#/c/46911 which was not
backported to 3.6.
Comment 12 Milan Zamazal 2015-12-21 07:59:38 EST
AFAICT I've never experienced this one problem with 3.6 and there is already explanation by Francesco and Nir.
Comment 13 Israel Pinto 2016-06-15 05:08:25 EDT
Verify with:
RHEVM: 4.0.0.2-0.1.el7ev
hosts:
OS Version:RHEL - 7.2 - 9.el7
Kernel Version:3.10.0 - 327.18.2.el7.x86_64
KVM Version:2.3.0 - 31.el7_2.10
LIBVIRT Version:libvirt-1.2.17-13.el7_2.4
VDSM Version:vdsm-4.18.999-79.gitfc45278.el7.centos

Steps:
1. Create VM
2. Run VM with "Run Once"
3. Migrate VM
4. Also tested without "Run Once"

VM migrated successfully.
Comment 14 Sandro Bonazzola 2016-07-05 03:48:49 EDT
oVirt 4.0.0 has been released, closing current release.

Note You need to log in before you can comment on or make changes to this bug.